Hand-eye calibration method of 3D camera eye outside hand based on tool ball and related device

Abstract

The invention discloses an out-of-hand hand-eye calibration method of a 3D camera eye based on a tool ball and a related device, relates to the field of computer vision, realizes an automatic hand-eye calibration process by controlling the tool ball to move to a plurality of preset calibration points, and reduces manual intervention. The tool ball is fixed on the mechanical arm, and the relative position of the tool ball and the end effector of the mechanical arm is kept unchanged, so that the coordinate of the end effector of the mechanical arm in a robot coordinate system can be used as the first coordinate of the center of the tool ball in the robot coordinate system, and the second coordinate of the center of the tool ball in a camera coordinate system is accurately obtained through 3D point cloud data; the precision of hand-eye calibration can be improved, and then the accuracy and stability of task execution of the robot are improved. The affine transformation matrix is obtained based on the plurality of first coordinates and the plurality of second coordinates, so that the calibration result is more representative and reliable, the problem of inaccurate calibration caused by single-point errors is reduced, and the adaptability and robustness of the system in different environments and tasks are enhanced.

Description

Technical Field

[0001] The present application relates to the field of computer vision technology, and in particular to a 3D camera eye-in-hand and eye-out-of-hand calibration method and related devices based on a tool ball. Background Art

[0002] With the continuous advancement of technology, robots equipped with 3D cameras and robotic arms are increasingly being used in a variety of fields, including industrial manufacturing, logistics, and healthcare. They have become a key technology for promoting production automation, improving efficiency, and increasing the level of intelligence. By acquiring and analyzing three-dimensional information of target objects in real time, robots can greatly improve work efficiency and product quality, while significantly reducing labor costs and human errors, resulting in higher productivity and economic benefits. 3D cameras play a vital role in industrial production. By accurately capturing three-dimensional images of the objects to be manipulated, they provide robots with accurate position information, enabling robotic arms to perform complex tasks such as object detection, classification, and grasping. In particular, during the intelligent classification of target objects, 3D cameras can help robotic arms determine the object's shape, size, and spatial position, enabling precise manipulation. In this process, hand-eye calibration is a key step in ensuring that robots can accurately perform tasks.

[0003] The purpose of hand-eye calibration is to establish a mapping relationship between the coordinate system of the robot arm's end effector (hand) (i.e., the robot coordinate system) and the camera coordinate system (eye), namely, the affine transformation matrix, which converts the camera coordinate system to the robot coordinate system. This allows the coordinates of any point in the camera coordinate system to be converted to the coordinates in the robot coordinate system, allowing the robot to perform subsequent recognition, guidance, and positioning based on the coordinates in the robot coordinate system.

[0004] Based on this, how to obtain the affine transformation matrix is ​​a difficult problem that technicians in this field urgently need to solve. Summary of the Invention

[0005] In view of the above problems, this application provides a 3D camera eye-in-hand and eye-out-hand calibration method and related devices based on a tool sphere to achieve the purpose of obtaining an affine transformation matrix. The specific solution is as follows:

[0006] In a first aspect, the present application provides a 3D camera eye-in-hand calibration method based on a tool ball, which is applied to a robot. The robot includes a robotic arm and a 3D camera. The tool ball is fixed to an end effector of the robotic arm, and includes:

[0007] Controlling the tool ball to move to a plurality of preset calibration points respectively;

[0008] For each of the preset calibration points, determining the coordinates of the preset calibration point in the robot coordinate system as the first coordinates of the center of the tool ball in the robot coordinate system;

[0009] For each of the preset calibration points, acquiring 3D point cloud data of the tooling sphere located at the preset calibration point through a 3D camera;

[0010] For each of the preset calibration points, based on the 3D point cloud data corresponding to the preset calibration point, obtaining a second coordinate of the center of the tool sphere in the camera coordinate system;

[0011] A first affine transformation matrix is ​​obtained based on the first coordinates corresponding to the plurality of preset calibration points and the second coordinates corresponding to the plurality of preset calibration points.

[0012] In a possible implementation, acquiring 3D point cloud data of the tool sphere located at the preset calibration point by using a 3D camera includes:

[0013] Acquire candidate 3D point cloud data at the preset calibration point through a 3D camera;

[0014] 3D point cloud data that matches a preset template sphere 3D point cloud model of the tool sphere is acquired from the candidate 3D point cloud data, where the preset template sphere 3D point cloud model includes surface normal vectors of a plurality of points on the spherical surface of the tool sphere.

[0015] In a possible implementation, the first affine transformation matrix includes: a rotation transformation matrix from the camera coordinate system to the robot coordinate system, a translation transformation matrix from the camera coordinate system to the robot coordinate system, and further includes:

[0016] Obtaining a third coordinate of an artificially moved point in the camera coordinate system and a fourth coordinate in the robot coordinate system; the artificially moved point is a position to which the center of the tool ball of the end effector of the robotic arm is manually moved;

[0017] Obtain a standard translation transformation matrix based on the rotation transformation matrix, the third coordinate, and the fourth coordinate;

[0018] The rotation transformation matrix and the standard translation transformation matrix are determined to be a second affine transformation matrix.

[0019] In a possible implementation, the method further includes:

[0020] Controlling the center of the tool ball fixed on the end effector of the robotic arm to move to each grid vertex of the rectangular grid;

[0021] For each mesh vertex, acquiring 3D point cloud data of the center of the tool sphere located at the mesh vertex through a 3D camera;

[0022] For each mesh vertex, based on the 3D point cloud data corresponding to the mesh vertex, obtaining the fifth coordinate of the center of the tool sphere in the camera coordinate system;

[0023] For each mesh vertex, obtaining a standard sixth coordinate in the robot coordinate system corresponding to the center of the tool ball located at the mesh vertex;

[0024] Obtaining a calculated sixth coordinate corresponding to each mesh vertex based on the second affine transformation matrix, the fifth coordinates corresponding to each of the mesh vertices, and tool data of the tool sphere, wherein the tool data includes a fixed position and posture of the center of the tool sphere relative to the end effector of the robot arm;

[0025] For each of the mesh vertices, a compensation value is obtained based on the standard sixth coordinate corresponding to the mesh vertex and the calculated sixth coordinate.

[0026] In a possible implementation, the rectangular parallelepiped mesh includes a plurality of sub-mesh units, each of which includes 8 mesh vertices, and further includes:

[0027] Acquire 3D point cloud data of the object to be operated through a 3D camera;

[0028] Obtaining the seventh coordinate of each point in the 3D point cloud data of the object to be operated in the camera coordinate system;

[0029] Based on the second affine transformation matrix and the seventh coordinate, obtaining an eighth coordinate of each point in the 3D point cloud data of the object to be operated in the robot coordinate system;

[0030] Determine that the 3D point cloud data of the object to be operated is located in a target subgrid unit in the rectangular grid;

[0031] For each point in the 3D point cloud data of the object to be operated, obtaining a compensation value for the point based on the compensation data corresponding to the eight grid vertices of the target subgrid unit;

[0032] acquiring a ninth coordinate of each point in the 3D point cloud data of the object to be operated based on the eighth coordinate of each point in the 3D point cloud data of the object to be operated and the compensation value of each point in the 3D point cloud data of the object to be operated;

[0033] The robot is controlled to operate on the ninth coordinate of each point in the 3D point cloud data of the object to be operated.

[0034] In a possible implementation, obtaining the compensation value of the point based on the compensation data corresponding to the eight grid vertices of the target subgrid unit includes:

[0035] Projecting the point onto the lower surface of the target subgrid unit to obtain a first point;

[0036] Projecting the point onto the upper surface of the target subgrid unit to obtain a second point;

[0037] Projecting the first point onto the first edge of the lower surface to obtain a third point;

[0038] Projecting the first point onto the second side of the lower surface to obtain a fourth point, where the first side is parallel to the second side;

[0039] Determining position parameters of the first point on a line segment with the third point and the fourth point as endpoints, where the position parameters of the point on the line segment include a ratio of a distance between the point and one endpoint of the line segment to a length of the line segment;

[0040] Calculating a compensation value of the third point based on compensation data corresponding to two mesh vertices of the first side and a position parameter of the third point on the first side;

[0041] Calculating a compensation value of the fourth point based on the compensation data corresponding to the two mesh vertices of the second side and a position parameter of the fourth point on the second side;

[0042] calculating a compensation value of the first point based on the compensation data corresponding to the third point, the compensation data corresponding to the fourth point, and the position parameter corresponding to the first point;

[0043] Projecting the second point onto the third side of the upper surface to obtain a fifth point;

[0044] Projecting the second point onto the fourth side of the upper surface to obtain a sixth point, where the third side is parallel to the fourth side and the third side is parallel to the first side;

[0045] determining position parameters of the second point located on a line segment having the fifth point and the sixth point as endpoints;

[0046] Calculating a compensation value of the fifth point based on the compensation data corresponding to the two mesh vertices of the third side and a position parameter of the fifth point located on the third side;

[0047] Calculating a compensation value of the sixth point based on compensation data corresponding to the two mesh vertices of the fourth side and a position parameter of the sixth point located on the fourth side;

[0048] calculating a compensation value of the second point based on the compensation data corresponding to the fifth point, the compensation data corresponding to the sixth point, and the position parameter corresponding to the second point;

[0049] Obtaining position parameters of the point located on a line segment with the first point and the second point as endpoints;

[0050] A compensation value of the point is obtained based on the compensation data of the first point, the compensation data of the second point, and a position parameter corresponding to the point.

[0051] In a possible implementation, the method for acquiring the tool data of the tool ball includes:

[0052] Obtaining the position coordinates of the flange center of the robot in the robot coordinate system;

[0053] Obtaining a second affine transformation matrix from a flange coordinate system to a robot coordinate system based on the pose coordinates, wherein the flange coordinate system refers to a coordinate system established with the center of the flange at the end of the robotic arm as the origin;

[0054] The tool data of the tool sphere is obtained based on the second affine transformation matrix and the coordinates of the center of the tool sphere in the camera coordinate system.

[0055] In a possible implementation, the method for dividing the rectangular parallelepiped grid includes:

[0056] Obtaining the starting coordinates of the rectangular grid in the robot coordinate system, the ending coordinates of the rectangular grid in the robot coordinate system, the first length of the rectangular grid, the first width of the rectangular grid, and the first height of the rectangular grid;

[0057] Obtain a first number of subgrid units divided in the X-axis direction of the robot coordinate system, a second number of subgrid units divided in the Y-axis direction, a third number of subgrid units divided in the Z-axis direction, and a second length, a second width, and a second height of each subgrid unit;

[0058] The rectangular parallelepiped mesh is generated based on the start coordinates, the end coordinates, the first length, the first width, the first height, the first number, the second number, the third number, the second length, the second width, and the second height.

[0059] A second aspect of the present application provides a 3D camera eye-in-hand and eye-out-of-hand calibration device based on a tool ball, which is applied to a robot. The robot includes a robotic arm and a 3D camera. The tool ball is fixed to the end effector of the robotic arm, and includes:

[0060] A first control movement module is used to control the tool ball to move to a plurality of preset calibration points respectively;

[0061] a first determining module, configured to determine, for each of the preset calibration points, a coordinate of the preset calibration point in the robot coordinate system as a first coordinate of the center of the tool ball in the robot coordinate system;

[0062] A first acquisition module is configured to acquire, for each of the preset calibration points, 3D point cloud data of the tooling sphere located at the preset calibration point through a 3D camera;

[0063] A second acquisition module is configured to acquire, for each of the preset calibration points, a second coordinate of the center of the tool sphere in the camera coordinate system based on the 3D point cloud data corresponding to the preset calibration point;

[0064] The third acquisition module is used to obtain a first affine transformation matrix based on the first coordinates corresponding to the multiple preset calibration points and the second coordinates corresponding to the multiple preset calibration points.

[0065] A third aspect of the present application provides a computer program product, comprising computer-readable instructions. When the computer-readable instructions are executed on an electronic device, the electronic device implements the tool ball-based 3D camera eye-in-hand and eye-out-of-hand calibration method of the first aspect or any implementation of the first aspect.

[0066] A fourth aspect of the present application provides an electronic device, comprising at least one processor and a memory connected to the processor, wherein:

[0067] The memory is used to store computer programs;

[0068] The processor is used to execute the computer program so that the electronic device can implement the tool ball-based 3D camera eye-outside-hand hand-eye calibration method of the above-mentioned first aspect or any implementation method of the first aspect.

[0069] In a fifth aspect, the present application provides a computer storage medium carrying one or more computer programs. When the one or more computer programs are executed by an electronic device, the electronic device can implement the tool ball-based 3D camera eye-outside-hand hand-eye calibration method according to the above-mentioned first aspect or any implementation of the first aspect.

[0070] By means of the above technical solution, the present application provides a 3D camera eye-in-hand and eye-out-of-hand calibration method based on a tool ball. By controlling the tool ball to move to multiple preset calibration points, an automated hand-eye calibration process is realized, which reduces manual intervention and improves the efficiency and convenience of calibration. Since the tool ball is fixed on the robotic arm and its relative position with the end effector of the robotic arm remains unchanged, the coordinates of the end effector of the robotic arm in the robot coordinate system can be used as the first coordinates of the center of the tool ball in the robot coordinate system. The second coordinates of the center of the ball in the camera coordinate system are accurately obtained through 3D point cloud data, which can improve the accuracy of hand-eye calibration and thus improve the accuracy and stability of the robot's task execution. The first affine transformation matrix is ​​obtained based on multiple first coordinates and multiple second coordinates, making the calibration results more representative and reliable, reducing the calibration inaccuracy problem caused by single-point error, and enhancing the adaptability and robustness of the system in different environments and tasks. BRIEF DESCRIPTION OF THE DRAWINGS

[0071] The above and other features, advantages, and aspects of the various embodiments of the present disclosure will become more apparent with reference to the following detailed description in conjunction with the accompanying drawings. Throughout the drawings, the same or similar reference numerals represent the same or similar elements. It should be understood that the drawings are schematic and that the originals and elements are not necessarily drawn to scale.

[0072] Figure 1a to Figure 1b Schematic diagram of two relative positions of the robotic arm and 3D camera in the robot provided in an embodiment of the present application;

[0073] Figure 2 A flowchart of a 3D camera eye-in-hand and eye-out-of-hand calibration method based on a tool sphere provided in an embodiment of the present application;

[0074] Figure 3 A schematic diagram of a rectangular grid provided in an embodiment of the present application;

[0075] Figure 4 A schematic diagram of a target sub-grid unit provided in an embodiment of the present application;

[0076] Figure 5 The main flow chart provided for the embodiments of this application;

[0077] Figure 6 A schematic diagram of the structure of a 3D camera eye-in-hand and eye-out-of-hand calibration device based on a tool ball provided in an embodiment of the present application;

[0078] Figure 7 A schematic diagram of the structure of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0079] The following describes the embodiments of the present application in conjunction with the accompanying drawings. The terms used in the implementation methods of the present application are only used to explain the specific embodiments of the present application and are not intended to limit the present application.

[0080] The embodiments of the present application are described below in conjunction with the accompanying drawings. Those skilled in the art will appreciate that, with the development of technology and the emergence of new scenarios, the technical solutions provided in the embodiments of the present application are also applicable to similar technical problems.

[0081] The terms "first", "second", etc. in the specification and claims of the present application and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequential order. It should be understood that the terms used in this way can be interchangeable under appropriate circumstances, and this is merely a way of distinguishing the objects of the same attributes when describing them in the embodiments of the present application. In addition, the terms "including" and "having" and any of their variations are intended to cover non-exclusive inclusions, so that the process, method, system, product or equipment comprising a series of units need not be limited to those units, but may include other units that are not clearly listed or inherent to these processes, methods, products or equipment.

[0082] There are two relative positions of the 3D camera and the robotic arm, which are explained below with reference to the accompanying drawings.

[0083] like Figure 1a to Figure 1b , which is a schematic diagram of two relative positions of the robotic arm and the 3D camera in the robot provided in an embodiment of the present application.

[0084] Figure 1a A schematic diagram showing an eye-on-hand, i.e., a 3D camera (eye) is mounted on the end effector (hand) of a robotic arm, and the 3D camera moves directly with the robotic arm.

[0085] Figure 1b A schematic diagram showing an eye-outside-hand configuration, where the 3D camera (eye) is outside the end effector (hand) of the robotic arm. The 3D camera (eye) is mounted on a fixed location outside the robotic arm (hand), typically a workbench or other stationary reference location.

[0086] pass Figure 1b It can be seen that the eye is outside the hand, and the movement of the robotic arm will not directly affect the viewing angle of the 3D camera.

[0087] This application is mainly aimed at Figure 1b The eye-outside-hand configuration shown in the figure designs an automated hand-eye calibration and accuracy optimization method.

[0088] In the related art, most of the hand-eye calibration methods with the eye outside the hand are to select multiple calibration points, obtain the coordinates of each calibration point in the camera coordinate system and the robot coordinate system, and calculate the calibration matrix through the mathematical affine transformation solution method. Therefore, after shooting the 3D point cloud data, this type of method requires manual acquisition of the coordinates of the calibration points in the camera internal coordinates and the robot coordinate system multiple times, and requires human eyes to check whether the end effector of the robotic arm is aligned with the selected calibration points. This process is not only time-consuming and labor-intensive, but also has human eye errors. In addition, considering that the internal parameter error of the 3D camera itself will directly lead to a reduction in the accuracy of the point cloud obtained by shooting, it will affect the accuracy of the hand-eye calibration, and then lead to the accuracy of the robot arm operating position of the robot based on vision guidance.

[0089] To address the above problems, this application uses a regular sphere with a known radius as a tool to design an automated hand-eye calibration and accuracy compensation method for a 3D camera eye outside the hand based on the tool sphere, thereby achieving automated calibration while reducing calibration errors, thereby further improving the automation level and accuracy of vision-guided robots.

[0090] This application can be applied to industrial robot scenarios.

[0091] The purpose of hand-eye calibration of 3D camera eye outside the hand is to obtain the 4×4 first affine transformation matrix from the camera coordinate system (eye) to the robot coordinate system (hand) That is, given the coordinates P of a certain point A in the camera coordinate system cam and the coordinates P in the robot coordinate system rob , then the following equation holds:

[0092]

[0093] in, X cam , Y cam , Z cam Respectively represent the coordinates of point A in the X-axis, Y-axis, and Z-axis directions in the camera coordinate system. X rob , Y rob , Z rob They represent the coordinates of point A in the X-axis, Y-axis, and Z-axis directions in the robot coordinate system respectively. represents the 3×3 rotation transformation matrix from the camera coordinate system (eye) to the robot coordinate system (hand), Represents the 3×1 translation transformation matrix from the camera coordinate system (eye) to the robot coordinate system (hand).

[0094] It is understandable that if the coordinates of more than 4 points in the camera coordinate system and the coordinates in the robot coordinate system are known, the first affine transformation matrix can be obtained by solving the 3D affine transformation.

[0095] Most of the eye-in-hand automated hand-eye calibration software in the related art cannot realize automated hand-eye calibration. It is necessary to manually operate the end effector of the robot's manipulator arm to the selected calibration point. Since the coordinates of the selected calibration point in the robot coordinate system cannot be known in advance, after moving to the vicinity of the selected calibration point, the human eye needs to carefully observe and operate the robot to accurately align the end effector of the manipulator arm with the calibration point. After alignment, the three-dimensional coordinates of the end effector of the manipulator arm are read from the robot teach pendant. It is assumed that the coordinates of the calibration point in the robot coordinate system are recorded as P rob Next, move the robot's arm away, use the 3D camera to take a picture to obtain 3D point cloud data, open the 3D point cloud data to obtain the three-dimensional coordinates of the selected calibration point in the camera coordinate system, and use the mouse to align the selected calibration point and read the three-dimensional coordinates of the calibration point in the camera coordinate system P cam Usually, at least 5 calibration points need to be selected. For example, repeat the above steps 6 to 10 times to obtain the coordinate values ​​of 6 to 10 groups of calibration points to solve the first affine transformation matrix. Understandably, the selected calibration points cannot lie in the same plane. Therefore, the steps of aligning the robot's end effector with the calibration point positions to obtain their coordinates in the robot coordinate system, and then taking photos with the 3D camera to obtain their coordinates in the camera coordinate system, need to be repeated at least five times. Furthermore, generally, the more calibration points selected, the more accurate the calculated calibration matrix, but this also significantly increases the workload and time consumption.

[0096] In response to the problem that automated hand-eye calibration cannot be achieved in related technologies, this application designs a method for automated hand-eye calibration of a 3D camera eye outside the hand based on a tool ball. The end effector of the robotic arm can automatically move to a preset calibration point and automatically control the 3D camera to take pictures. In addition, this application provides a sphere center fitting algorithm based on a 3D point cloud, which can automatically fit the three-dimensional coordinates of the center of the tool ball in the camera coordinate system based on the 3D point cloud data obtained by the 3D camera. Next, based on the three-dimensional coordinates of the preset calibration point in the robot coordinate system and the three-dimensional coordinates of the center of the tool ball in the camera coordinate system, the first affine transformation matrix can be calculated based on the affine transformation solution algorithm.

[0097] The following describes a 3D camera eye-outside-hand hand-eye calibration method and related devices based on a tool ball provided in an embodiment of the present application.

[0098] Reference Figure 2 , Figure 2 A flow chart of a 3D camera eye-in-hand hand-eye calibration method based on a tool ball provided in an embodiment of the present application is shown as follows: Figure 2As shown, a 3D camera eye-in-hand and eye-out-of-hand calibration method based on a tool ball provided in an embodiment of the present application may include steps S201 to S205, and these steps are described in detail below.

[0099] Step S201: controlling the tool ball to move to a plurality of preset calibration points respectively.

[0100] Exemplarily, the center of the tool ball is controlled to move to a preset calibration point.

[0101] Exemplarily, the end effector of the robotic arm is controlled by programming to move the tool ball to a preset calibration point.

[0102] For example, the preset calibration point can be Figure 3 Mesh vertices of a box mesh are shown.

[0103] Exemplarily, the preset calibration point is not a mesh vertex of the cuboid mesh, but may be a preset point.

[0104] Step S202: for each of the preset calibration points, determining the coordinates of the preset calibration point in the robot coordinate system as the first coordinates of the center of the tool ball in the robot coordinate system.

[0105] It can be understood that since the tool ball is fixed on the robot arm and its relative position to the end effector of the robot arm remains unchanged, the coordinate P of the end effector of the robot arm in the robot coordinate system can be r-rob (i.e. the coordinates of the preset calibration point) as the first coordinate P of the center of the tool ball in the robot coordinate system o-rob That is (P o-rob ≈P r-rob ).

[0106] For example, the number of preset calibration points can be determined based on actual conditions. For example, the number of preset calibration points is N, where N≥6. That is, N groups of P can be obtained. o-cam and P o-rob .

[0107] Step S203: For each of the preset calibration points, obtain 3D point cloud data of the tooling sphere located at the preset calibration point through a 3D camera.

[0108] Exemplarily, a 3D camera is automatically controlled to shoot and acquire 3D point cloud data.

[0109] Step S204: for each of the preset calibration points, based on the 3D point cloud data corresponding to the preset calibration point, obtaining the second coordinates of the center of the tooling sphere in the camera coordinate system.

[0110] For example, the sphere center fitting algorithm based on the 3D point cloud can be used to obtain the coordinates P of the sphere center in the camera coordinate system by taking the preset calibration point as the position of the sphere center of the tool sphere. o-cam .

[0111] Step S205: obtaining a first affine transformation matrix based on the first coordinates corresponding to the plurality of preset calibration points and the second coordinates corresponding to the plurality of preset calibration points.

[0112] Exemplarily, the first affine transformation matrix can be obtained through 3D affine transformation.

[0113] 3D affine transformation is a spatial transformation used to describe linear mapping and displacement between three-dimensional point sets. Affine transformation includes operations such as scaling, rotation, translation, and shearing. In the hand-eye calibration process, usually only rotation transformation (corresponding to the rotation transformation matrix) and translation transformation (corresponding to the translation transformation matrix) are involved. Affine transformation can be represented by a 4×4 matrix, acting on homogeneous coordinates, and can be expressed by the following equation:

[0114]

[0115] in, represents a 3×3 rotation transformation matrix, Represents a 3×1 translation transformation matrix.

[0116] The core problem of solving 3D affine transformation is to calculate the first affine transformation matrix based on the correspondence between the coordinates in the camera coordinate system and the coordinates in the robot coordinate system. Assume that the center of the tool ball is at coordinate P in the camera coordinate system i =[x i ,y i ,z i ,1], the corresponding coordinate P in the robot coordinate system i ′=[x i ′,y i ′,z i ′,1], then the following affine transformation equation holds:

[0117]

[0118] For each point pair (P i ,P i ′), substituting the coordinate values ​​and expanding them, we can get the following equations:

[0119] x i ′=r 11 x i +r 12 y i +r 13z i +t x

[0120] y i ′=r 21 x i +r 22 y i +r 23 z i +t y

[0121] z i ′=r 31 x i +r 32 y i +r 33 z i +t z

[0122] Assume there are n pairs of points (P i ,P i ′), for n pairs of points (i = 1, 2, ..., n), the equation can be transformed into the following matrix form:

[0123]

[0124] remember

[0125] M′=[r 11 r 12 r 13 t x r 21 r 22 r 23 t y r 31 r 32 r 33 t z ] T , then the above matrix can be simplified as:

[0126] P′=A*M′

[0127] M′ can be solved by the least squares method. The goal of the least squares method is to minimize the error function: The analytical solution is: M′=(A T A) -1 A T After solving M', M' can be converted back to the first affine transformation matrix of 4×4

[0128] The embodiment of the present application provides a 3D camera eye-in-hand and eye-out-of-hand calibration method based on a tool ball. By controlling the tool ball to move to multiple preset calibration points, an automated hand-eye calibration process is achieved, which reduces manual intervention and improves the efficiency and convenience of calibration. Since the tool ball is fixed on the robotic arm and its relative position to the end effector of the robotic arm remains unchanged, the coordinates of the end effector of the robotic arm in the robot coordinate system can be used as the first coordinates of the center of the tool ball in the robot coordinate system. The second coordinates of the center of the tool ball in the camera coordinate system are accurately obtained through 3D point cloud data, which can improve the accuracy of hand-eye calibration and thus improve the accuracy and stability of the robot's task execution. The first affine transformation matrix is ​​obtained based on multiple first coordinates and multiple second coordinates, making the calibration result more representative and reliable, reducing the calibration inaccuracy problem caused by single-point error, and enhancing the adaptability and robustness of the system in different environments and tasks.

[0129] There are various implementations of step S203. This embodiment of the present application provides, but is not limited to, steps A1 through A2. It is understood that the candidate 3D point cloud data at the preset calibration points acquired by the 3D camera includes both the 3D point cloud data of the tooling sphere and the 3D point cloud data of noise, so it is necessary to distinguish and obtain the 3D point cloud data of the tooling sphere. Exemplarily, step S204 includes steps A1 through A2.

[0130] Step A1: Acquire candidate 3D point cloud data at the preset calibration point through a 3D camera.

[0131] Step A2: Acquire 3D point cloud data from the candidate 3D point cloud data that matches the preset template sphere 3D point cloud model of the tool sphere.

[0132] The preset template sphere 3D point cloud model includes surface normal vectors of multiple points on the spherical surface of the tool sphere.

[0133] For example, the candidate 3D point cloud data can be cropped in the X, Y, and Z axis directions respectively to remove noisy background to improve the accuracy of subsequent point cloud matching; and then matched with the preset template sphere 3D point cloud model.

[0134] For example, the 3D point cloud data that matches the preset template sphere 3D point cloud model of the tool sphere can be located in the candidate 3D point cloud data based on the Halcon matching operator, and the 6-DOF pose of the 3D point cloud data can be output. In step S204, a 3D pose transformation can be performed based on the obtained 6-DOF pose to obtain a transformed template sphere point cloud. Based on the transformed template sphere point cloud, the 3D coordinates of the sphere center are output, and the second coordinate P of the tool sphere center in the camera coordinate system can be obtained. o-cam .

[0135] Exemplarily, the process of obtaining the preset template sphere 3D point cloud model includes the following steps A21 to A24.

[0136] Step A21: Measure the radius of the tooling sphere.

[0137] Use a measuring tool (such as a caliper) to measure the radius r of the tooling ball fixed to the robot arm.

[0138] Step A22: Create a mathematical model of the sphere of the tool ball.

[0139] Using the measured radius r, create a sphere with the same radius as the tool sphere, either programmatically or using 3D modeling software.

[0140] For example, the mathematical model of a sphere is: (xx c ) 2 +(yy c ) 2 +(zz c ) 2 =r 2 , where (x c ,y c ,z c ) are the coordinates of the center of the sphere.

[0141] In order to create a 3D point cloud model of the template sphere, a series of points can be uniformly sampled on the surface of the tool sphere. The coordinates of these points can be generated by the following parametric equation: c +r·sin(θ)·cos(φ)y=y c +r·sin(θ)·sin(φ)z=z c +r·cos(θ), where θ is the polar angle (from 0 to π) and φ is the azimuthal angle (from 0 to 2π). By sampling θ and φ with a fixed step size, the point cloud coordinates on the surface of the sphere can be generated.

[0142] Step A23: Python or other programming languages ​​can be used to uniformly sample a series of points from the spherical surface of the tool sphere and import the points into 3D modeling software to obtain a 3D point cloud of the template sphere.

[0143] If more complex processing or visualization is required, the generated point cloud data can be exported to common 3D point cloud formats (such as PLY or PCD) and then further processed in 3D modeling software (such as CloudCompare, MeshLab).

[0144] Step A24: Determine the template sphere 3D point cloud with the feature descriptor as a preset template sphere 3D point cloud model.

[0145] The following describes the process of obtaining feature descriptors. Feature descriptors include surface normal vectors and coordinates of multiple points.

[0146] The surface normal vector represents the vertical direction of the sphere surface at that point, which can provide important geometric information for point cloud matching and help determine the posture of the sphere in the scene.

[0147] The fitting method based on neighborhood points selects several points in the neighborhood of each point in the template sphere 3D point cloud (for example, select the k points closest to the point). Use these neighborhood points to fit a local plane, and calculate the normal vector of the plane as the surface normal vector of the point. The plane can be fitted by the least squares method, and the coefficients of the plane equation can be solved to obtain the normal vector. Based on the surface normal vector of each point and the coordinate information of the point, a feature descriptor that can describe the surface of the template sphere 3D point cloud is constructed. For example, the various components of the normal vector and the coordinate components can be combined into a feature vector.

[0148] The preset template sphere 3D point cloud model is used for matching operations in the scene 3D point cloud. The preset template sphere 3D point cloud model contains the key geometric features of the tool sphere, making it possible to accurately identify the tool sphere's position and posture in the scene during the matching process.

[0149] It can be understood that the first affine transformation matrix includes: a rotation transformation matrix from the camera coordinate system to the robot coordinate system, and a translation transformation matrix from the camera coordinate system to the robot coordinate system.

[0150] Since the relative position of the end effector of the manipulator and the tool ball is fixed, the tool ball is fixed on the end effector of the manipulator and its relative position with the end effector of the manipulator remains unchanged. This means that when the end effector of the manipulator drives the tool ball to move, the rotation direction of the end effector of the manipulator and the tool ball is consistent. Therefore, the coordinate P of the end effector of the manipulator is set to r-rob Approximately the coordinate P of the center of the tool ball in the robot coordinate system o-rob When , the obtained rotation transformation matrix is accurate.

[0151] Translation transformation matrix The determination of requires the accurate knowledge of the absolute position of the center of the tool ball in the robot coordinate system. However, in actual operation, it is usually difficult to directly obtain the initial precise position of the center of the tool ball in the robot coordinate system. Although the coordinates P of the end effector of the manipulator can be used to determine the initial position of the tool ball in the robot coordinate system, the initial position of the tool ball in the robot coordinate system can be directly obtained. o-rob To approximate the coordinates P of the center of the tool ball r-rob (i.e. P o-rob ≈P r-rob), but this approximation itself may have certain errors. This initial position error will accumulate and propagate in the subsequent translation transformation matrix calculation, resulting in deviations in the final translation transformation matrix.

[0152] In summary, the above Figure 2 The rotation transformation matrix calculated by the mentioned method is is accurate, the translation transformation matrix There are errors. Next, the translation transformation matrix is ​​corrected by the designed offset matrix correction method. The correction process includes the following steps B1 to B3.

[0153] Ideally, the first affine transformation matrix should be calculated based on the coordinates of the tool ball center in the camera coordinate system and the robot coordinate system. However, in reality, it is difficult to obtain the coordinates of the tool ball center in the robot coordinate system when the tool data of the tool ball center is unknown. However, considering that the tool ball is fixed on the robot end effector, the relative position of the tool ball center and the robot end effector is fixed, and considering that the coordinates of the robot end effector in the robot coordinate system are easy to obtain, in the actual calculation process, the first affine transformation matrix is ​​calculated based on the coordinates of the tool ball center in the camera coordinate system and the coordinates of the robot end effector in the robot coordinate system. Since the relative position of the tool ball center and the robot end effector is fixed, the rotation transformation matrix in the first affine transformation matrix calculated in the above way is is accurate, and the translation transformation matrix Therefore, this application designs an offset matrix correction method to correct the translation matrix. Correction to obtain accurate translation transformation matrix Finally get the accurate second affine transformation matrix

[0154] Step B1: obtaining a third coordinate of an artificial moving point in the camera coordinate system and a fourth coordinate in the robot coordinate system; the artificial moving point is a position to which the center of the tool ball of the end effector of the robot arm is manually moved.

[0155] Step B2: Obtain a standard translation transformation matrix based on the rotation transformation matrix, the third coordinate, and the fourth coordinate.

[0156] Step B3: Determine the rotation transformation matrix and the standard translation transformation matrix as a second affine transformation matrix.

[0157] Assume that the manually moved point is point A, and the coordinates P of point A in the camera's internal coordinate system are given as cam and the coordinates P in the robot coordinate system rob , then the following equation holds:

[0158]

[0159] Since the rotation transformation matrix It has been obtained through 3D affine transformation, so you only need to manually collect the coordinates P of a point A in the camera coordinate system cam and its corresponding coordinate P in the robot coordinate system rob , using the formula The exact translation transformation matrix can be obtained Then based on the formula Get the exact affine transformation matrix

[0160] It is understandable that in the related art, the end of the robotic arm needs to be manually moved at least 6 times to collect 6 or more groups of data in order to obtain the affine transformation matrix. The applicant only needs to move the end of the robotic arm once.

[0161] The accuracy of hand-eye calibration directly impacts the precision and stability of a robot's tasks. Large errors in hand-eye calibration lead to large errors in the affine transformation matrix, which can cause the robot arm to be unable to accurately identify the target object's position, further impacting classification and manipulation. Therefore, ensuring accurate hand-eye calibration is crucial, not only improving the robot's efficiency but also ensuring operational reliability and accuracy.

[0162] In summary, with the deepening application of robots in various industries, developing automated hand-eye calibration methods and improving the accuracy of hand-eye calibration have become one of the key factors in optimizing system performance and promoting the development of intelligent manufacturing.

[0163] Since the accuracy of hand-eye calibration depends on the precision of the 3D camera's internal parameters (such as focal length, principal point position, distortion coefficient, etc.), any error in the 3D camera's internal parameters will cause errors in the point cloud of the hand-eye calibration result, especially in areas at long distances or at the edges of the image. Because these point cloud errors originate from parameter errors in the 3D camera itself, most hand-eye calibration methods in the related art do not fully consider these error factors before and after hand-eye calibration. As a result, the point cloud obtained after completion of the hand-eye calibration methods in the related art is less accurate, which in turn affects the accuracy of the robot when performing tasks.

[0164] In response to the problem of low point cloud accuracy caused by camera internal parameter errors and hand-eye calibration errors, this application designs a 3D point cloud accuracy compensation method based on a tool ball. By calculating the tool data of the tool ball in the robot coordinate system, the coordinates of the center of the tool ball in the machine coordinate system can be obtained. By controlling the camera to automatically take pictures and the center fitting algorithm, the coordinates of the tool ball in the camera coordinate system can be obtained. The space corresponding to the robot coordinate system is evenly divided. First, the space is divided into several rectangular areas along the plane in the plane, and then each rectangular area is stretched in the vertical direction to form a three-dimensional sub-grid unit (the shape of the sub-grid unit is a cuboid). Multiple sub-grid units constitute a large cuboid grid. The vertex of each sub-grid unit (i.e., the grid vertex) is taken as the target position of the tool ball center movement to obtain three-dimensional compensation data. Based on the coordinates of the tool ball in the machine coordinate system, a three-dimensional global accuracy compensation algorithm based on 3D point cloud is designed to perform accuracy compensation on the calibrated point cloud.

[0165] The compensation process is described below. The process includes the following steps C1 to C6.

[0166] Step C1: Control the center of the tool ball fixed on the end effector of the robot arm to move to each grid vertex of the rectangular grid.

[0167] The rectangular grid includes multiple subgrid units, and each subgrid unit includes 8 grid vertices.

[0168] Step C2: For each mesh vertex, obtain 3D point cloud data of the center of the tool sphere located at the mesh vertex through a 3D camera.

[0169] Step C3: For each mesh vertex, based on the 3D point cloud data corresponding to the mesh vertex, obtain the fifth coordinate of the center of the tool sphere in the camera coordinate system.

[0170] The robot arm is programmed to drive the center of the tool ball to each grid vertex, and then the 3D camera is programmed to shoot and obtain a 3D point cloud. The 3D point cloud-based center fitting algorithm can be used to fit the fifth coordinate P of the tool ball center in the camera coordinate system from the 3D point cloud data obtained by shooting. o-cam .

[0171] Step C4: For each mesh vertex, obtain the standard sixth coordinate in the robot coordinate system corresponding to the center of the tool ball located at the mesh vertex.

[0172] The standard sixth coordinate is obtained based on the tool data of the tool ball center. The tool data is input into the robot to convert the coordinates of the tool ball center in the robot coordinate system. The robot arm is controlled by programming to drive the tool ball center to each grid vertex. The coordinate of the grid vertex is the standard sixth coordinate P of the tool ball center in the robot coordinate system at this moment. o-rob ′.

[0173] Step C5: Based on the second affine transformation matrix, the fifth coordinates corresponding to the multiple mesh vertices and the tool data of the tool ball, obtain the calculated sixth coordinate corresponding to each mesh vertex, and the tool data includes the fixed position and posture of the center of the tool ball relative to the end effector of the robotic arm.

[0174] By the second affine transformation matrix Set this fifth coordinate P o-cam Converted to the robot coordinate system, the formula is as follows: To obtain the sixth coordinate P o-rob The sixth coordinate is calculated as the homogeneous coordinate of the center of the tool ball in the robot coordinate system.

[0175] There is an error between the calculated sixth coordinate of the tool ball's center in the robot coordinate system and the standard sixth coordinate calculated based on the second affine transformation matrix and the sphere center fitting algorithm. This error is mainly caused by the calibration process and the camera hardware error. Therefore, the standard sixth coordinate needs to be used as the reference value to compensate for the calculated sixth coordinate.

[0176] For example, the robot arm is programmed to drive the center of the tool ball to the grid vertex of each sub-grid unit, and the 3D camera is programmed to take pictures to obtain 3D point cloud data. The coordinates of the center of the tool ball in the camera coordinate system are fitted from the 3D point cloud data, and then the coordinates are transformed by the second affine transformation matrix. After converting these coordinates to the robot coordinate system, the converted coordinates can be regarded as calculated coordinates (i.e., calculated sixth coordinates). Using the tool data of the tool ball, after the center of the tool ball reaches the grid vertex of each sub-grid unit, the coordinates of the center of the tool ball in the robot coordinate system can be read and obtained. These coordinates can be regarded as standard coordinates (i.e., standard sixth coordinates). The purpose of compensation is to compensate the calculated coordinates (i.e., calculated sixth coordinates) based on the standard coordinates (i.e., standard sixth coordinates).

[0177] Step C6: For each mesh vertex, obtain a compensation value based on the standard sixth coordinate corresponding to the mesh vertex and the calculated sixth coordinate.

[0178] It is understood that the compensation values ​​corresponding to the multiple mesh vertices can be stored in a structure array to ensure the orderliness and efficiency of data storage and provide a reliable data basis for the subsequent sub-grid unit positioning. For example, the compensation values ​​corresponding to the multiple mesh vertices can be stored in the structure array in sequence according to the position of the rectangular grid. For example, the compensation data of each mesh vertex is such as the standard sixth coordinate (X rob ,Y rob ,Z rob ) and calculate the sixth coordinate (X cam ,Y cam ,Z cam ) can be stored into the structure array.

[0179] Exemplarily, the structure array stores the correspondence between the coordinates of the mesh vertices and the compensation data.

[0180] Exemplarily, the compensation data further includes: an index value i of the row where the grid vertex is located, an index value j of the column where the grid vertex is located, and an index value k of the height of the sub-grid unit.

[0181] For example, the compensation data of each mesh vertex can be stored in a structure array by looping through the mesh vertices of the cuboid mesh.

[0182] The following describes a method for obtaining the rectangular parallelepiped mesh mentioned in the above solution. The process includes the following steps D1 to D3.

[0183] Step D1: Obtain the starting coordinates of the rectangular grid in the robot coordinate system, the ending coordinates of the rectangular grid in the robot coordinate system, the first length of the rectangular grid, the first width of the rectangular grid, and the first height of the rectangular grid.

[0184] A rectangular grid consists of multiple subgrid units.

[0185] like Figure 3 , which is a schematic diagram of a rectangular grid provided in an embodiment of the present application.

[0186] Meshing a space into cuboids typically involves dividing a three-dimensional cuboid into many smaller cubes or cuboids for use in calculations, simulations, or analysis. This type of meshing can be applied in finite element analysis, computer graphics, 3D modeling, robot path planning, and physical simulation.

[0187] like Figure 3 As shown, the starting coordinates can be (x min ,y min ,z min ), the end point coordinates can be (x max ,y max ,zmax ), the first length is L x , the first width is L y , the first height is L z . Then the following equation holds:

[0188] L x =x max -x min

[0189] L y =y max -y min

[0190] L z =z max -z min

[0191] Step D2: Obtain the first number of sub-grid units divided in the X-axis direction of the robot coordinate system, the second number of sub-grid units divided in the Y-axis direction, the third number of sub-grid units divided in the Z-axis direction, and the second length, second width, and second height of each sub-grid unit.

[0192] like Figure 3 Each smallest cuboid shown is a subgrid unit. Figure 3 This is only an example and does not limit the number of sub-grid units contained in the rectangular grid.

[0193] Assume that the first number is n x , the second number is n y , the third number is n z , the second length is Δ x , the second width is Δ y , the second height is Δ z , then the following equation holds:

[0194]

[0195] Step D3: Generate the rectangular parallelepiped mesh based on the starting coordinates, the ending coordinates, the first length, the first width, the first height, the first number, the second number, the third number, the second length, the second width, and the second height.

[0196] Mesh vertices are all mesh intersections generated within the cuboid mesh, evenly distributed along the X, Y, and Z axes.

[0197] The x-coordinates of the mesh vertices distributed along the X-axis are: r =x min +r·Δx, r=0,1,…,n x .

[0198] The y coordinates of the mesh vertices distributed along the Y axis are: s =y min +sΔy,s=0,1,…,n y .

[0199] The z coordinates of the mesh vertices distributed along the Z axis are: t =z min +t·Δz,t=0,1,…,n z .

[0200] The total number of sub-grid units is: N = (n x +1)×(n y +1)×(n z +1).

[0201] The following describes a process for acquiring the tool data of the tool ball mentioned in the above method. The process includes the following steps E1 to E3.

[0202] Step E1: Obtain the position coordinates of the flange center of the robot in the robot coordinate system.

[0203] The robot's flange center is the center point of the robot's end flange. This point is a key location in the robot's coordinate system. It is usually the origin of the robot's tool coordinate system, which is used to determine the position and posture of the end effector in space.

[0204] The position coordinates P of the robot's flange center in the robot coordinate system f-rob The pose coordinates P can be directly read by the robot. f-rob It is expressed as follows:

[0205] P f-rob =[x f-rob ,y f-rob ,z f-rob ,α,β,γ] T

[0206] Among them, x f-rob ,y f-rob ,z f-rob They represent the displacement of the flange center along the x, y, and z axes in the robot coordinate system, respectively; α, β, and γ represent the rotation angles of the flange center around the x, y, and z axes in the robot coordinate system, respectively.

[0207] Step E2: obtaining a second affine transformation matrix from the flange coordinate system to the robot coordinate system based on the posture coordinates, wherein the flange coordinate system refers to a coordinate system established with the center of the flange at the end of the robot arm as the origin.

[0208] The second affine transformation matrix from the flange coordinate system to the robot coordinate system can be calculated by the following formula

[0209] Step E3: Based on the second affine transformation matrix and the coordinates of the center of the tool sphere in the camera coordinate system, obtain the tool data of the tool sphere.

[0210] Based on the accurate affine transformation matrix from the camera coordinate system to the robot coordinate system And the coordinate P of the tool ball center in the camera coordinate system obtained based on the ball center fitting algorithm o-cam , then the coordinates P of the center of the tool ball in the camera coordinate system are o-rob Through the matrix equation Obtained by calculation. Note the homogeneous coordinate P of the tool ball center in the flange coordinate system o-f =[x o-f ,y o-f ,z o-f ,1] T , that is, the tool data of the tool ball center, P o-f This can be calculated using the following equation:

[0211] In an optional implementation, in actual application, after the robot's object to be operated reaches the eighth coordinate in the robot coordinate system, the eighth coordinate needs to be compensated to obtain the compensated ninth coordinate, and then the robot is controlled to operate according to the ninth coordinate. The specific method includes the following steps F1 to F7.

[0212] Step F1: Acquire 3D point cloud data of the object to be operated through a 3D camera.

[0213] Step F2: Obtain the seventh coordinate of each point in the 3D point cloud data of the object to be operated in the camera coordinate system.

[0214] Step F3: Based on the second affine transformation matrix and the seventh coordinate, obtain the eighth coordinate of each point in the 3D point cloud data of the object to be operated in the robot coordinate system.

[0215] Step F4: Determine that the 3D point cloud data of the object to be operated is located in a target sub-grid unit in the rectangular grid.

[0216] Exemplarily, the space where the target sub-grid unit is located includes the eighth coordinate.

[0217] Exemplarily, different points in the 3D point cloud data of the object to be operated may be located in the same target sub-grid unit, or may be located in different target sub-grid units.

[0218] It is understandable that some points in the 3D point cloud data of the object to be operated may exceed the rectangular grid. For the points exceeding the rectangular grid, the projection mapping method can be used to map these points to the outermost plane of the rectangular grid, and then calculate their compensation values.

[0219] Some points in the 3D point cloud data of the object to be operated are outside the set rectangular grid. The index values ​​(r, s, t) calculated for these points are considered out of bounds. The following methods can be used to ensure that the index is not out of bounds:

[0220] r=max(0,min(r,n x -1)),s=max(0,min(s,n y -1)),t=max(0,min(t,n z -1)).

[0221] Assume that the 3D point cloud data of the object to be operated is represented by a set P = {p i |p i =(x i ,y i ,z i ),i=1,2,…,n1}, where p i Represents any point in the 3D point cloud data, whose coordinates are (x i ,y i ,z i ), n1 represents the number of points in the 3D point cloud data. The rectangular grid is defined as a finite three-dimensional space G = {(x, y, z) | x∈ [x min ,x max ],y∈[y min ,y max ],z∈[z min ,z max ]},like Figure 3 shown.

[0222] For any point p in the 3D point cloud data of the object to be operated i , if p i ∈G, then the point can be located in a target subgrid unit S in the rectangular grid. rst In, such as Figure 4 As shown, the target subgrid unit S rst It can be expressed as follows:

[0223]

[0224] Step F5: Based on the compensation data corresponding to the eight grid vertices of the target sub-grid unit, a compensation value of each point in the 3D point cloud data of the object to be operated is obtained.

[0225] For example, based on the compensation data corresponding to the eight mesh vertices of the target sub-grid unit, an interpolation algorithm is used to calculate the compensation value of the point, thereby ensuring the smoothness and accuracy of the compensation. Finally, the original coordinate value of the point (i.e., the eighth coordinate) is added to the corresponding compensation value to achieve full compensation of the point cloud accuracy.

[0226] For example, an interpolation algorithm can be used to calculate the compensation value for each point in the target sub-grid unit based on the compensation data corresponding to each of the eight grid vertices. This ensures smoothness and accuracy of the compensation. Finally, the original coordinates of each point are added to the corresponding compensation value to achieve comprehensive compensation of the point cloud accuracy.

[0227] Assume that the 8 grid vertices of the target sub-grid unit are: P i1 ,P i2 ,P i3 ,P i4 ,P i5 ,P i6 ,P i7 ,P i8 ; This point is p i , p i The sixth coordinate of the calculation is P i-cam ; The calculated sixth coordinate of each mesh vertex is P ij-cam , the standard sixth coordinate of each mesh vertex is P ij-rob , where j = 1, 2,…8.

[0228] The purpose of point cloud accuracy compensation is to compensate for the known P i-cam , P ij-cam and P ij-rob Under the conditions of P ij-cam and P ij-rob P i-cam Compensate to obtain P i-rob .P i-cam , P ij-cam and P ij-rob The expression of is as follows: i-cam =[x i-cam ,y i-cam ,z i-cam ];P ij-cam =[x ij-cam ,y ij-cam ,z ij-cam ];P ij-rob =[x ij-rob ,y ij-rob ,z ij-rob ].

[0229] Among them, x i1-rob =x i3-rob=x i5-rob =x i7-rob , x i2-rob =x i4-rob =x i6-rob =x i8-rob ,y i1-rob =y i2-rob =y i5-rob =y i6-rob ,y i3-rob =y i4-rob =y i7-rob =y i8-rob , z i1-rob =z i2-rob =z i3-rob =z i4-rob , z i5-rob =z i6-rob =z i7-rob =z i8-rob .

[0230] Step F6: Acquire the ninth coordinate of each point in the 3D point cloud data of the object to be operated based on the eighth coordinate of each point in the 3D point cloud data of the object to be operated and the compensation value of each point in the 3D point cloud data of the object to be operated.

[0231] Step F7: Control the robot to operate on the ninth coordinate of each point in the 3D point cloud data of the object to be operated.

[0232] It is understandable that there are multiple ways to implement step F5. The embodiment of the present application provides but is not limited to the following steps G01 to G16.

[0233] The following combination Figure 4 Steps G01 to G16 are described.

[0234] Step G01: Project the point onto the lower surface of the target sub-grid unit to obtain a first point.

[0235] Assume that the 8 grid vertices of the target sub-grid unit are: P i1 ,P i2 ,P i3 ,P i4 ,P i5 ,P i6 ,P i7 ,P i8 ; This point is p i , p i The sixth coordinate of the calculation is P i-cam ; The calculated sixth coordinate of each mesh vertex is P ij-cam , the standard sixth coordinate of each mesh vertex is P ij-rob , where j = 1, 2,…8.

[0236] Combine Figure 4 It can be seen that the lower surface of the target sub-grid unit is {P i1 ,P i2 ,P i3 ,P i4} is a rectangle with endpoints; point p i Projecting onto the lower surface gives the first point P i13 , we can get the first point P i13 The calculation expression of the sixth coordinate is as follows: P i13-cam =[x i-cam ,y i-cam ,z i1-cam ].

[0237] Step G02: Project the point onto the upper surface of the target sub-grid unit to obtain a second point.

[0238] Combine Figure 4 It can be seen that the upper surface of the target sub-grid unit is {P i5 ,P i6 ,P i7 ,P i8} is a rectangle with endpoints; point p i Projecting onto the upper surface gives the second point P i14 , we can get the second point P i14 The calculation expression of the sixth coordinate is as follows: P i14-cam =[x i-cam ,y i-cam ,z i5-cam ].

[0239] Step G03: Project the first point onto the first edge of the lower surface to obtain a third point.

[0240] like Figure 4 As shown, the first side is P i1 ,P i2 The third point is P i9 .

[0241] Step G04: Project the first point onto the second side of the lower surface to obtain a fourth point, where the first side and the second side are parallel.

[0242] like Figure 4 As shown, the second side is P i3 ,P i4 The fourth point is P i10 .

[0243] Step G05: Determine the position parameters of the first point on the line segment with the third point and the fourth point as endpoints. The position parameters of the point on the line segment include the ratio of the distance between the point and one endpoint of the line segment to the length of the line segment.

[0244] For example, the position parameters corresponding to the first point are:

[0245]

[0246] Step G06: Calculate the compensation value of the third point based on the compensation data corresponding to the two mesh vertices of the first side and the position parameter of the third point on the first side.

[0247] You can point P i1 ,P i2 Perform difference calculation to get the third point P i9 The standard sixth coordinate P i9-rob And calculate the sixth coordinate P i9-cam ; The expression is as follows:

[0248] P i9-cam =[x i-cam ,y i1-cam ,z i1-cam ]

[0249]

[0250] Based on the third point P i9 The standard sixth coordinate P i9-rob And calculate the sixth coordinate P i9-cam Get the compensation value of the third point.

[0251] Step G07: Calculate the compensation value of the fourth point based on the compensation data corresponding to the two mesh vertices of the second side and the position parameter of the fourth point on the second side.

[0252] Similar to the process of obtaining the compensation value of the third point, the fourth point P i10 The standard sixth coordinate P i10-rob And calculate the sixth coordinate P i10-cam ; The expression is as follows:

[0253] P i10-cam =[x i-cam ,y i3-cam ,z i3-cam ]

[0254]

[0255] Based on the fourth point P i10 The standard sixth coordinate P i10-robAnd calculate the sixth coordinate P i10-cam Get the compensation value of the fourth point.

[0256] Step G08: Calculate the compensation value of the first point based on the compensation data corresponding to the third point, the compensation data corresponding to the fourth point, and the position parameter corresponding to the first point.

[0257] By the third point P i9 and the fourth P i10 Interpolation calculation of the first point P i13 The standard sixth coordinate P i13-rob , the expression is as follows:

[0258]

[0259] Based on the first point P i13 The standard sixth coordinate P i13-rob And calculate the sixth coordinate to obtain the compensation value of the third point.

[0260] Step G09: Project the second point onto the third side of the upper surface to obtain a fifth point.

[0261] like Figure 4 As shown, the third side is the point P i5 P i6 is the endpoint of the line segment, the fifth point is P i11 .

[0262] Step G10: Project the second point onto the fourth side of the upper surface to obtain a sixth point, wherein the third side is parallel to the fourth side and the third side is parallel to the first side.

[0263] like Figure 4 As shown, the fourth side is the point P i7 P i8 The sixth point is P i12 .

[0264] Step G11: Determine the position parameters of the second point located on a line segment with the fifth point and the sixth point as endpoints.

[0265] The position parameters corresponding to the second point are:

[0266]

[0267] Step G12: Calculate the compensation value of the fifth point based on the compensation data corresponding to the two mesh vertices of the third side and the position parameter of the fifth point on the third side.

[0268] Through point P i5 and point P i6Perform the difference algorithm to calculate the fifth point P i11 Calculate the sixth coordinate P i11-cam and the standard sixth coordinate P i11-rob :

[0269] P i11-cam =[x i-cam ,y i5-cam ,z i5-cam ]

[0270]

[0271] Based on the fifth point P i11 Calculate the sixth coordinate P i11-cam and the standard sixth coordinate P i11-rob Get the compensation value of the fifth point.

[0272] Step G13: Calculate the compensation value of the sixth point based on the compensation data corresponding to the two mesh vertices of the fourth side and the position parameter of the sixth point on the fourth side.

[0273] Through point P i7 and point P i8 Perform the difference algorithm to calculate the sixth point P i12 Calculate the sixth coordinate P i12-cam and the standard sixth coordinate P i12-rob :

[0274] P i12-cam =[x i-cam ,y i7-cam ,z i7-cam ]

[0275]

[0276] Based on the sixth point P i12 Calculate the sixth coordinate P i12-cam and the standard sixth coordinate P i12-rob Get the compensation value of the sixth point.

[0277] Step G14: Calculating a compensation value of the second point based on the compensation data corresponding to the fifth point, the compensation data corresponding to the sixth point, and the position parameter corresponding to the second point.

[0278] Second point P i14 The corresponding positional parameters are:

[0279]

[0280] Interpolate the fifth and sixth points to get the compensation value of the second point. The standard sixth coordinate P of the second point is obtained by the following formula i14-rob :

[0281]

[0282] A compensation value of the second point is obtained based on the standard sixth coordinate and the calculated sixth coordinate of the second point.

[0283] A compensation value of the first point is obtained based on the standard sixth coordinate and the calculated sixth coordinate of the first point.

[0284] Step G15: Obtain the position parameters of the point on the line segment with the first point and the second point as endpoints.

[0285] The location parameters of this point include:

[0286]

[0287] Step G16: Obtaining a compensation value of the point based on the compensation data of the first point, the compensation data of the second point, and the position parameter corresponding to the point.

[0288] Click p i Standard sixth coordinate P i-rob Through point P i13 and P i14 The interpolation calculation is obtained, and the calculation expression is as follows:

[0289]

[0290] Based on point p i Standard sixth coordinate P i-rob And calculate the sixth coordinate to obtain the compensation value of the point.

[0291] The embodiment of the present application provides a 3D camera eye outside the hand automatic hand-eye calibration method and accuracy compensation method based on a tool ball. The main process of the present application is as follows: Figure 5 As shown in the figure, the 3D camera is first automatically calibrated with the hand and eye based on the tool sphere, and then the data of the 3D global point cloud accuracy compensation is obtained based on the tool sphere. Finally, based on the obtained compensation data, the calibrated 3D point cloud is accuracy compensated using the 3D global point cloud accuracy compensation algorithm based on the tool sphere.

[0292] The above describes a tool ball-based 3D camera eye-in-hand and eye-out-of-hand calibration method provided in an embodiment of the present application. The following describes a device for executing the tool ball-based 3D camera eye-in-hand and eye-out-of-hand calibration method.

[0293] See also Figure 6 , Figure 6 This is a schematic diagram of the structure of a 3D camera eye-in-hand hand-eye calibration device based on a tool ball provided in an embodiment of the present application. Figure 6As shown, the tool ball-based 3D camera eye-in-hand and eye-out-of-hand calibration device is applied to a robot, wherein the robot includes a robotic arm and a 3D camera, and the tool ball is fixed on the end effector of the robotic arm, and includes:

[0294] A first control movement module 601 is used to control the tool ball to move to a plurality of preset calibration points respectively;

[0295] A first determining module 602 is configured to determine, for each of the preset calibration points, the coordinates of the preset calibration point in the robot coordinate system as the first coordinates of the center of the tool ball in the robot coordinate system;

[0296] A first acquisition module 603 is configured to acquire, for each of the preset calibration points, 3D point cloud data of the tooling sphere located at the preset calibration point using a 3D camera;

[0297] A second acquisition module 604 is configured to acquire, for each of the preset calibration points, a second coordinate of the center of the tool sphere in the camera coordinate system based on the 3D point cloud data corresponding to the preset calibration point;

[0298] The third acquisition module 605 is configured to obtain a first affine transformation matrix based on the first coordinates corresponding to the plurality of preset calibration points and the second coordinates corresponding to the plurality of preset calibration points.

[0299] In an optional implementation, the first acquisition module includes:

[0300] A first acquiring unit, configured to acquire candidate 3D point cloud data at the preset calibration point through a 3D camera;

[0301] The second acquisition unit is used to acquire 3D point cloud data in the candidate 3D point cloud data that matches the preset template sphere 3D point cloud model of the tool sphere, and the preset template sphere 3D point cloud model includes surface normal vectors of multiple points on the spherical surface of the tool sphere.

[0302] In an optional implementation, the first affine transformation matrix includes: a rotation transformation matrix from the camera coordinate system to the robot coordinate system, a translation transformation matrix from the camera coordinate system to the robot coordinate system, and further includes:

[0303] A fourth acquisition module is configured to acquire a third coordinate of an artificially moved point in the camera coordinate system and a fourth coordinate in the robot coordinate system; the artificially moved point is a position to which the center of the tool ball of the end effector of the robot arm is manually moved;

[0304] a fifth acquisition module, configured to obtain a standard translation transformation matrix based on the rotation transformation matrix, the third coordinate, and the fourth coordinate;

[0305] The second determining module is used to determine that the rotation transformation matrix and the standard translation transformation matrix are a second affine transformation matrix.

[0306] In an optional implementation, the method further includes:

[0307] A second control movement module is used to control the center of the tool ball fixed on the end effector of the robot arm to move to each grid vertex of the rectangular grid;

[0308] a sixth acquisition module, configured to acquire, for each mesh vertex, 3D point cloud data of the center of the tool sphere located at the mesh vertex through a 3D camera;

[0309] a seventh acquisition module, configured to acquire, for each mesh vertex, a fifth coordinate of the center of the tool sphere in a camera coordinate system based on the 3D point cloud data corresponding to the mesh vertex;

[0310] an eighth acquisition module, configured to acquire, for each mesh vertex, a standard sixth coordinate corresponding to the center of the tool ball located at the mesh vertex in the robot coordinate system;

[0311] a ninth acquisition module, configured to acquire a calculated sixth coordinate corresponding to each mesh vertex based on the second affine transformation matrix, the fifth coordinates corresponding to each of the mesh vertices, and tool data of the tool sphere, wherein the tool data includes a fixed position and posture of a center of the tool sphere relative to an end effector of the robotic arm;

[0312] A tenth acquisition module is configured to acquire a compensation value for each mesh vertex based on the standard sixth coordinate and the calculated sixth coordinate corresponding to the mesh vertex.

[0313] In an optional implementation, the rectangular parallelepiped mesh includes a plurality of sub-mesh units, each sub-mesh unit includes 8 mesh vertices, and further includes:

[0314] Acquire 3D point cloud data of the object to be operated through a 3D camera;

[0315] an eleventh obtaining module, configured to obtain the seventh coordinate of each point in the 3D point cloud data of the object to be operated in the camera coordinate system;

[0316] a twelfth acquisition module, configured to acquire, based on the second affine transformation matrix and the seventh coordinate, an eighth coordinate of each point in the 3D point cloud data of the object to be operated in the robot coordinate system;

[0317] A third determining module is used to determine that the 3D point cloud data of the object to be operated is located in a target sub-grid unit in the rectangular grid;

[0318] a thirteenth acquisition module, configured to obtain, for each point in the 3D point cloud data of the object to be operated, a compensation value of the point based on the compensation data corresponding to the eight grid vertices of the target subgrid unit;

[0319] a fourteenth acquisition module, configured to acquire a ninth coordinate of each point in the 3D point cloud data of the object to be operated based on the eighth coordinate of each point in the 3D point cloud data of the object to be operated and the compensation value of each point in the 3D point cloud data of the object to be operated;

[0320] The control operation module is used to control the robot to operate on the ninth coordinate of each point in the 3D point cloud data of the object to be operated.

[0321] In an optional implementation, the thirteenth obtaining module includes:

[0322] a first projection unit, configured to project the point onto a lower surface of the target subgrid unit to obtain a first point;

[0323] a second projection unit, configured to project the point onto an upper surface of the target subgrid unit to obtain a second point;

[0324] a third projection unit, configured to project the first point onto the first edge of the lower surface to obtain a third point;

[0325] a fourth projection unit, configured to project the first point onto the second side of the lower surface to obtain a fourth point, wherein the first side is parallel to the second side;

[0326] a first determining unit, configured to determine a position parameter of the first point located on a line segment having the third point and the fourth point as endpoints, the position parameter of the point located on the line segment comprising a ratio of a distance between the point and one endpoint of the line segment to a length of the line segment;

[0327] a first calculation unit, configured to calculate a compensation value of the third point based on compensation data corresponding to two mesh vertices of the first side and a position parameter of the third point on the first side;

[0328] a second calculation unit, configured to calculate a compensation value of the fourth point based on the compensation data corresponding to the two mesh vertices of the second side and a position parameter of the fourth point on the second side;

[0329] a third calculating unit, configured to calculate a compensation value of the first point based on the compensation data corresponding to the third point, the compensation data corresponding to the fourth point, and the position parameter corresponding to the first point;

[0330] a fifth projection unit, configured to project the second point onto a third side of the upper surface to obtain a fifth point;

[0331] a sixth projection unit, configured to project the second point onto the fourth side of the upper surface to obtain a sixth point, wherein the third side is parallel to the fourth side and the third side is parallel to the first side;

[0332] a second determining unit, configured to determine a position parameter of the second point located on a line segment having the fifth point and the sixth point as endpoints;

[0333] a fourth calculation unit, configured to calculate a compensation value of the fifth point based on the compensation data corresponding to the two mesh vertices of the third side and a position parameter of the fifth point located on the third side;

[0334] a fifth calculation unit, configured to calculate a compensation value of the sixth point based on the compensation data corresponding to the two mesh vertices of the fourth side and a position parameter of the sixth point located on the fourth side;

[0335] a sixth calculating unit, configured to calculate a compensation value of the second point based on the compensation data corresponding to the fifth point, the compensation data corresponding to the sixth point, and the position parameter corresponding to the second point;

[0336] a third acquiring unit, configured to acquire a position parameter of the point located on a line segment having the first point and the second point as endpoints;

[0337] The fourth acquiring unit is configured to acquire a compensation value of the point based on the compensation data of the first point, the compensation data of the second point, and a position parameter corresponding to the point.

[0338] In an optional implementation, the method further includes:

[0339] A fifteenth acquisition module is used to obtain the position coordinates of the flange center of the robot in the robot coordinate system;

[0340] a sixteenth acquisition module, configured to acquire a second affine transformation matrix from a flange coordinate system to a robot coordinate system based on the pose coordinates, wherein the flange coordinate system refers to a coordinate system established with the center of the flange at the end of the robotic arm as the origin;

[0341] A seventeenth acquisition module is used to obtain the tool data of the tool sphere based on the second affine transformation matrix and the coordinates of the center of the tool sphere in the camera coordinate system.

[0342] In an optional implementation, the method further includes:

[0343] An eighteenth acquisition module is configured to acquire a starting coordinate of the rectangular grid in the robot coordinate system, an ending coordinate of the rectangular grid in the robot coordinate system, a first length of the rectangular grid, a first width of the rectangular grid, and a first height of the rectangular grid;

[0344] A nineteenth acquisition module is configured to acquire a first number of sub-grid units divided in the X-axis direction of the robot coordinate system, a second number of sub-grid units divided in the Y-axis direction, a third number of sub-grid units divided in the Z-axis direction, and a second length, a second width, and a second height of each sub-grid unit;

[0345] A generation module is used to generate the rectangular parallelepiped grid based on the starting coordinates, the ending coordinates, the first length, the first width, the first height, the first number, the second number, the third number, the second length, the second width and the second height.

[0346] An electronic device is also provided in an embodiment of the present application. Figure 7 , which shows a schematic diagram of the structure of an electronic device suitable for implementing the embodiments of the present application. The electronic device in the embodiments of the present application may include but is not limited to fixed terminals such as mobile phones, laptops, PDAs (personal digital assistants), PADs (tablet computers), desktop computers, etc. Figure 7 The electronic device shown is merely an example and should not limit the functions and scope of use of the embodiments of the present application.

[0347] like Figure 7 As shown, the electronic device may include a processing device (e.g., a central processing unit, a graphics processing unit, etc.) 701, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 702 or a program loaded from a storage device 708 into a random access memory (RAM) 703. When the electronic device is powered on, the RAM 703 also stores various programs and data required for the operation of the electronic device. The processing device 701, the ROM 702, and the RAM 703 are connected to each other via a bus 704. An input / output (I / O) interface 705 is also connected to the bus 704.

[0348] Typically, the following devices may be connected to the I / O interface 705: an input device 706 including, for example, a touch screen, a touchpad, a keyboard, a mouse, a camera, a microphone, an accelerometer, a gyroscope, etc.; an output device 707 including, for example, a liquid crystal display (LCD), a speaker, a vibrator, etc.; a storage device 708 including, for example, a memory card, a hard disk, etc.; and a communication device 709. The communication device 709 may allow the electronic device to communicate with other devices wirelessly or by wire to exchange data. Although Figure 7The electronic device is shown with various devices, but it should be understood that it is not required to implement or possess all of the devices shown. More or fewer devices may be implemented or possessed instead.

[0349] An embodiment of the present application also provides a computer program product including computer-readable instructions. When the computer-readable instructions are executed on an electronic device, the electronic device implements any one of the tool ball-based 3D camera eye-in-hand and eye-out-of-hand calibration methods provided in the embodiments of the present application.

[0350] A computer-readable storage medium is also provided in an embodiment of the present application. The storage medium carries one or more computer programs. When the one or more computer programs are executed by an electronic device, the electronic device can implement any one of the tool ball-based 3D camera eye-in-hand and eye-out-of-hand calibration methods provided in the embodiments of the present application.

[0351] It should also be noted that the device embodiments described above are merely illustrative, wherein the units described as separate components may or may not be physically separate, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed across multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the present embodiment. In addition, in the drawings of the device embodiments provided in this application, the connection relationship between the modules indicates that there is a communication connection between them, which can be specifically implemented as one or more communication buses or signal lines.

[0352] Through the description of the above embodiments, those skilled in the art can clearly understand that the present application can be implemented by means of software plus necessary general hardware, and of course can also be implemented by special hardware including application-specific integrated circuits, special CPUs, special memories, special components, etc. In general, all functions performed by computer programs can be easily implemented with corresponding hardware, and the specific hardware structures used to implement the same function can also be diverse, such as analog circuits, digital circuits or special circuits, etc. However, for the present application, software program implementation is a better implementation method in most cases. Based on this understanding, the technical solution of the present application is essentially or the part that contributes to the prior art can be embodied in the form of a software product, which is stored in a readable storage medium, such as a computer's floppy disk, USB flash drive, mobile hard disk, ROM, RAM, magnetic disk or optical disk, etc., and includes a number of instructions to enable a computer device (which can be a personal computer, training equipment, or network equipment, etc.) to execute the methods described in each embodiment of the present application.

[0353] In the above embodiments, all or part of the embodiments may be implemented by software, hardware, firmware, or any combination thereof. When implemented by software, all or part of the embodiments may be implemented in the form of a computer program product.

[0354] The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, the process or function described in the embodiment of the present application is generated in whole or in part. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable devices. The computer instructions can be stored in a computer-readable storage medium, or transmitted from one computer-readable storage medium to another computer-readable storage medium. For example, the computer instructions can be transmitted from a website, a computer, a training device or a data center by wired (e.g., coaxial cable, optical fiber, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) mode to another website, a computer, a training device or a data center. The computer-readable storage medium can be any available medium that a computer can store or a data storage device such as a training device, a data center, etc. that includes one or more available media integrations. The available medium can be a magnetic medium, (e.g., a floppy disk, a hard disk, a tape), an optical medium (e.g., a DVD), or a semiconductor medium (e.g., a solid-state drive (SSD)).

Claims

1. A 3D camera eye-in-hand calibration method based on a tool ball, characterized in that: Applied to a robot, the robot includes a robotic arm and a 3D camera, the tool ball is fixed on the end effector of the robotic arm, and includes: Controlling the tool ball to move to a plurality of preset calibration points respectively; For each of the preset calibration points, determining the coordinates of the preset calibration point in the robot coordinate system as the first coordinates of the center of the tool ball in the robot coordinate system; For each of the preset calibration points, acquiring 3D point cloud data of the tooling sphere located at the preset calibration point through a 3D camera; For each of the preset calibration points, based on the 3D point cloud data corresponding to the preset calibration point, obtaining a second coordinate of the center of the tool sphere in the camera coordinate system; A first affine transformation matrix is ​​obtained based on the first coordinates corresponding to the plurality of preset calibration points and the second coordinates corresponding to the plurality of preset calibration points.

2. The tool sphere-based 3D camera eye-in-hand calibration method according to claim 1, characterized in that: The acquiring of 3D point cloud data of the tool sphere located at the preset calibration point by a 3D camera includes: Acquire candidate 3D point cloud data at the preset calibration point through a 3D camera; 3D point cloud data that matches a preset template sphere 3D point cloud model of the tool sphere is obtained from the candidate 3D point cloud data, where the preset template sphere 3D point cloud model includes surface normal vectors of multiple points on the spherical surface of the tool sphere.

3. The tool sphere-based 3D camera eye-in-hand and eye-out-of-hand calibration method according to any one of claims 1 to 2, characterized in that: The first affine transformation matrix includes: a rotation transformation matrix from the camera coordinate system to the robot coordinate system, a translation transformation matrix from the camera coordinate system to the robot coordinate system, and further includes: Obtaining a third coordinate of an artificially moved point in the camera coordinate system and a fourth coordinate in the robot coordinate system; the artificially moved point is a position to which the center of the tool ball of the end effector of the robotic arm is manually moved; Obtain a standard translation transformation matrix based on the rotation transformation matrix, the third coordinate, and the fourth coordinate; The rotation transformation matrix and the standard translation transformation matrix are determined to be a second affine transformation matrix.

4. The tool sphere-based 3D camera eye-in-hand and eye-out-of-hand calibration method according to claim 3, characterized in that: Also includes: Controlling the center of the tool ball fixed on the end effector of the robotic arm to move to each grid vertex of the rectangular grid; For each mesh vertex, acquiring 3D point cloud data of the center of the tool sphere located at the mesh vertex through a 3D camera; For each mesh vertex, based on the 3D point cloud data corresponding to the mesh vertex, obtaining the fifth coordinate of the center of the tool sphere in the camera coordinate system; For each mesh vertex, obtaining a standard sixth coordinate in the robot coordinate system corresponding to the center of the tool ball located at the mesh vertex; Obtaining calculated sixth coordinates corresponding to the plurality of mesh vertices based on the second affine transformation matrix, the fifth coordinates corresponding to the plurality of mesh vertices, and tool data of the tool sphere, wherein the tool data includes a fixed position and posture of the center of the tool sphere relative to the end effector of the robot arm; For each of the mesh vertices, a compensation value is obtained based on the standard sixth coordinate corresponding to the mesh vertex and the calculated sixth coordinate.

5. The 3D camera eye-in-hand and eye-out-of-hand calibration method based on a tool ball according to claim 4, characterized in that: The cuboid grid includes a plurality of subgrid units, each of which includes 8 grid vertices, and further includes: Acquire 3D point cloud data of the object to be operated through a 3D camera; Obtaining the seventh coordinate of each point in the 3D point cloud data of the object to be operated in the camera coordinate system; Based on the second affine transformation matrix and the seventh coordinate, obtaining an eighth coordinate of each point in the 3D point cloud data of the object to be operated in the robot coordinate system; Determine that the 3D point cloud data of the object to be operated is located in a target subgrid unit in the rectangular parallelepiped grid; For each point in the 3D point cloud data of the object to be operated, obtaining a compensation value for the point based on compensation data corresponding to eight mesh vertices of the target sub-mesh unit, wherein the compensation data for the mesh vertex includes a standard sixth coordinate and a calculated sixth coordinate corresponding to the mesh vertex; acquiring a ninth coordinate of each point in the 3D point cloud data of the object to be operated based on the eighth coordinate of each point in the 3D point cloud data of the object to be operated and the compensation value of each point in the 3D point cloud data of the object to be operated; The robot is controlled to operate on the ninth coordinate of each point in the 3D point cloud data of the object to be operated.

6. The tool sphere-based 3D camera eye-in-hand and eye-out-of-hand calibration method according to claim 5, characterized in that: The obtaining of the compensation value of the point based on the compensation data corresponding to the eight grid vertices of the target subgrid unit includes: Projecting the point onto the lower surface of the target subgrid unit to obtain a first point; Projecting the point onto the upper surface of the target subgrid unit to obtain a second point; Projecting the first point onto the first edge of the lower surface to obtain a third point; Projecting the first point onto the second side of the lower surface to obtain a fourth point, where the first side is parallel to the second side; Determining position parameters of the first point on a line segment with the third point and the fourth point as endpoints, where the position parameters of the point on the line segment include a ratio of a distance between the point and one endpoint of the line segment to a length of the line segment; Calculating a compensation value of the third point based on compensation data corresponding to two mesh vertices of the first side and a position parameter of the third point on the first side; Calculating a compensation value of the fourth point based on compensation data corresponding to two mesh vertices of the second side and a position parameter of the fourth point on the second side; calculating a compensation value of the first point based on the compensation data corresponding to the third point, the compensation data corresponding to the fourth point, and the position parameter corresponding to the first point; Projecting the second point onto the third side of the upper surface to obtain a fifth point; Projecting the second point onto the fourth side of the upper surface to obtain a sixth point, where the third side is parallel to the fourth side and the third side is parallel to the first side; determining position parameters of the second point located on a line segment having the fifth point and the sixth point as endpoints; Calculating a compensation value of the fifth point based on the compensation data corresponding to the two mesh vertices of the third side and a position parameter of the fifth point located on the third side; Calculating a compensation value of the sixth point based on compensation data corresponding to two mesh vertices of the fourth side and a position parameter of the sixth point located on the fourth side; calculating a compensation value of the second point based on the compensation data corresponding to the fifth point, the compensation data corresponding to the sixth point, and the position parameter corresponding to the second point; Obtaining position parameters of the point located on a line segment with the first point and the second point as endpoints; A compensation value of the point is obtained based on the compensation data of the first point, the compensation data of the second point, and a position parameter corresponding to the point.

7. The tool sphere-based 3D camera eye-in-hand and eye-out-of-hand calibration method according to any one of claims 4 to 6, characterized in that: The method for acquiring the tool data of the tool ball includes: Obtaining the position coordinates of the flange center of the robot in the robot coordinate system; Obtaining a second affine transformation matrix from a flange coordinate system to a robot coordinate system based on the pose coordinates, wherein the flange coordinate system refers to a coordinate system established with the center of the flange at the end of the robotic arm as the origin; The tool data of the tool sphere is obtained based on the second affine transformation matrix and the coordinates of the center of the tool sphere in the camera coordinate system.

8. The tool sphere-based 3D camera eye-in-hand and eye-out-of-hand calibration method according to any one of claims 4 to 6, characterized in that: The rectangular parallelepiped grid includes a plurality of sub-grid units, and the method for dividing the rectangular parallelepiped grid includes: Obtaining the starting coordinates of the rectangular grid in the robot coordinate system, the ending coordinates of the rectangular grid in the robot coordinate system, the first length of the rectangular grid, the first width of the rectangular grid, and the first height of the rectangular grid; Obtain a first number of subgrid units divided in the X-axis direction of the robot coordinate system, a second number of subgrid units divided in the Y-axis direction, a third number of subgrid units divided in the Z-axis direction, and a second length, a second width, and a second height of each subgrid unit; The rectangular parallelepiped mesh is generated based on the start coordinates, the end coordinates, the first length, the first width, the first height, the first number, the second number, the third number, the second length, the second width, and the second height.

9. A 3D camera eye-in-hand and eye-out-hand calibration device based on a tool ball, characterized in that: Applied to a robot, the robot includes a robotic arm and a 3D camera, the tool ball is fixed on the end effector of the robotic arm, and includes: A first control movement module is used to control the tool ball to move to a plurality of preset calibration points respectively; a first determining module, configured to determine, for each of the preset calibration points, a coordinate of the preset calibration point in the robot coordinate system as a first coordinate of the center of the tool ball in the robot coordinate system; A first acquisition module is configured to acquire, for each of the preset calibration points, 3D point cloud data of the tooling sphere located at the preset calibration point through a 3D camera; A second acquisition module is configured to acquire, for each of the preset calibration points, a second coordinate of the center of the tool sphere in the camera coordinate system based on the 3D point cloud data corresponding to the preset calibration point; The third acquisition module is used to obtain a first affine transformation matrix based on the first coordinates corresponding to the multiple preset calibration points and the second coordinates corresponding to the multiple preset calibration points.

10. A computer program product, characterized in that The method comprises computer-readable instructions, which, when executed on an electronic device, enable the electronic device to implement the tool sphere-based 3D camera eye-in-hand and eye-out-of-hand calibration method as claimed in any one of claims 1 to 8.

11. An electronic device, characterized in that: comprising at least one processor and a memory connected to the processor, wherein: The memory is used to store computer programs; The processor is used to execute the computer program so that the electronic device can implement the tool sphere-based 3D camera eye-in-hand and eye-out-of-hand calibration method as described in any one of claims 1 to 8.

12. A computer storage medium, characterized in that The storage medium carries one or more computer programs, which, when executed by an electronic device, enable the electronic device to implement the tool ball-based 3D camera eye-outside-hand hand-eye calibration method as described in any one of claims 1 to 8.

Images