Mechanical hand calibration and spatial angle measurement method suitable for tilt camera and robot

Abstract

The application discloses a mechanical hand calibration and space angle measurement method suitable for a tilt camera and a robot, and comprises the following steps: controlling a mechanical hand end to move along a preset calibration plane to different positions for multiple times, recording the mechanical hand end coordinates after each movement, and identifying and recording the pixel coordinates of all calibration points on a calibration plate image shot by a camera through an image processing algorithm. According to the mechanical hand end coordinates and the pixel coordinates of all calibration points on the calibration plate, a projection transformation matrix between the mechanical hand end coordinates and the pixel coordinates of the center points of all calibration points on the calibration plate is calculated by using a least square method. The camera intrinsic parameter matrix is applied to decompose each projection transformation matrix, and the space angle between the camera imaging plane and the mechanical hand end movement plane, or the space angle between the camera imaging plane, the mechanical hand end movement plane and the calibration plate plane is calculated. The calibration process of the application is simple, the result is high in precision, and the calibration error caused by the camera installation angle can be eliminated.

Description

Technical Field

[0001] This invention relates to the field of industrial robot vision calibration technology, and in particular to a method for calibrating a manipulator and measuring spatial angles using a tilt camera. Background Technology

[0002] Industrial robots can improve production efficiency and processing precision in manufacturing and are widely used in many industries such as machinery, electronics, automotive, and metallurgy. One of the most basic applications of industrial robots is to move a target object from a certain position angle to a new position angle. In this process, industrial robots need to solve two problems: First, the initial position and the target position of the target object may be offset; if the offset cannot be corrected, the movement will be inaccurate. Second, the target object may rotate, and many operations on the object require the manipulator to adapt to the current angle of the object beforehand; otherwise, it will affect the accuracy or even cause the operation to fail. To solve these two problems, in practical applications, the rotatable rotating mechanism or motion platform at the end effector of the industrial robot's manipulator is often used to manipulate the target object. Before and during the operation, a camera is used to photograph the target object, and the position and angle of the object are obtained through image processing. Then, the rotating mechanism at the end effector of the manipulator is controlled to translate and rotate, thereby achieving precise positioning and movement of the object. For the calibration plate image taken by the camera installed at a certain position, there is a corresponding relationship between the coordinate angle of the object and the actual position angle of the manipulator when manipulating the object. The process and algorithms used to determine and apply this correspondence are called calibrating the relationship between the camera and the robot, or simply robot calibration or hand-eye calibration.

[0003] Machine vision has two fundamental applications in the movement of objects by robotic arms: the first is to obtain the relative position of the target object at the end effector of the robotic arm, in which case the camera is often mounted in a fixed position; the second is to obtain the absolute position of the target object, in which case the camera is often mounted on the robotic arm itself. These two scenarios can also be simply referred to as "eye outside the hand" and "eye on the hand." Many relatively complex applications are essentially organic combinations of these two scenarios. For example, when two robotic arms exchange target objects, for each robotic arm, the object on the other robotic arm can be regarded as a fixed object whose absolute position has been obtained. Therefore, as long as the calibration method can calibrate both scenarios, it can meet most of the application requirements of robotic arms.

[0004] The current mainstream calibration methods can be roughly divided into three categories:

[0005] The first type of method physically decouples the translation and rotation of the robot's end effector, thereby reducing the computational complexity in calibration and application. One scenario involves precise mounting to minimize the error between the center of the camera or target object and the rotation center of the robot's end effector. Another scenario is where the robot's end effector has no rotational function or requires rotation. In this type of method, there are only independent translational and rotational transformations between the robot's coordinates and the object's spatial coordinates. In application, it is only necessary to calculate the results of the two transformations separately and then directly add them together.

[0006] The second type of method is based on the fixing method and relative coordinates of the camera / manipulator at the end effector of the robotic arm, and performs unified matrix operations on the absolute coordinates of the robotic arm and the object. During calibration, the end effector coordinate angles after each translation and rotation of the robotic arm and the coordinates of the object in the camera image are collected. Then, all data are uniformly input into the operation matrix for solution. For example, when the robotic arm, camera, and object are absolutely parallel, the rotation angle can be converted into a rotation matrix, the matrix equation can be directly constructed, and the complete coefficient matrix can be calculated by substituting the data. In application, the original data can be directly substituted into the matrix equation to obtain the target data. The advantage of this type of method is that the calculation process is purely mathematical, which is more efficient and more stable. The disadvantage is that the construction of the operation matrix is ​​more complex, and it is more troublesome to modify and apply specific physical parameters.

[0007] The third type of method requires detailed analysis of every coordinate of the robotic arm, camera, and object during the imaging process, deriving calculations based on physical meanings such as spatial geometric relationships. For example, when the object rotates at the end effector of the robotic arm, every coordinate of the object on the image is recorded. Then, these coordinates are fitted with a circle to find the coordinates of the circle's center on the image, i.e., the pixel coordinates of the center of rotation of the robotic arm's end effector in the image. The advantage of this type of method is that the physical meaning of each parameter is clearer, and various methods can be flexibly applied during the calculation process. The disadvantage is that it is necessary to clearly understand the actual meaning and differences of various parameters to avoid errors or even singularities caused by unsuitable mathematical expressions.

[0008] Most current mainstream calibration methods require the camera's imaging plane to be parallel to the calibration plane to eliminate calculation errors caused by camera tilt distortion. The potential problems with this approach are: first, it places high demands on the installation accuracy of the camera and calibration object, making it difficult to eliminate the impact of installation and image processing errors on the accuracy of the calibration results; second, it limits the applicability of the calibration method, necessitating approximate corrections to the calculation results in some tilted camera applications to meet practical requirements. Summary of the Invention

[0009] The technical problem to be solved by the present invention is to address at least one defect of the related technologies mentioned in the background: the current mainstream calibration methods limit the scope of application, most of which require the camera imaging plane to be parallel to the calibration plane, making it difficult to eliminate the influence of installation errors and image processing errors on the accuracy of the calibration results. The present invention provides a robotic arm calibration and spatial angle measurement method suitable for tilt cameras.

[0010] The technical solution adopted by this invention to solve its technical problem is: to construct a method for calibrating a robotic arm and measuring spatial angles suitable for tilt cameras, comprising the following steps:

[0011] Translation recording steps: Control the end effector of the robot arm to translate to different positions along the preset calibration plane multiple times, record the coordinates of the end effector of the robot arm after each translation, and use image processing algorithms to identify and record the pixel coordinates of all calibration points on the calibration board image captured by the camera;

[0012] Projective transformation matrix calculation steps: Based on the coordinates of the robot end point and the pixel coordinates of all calibration points on the calibration plate in the translation recording step, the projective transformation matrix between the coordinates of the robot end point and the pixel coordinates of the center point of all calibration points on the calibration plate is calculated using the least squares method;

[0013] Spatial angle calculation steps: Apply the camera intrinsic parameter matrix to decompose each of the projective transformation matrices, and calculate the spatial angle between the camera imaging plane and the robot end translation plane, or between the camera imaging plane, the robot end translation plane and the calibration plate plane.

[0014] Preferably, in the robotic arm calibration and spatial angle measurement method for tilt cameras described in this invention, the method further includes:

[0015] Rotation and translation recording steps: control the rotation mechanism of the end effector of the robot arm to rotate to different angles along the preset calibration plane multiple times and then translate it. Record the coordinates of the end effector of the robot arm after each rotation and translation. Use image processing algorithms to identify and record the pixel coordinates of all calibration points on the calibration board image captured by the camera.

[0016] Distance calculation step: Based on the coordinates of the robot end effector and the pixel coordinates of all calibration points on the calibration board in the rotation and translation recording step, the actual distance from the robot end effector to the camera imaging center is calculated using the least squares method. This yields the actual coordinates of the camera imaging center and the center points of all calibration points on the calibration board in the robot space coordinate system, thus completing the calibration of the relationship between the camera and the robot.

[0017] Preferably, in the robotic arm calibration and spatial angle measurement method for tilt cameras described in this invention, the method further includes the following step before the translation recording step:

[0018] Setting steps: Set the translation plane of the end effector and the rotation plane of the rotation mechanism of the end effector as the preset calibration plane.

[0019] Preferably, in the method for calibrating a robot arm and measuring a spatial angle for a tilting camera according to the present invention, the method is applicable to the first case: the camera is tilted and installed in a fixed position, and the calibration plate of the checkerboard pattern is tilted and installed on the end of the robot arm or on the rotating mechanism of the end of the robot arm;

[0020] The method is also applicable to a second case: the camera is tilted and mounted on the end of the robotic arm or on the rotating mechanism of the end of the robotic arm, and the calibration plate of the dot array is tilted and mounted in a fixed position.

[0021] The method is also applicable to a third case: the camera is vertically mounted on the end of the robotic arm or on the rotating mechanism of the end of the robotic arm, and the calibration plate with a single dot is horizontally mounted in a fixed position.

[0022] Preferably, in the robotic arm calibration and spatial angle measurement method for tilt cameras described in this invention, when the method is applicable to the first case, the translation recording step includes:

[0023] The robot end effector is controlled to translate freely within the preset calibration plane, and all calibration points on the calibration plate are kept within the calibration plate image captured by the camera. The coordinates of the robot end effector after each translation are recorded.

[0024] Based on the calibration board image captured by the camera, edge detection and line detection algorithms are applied to the checkerboard pattern on the calibration board, and lines at different angles are separated by clustering algorithm;

[0025] Find the intersection points of all intersecting lines, and sort all the intersection points into a calibration point array by rows and columns using the convex hull algorithm to obtain the pixel coordinates of all calibration points on the calibration board.

[0026] When the method is applicable to the second case, the translation recording step includes:

[0027] After calibration begins, the starting position of the last calibration is read from the memory, and the robot arm is controlled to move to that position.

[0028] The robotic arm end effector is controlled to sequentially translate to a position array centered on the current position, traversing all positions in the position array and returning to the center position, and the coordinates of the robotic arm end effector after each translation are recorded;

[0029] The elliptical regions after deformation of all dots on the calibration plate are obtained by the region detection algorithm. The centroid of each elliptical region is used as a temporary calibration point and sorted into a calibration point array by the convex hull algorithm.

[0030] Based on the calibration point array, calculate the horizontal and vertical external tangents of the elliptical regions in each row and column, and use the intersection of the diagonals of the external tangents of each elliptical region as the real calibration point to obtain the pixel coordinates of all calibration points on the calibration board.

[0031] When the method is applicable to the third case, the translation recording step includes:

[0032] The robotic arm end effector is controlled to translate sequentially in eight directions within the preset calibration plane. During each translation, an image processing algorithm is used to determine and control the calibration points on the calibration plate within the calibration plate image captured by the camera, and the coordinates of the robotic arm end effector after each translation are recorded.

[0033] The pixel coordinates of the centroid of the region on the calibration board are obtained by the region detection algorithm and used as the pixel coordinates of all calibration points on the calibration board.

[0034] Preferably, in the method for calibrating a manipulator and measuring spatial angles for a tilting camera according to the present invention, when the method is applicable to the first and second cases, the step of calculating the projective transformation matrix includes:

[0035] The least squares method is used to calculate the projective transformation matrix between the coordinates on the calibration plate of all calibration points and the pixel coordinates of all calibration points on the calibration plate after each translation of the robot end in the translation recording step.

[0036] The pixel coordinates of the center point of all calibration points on the calibration board are calculated using the projective transformation matrix between the coordinates on the calibration board and the pixel coordinates of all calibration points on the calibration board. This matrix is ​​then used as input data to calculate the projective transformation matrix between the coordinates of the robot end effector and the pixel coordinates of the center point of all calibration points on the calibration board using the least squares method.

[0037] When the method is applicable to the third case, the steps for calculating the projective transformation matrix include:

[0038] Using a single dot on the calibration plate as the center point of the calibration point, the least squares method is used to calculate the projective transformation matrix between the coordinates of the robot end effector and the pixel coordinates of the center point of all calibration points on the calibration plate.

[0039] Preferably, in the robotic arm calibration and spatial angle measurement method for tilt cameras described in this invention, when the method is applicable to the first case, the spatial angle calculation step includes:

[0040] The Zhang Zhengyou calibration method is used to process the pixel coordinates of all calibration points on the calibration board at different positions and the coordinates on the calibration board of all calibration points on the calibration board obtained in the translation recording step, and the camera intrinsic parameter matrix is ​​calculated.

[0041] By combining the camera intrinsic parameter matrix, the singular value decomposition method is used to decompose the projective transformation matrix between the coordinates of the robot end effector and the pixel coordinates of the center points of all calibration points on the calibration plate, and the spatial angle between the camera imaging plane and the translation plane of the robot end effector is calculated.

[0042] Then, by combining the camera intrinsic parameter matrix, the projective geometry method is used to decompose the projective transformation matrix between the coordinates on the calibration plate of all calibration points and the pixel coordinates of all calibration points on the calibration plate after each translation of the robot end effector, and the spatial angle between the camera imaging plane and the calibration plate plane is calculated.

[0043] By combining the spatial angle between the camera imaging plane and the translation plane of the robot end effector and the spatial angle between the camera imaging plane and the calibration plate plane, the spatial angle between the calibration plate plane and the translation plane of the robot end effector is calculated;

[0044] When the method is applicable to the second case, the spatial angle calculation steps include:

[0045] The camera intrinsic parameter matrix is ​​read from the memory. The projective transformation matrix between the coordinates of the robot end effector and the pixel coordinates of the center points of all calibration points on the calibration plate is decomposed using the projective geometry method based on the camera intrinsic parameter matrix. The spatial angle between the camera imaging plane and the translation plane of the robot end effector is calculated.

[0046] Then, by combining the camera intrinsic parameter matrix, the singular value decomposition method is used to decompose the projective transformation matrix between the coordinates on the calibration plate of all calibration points and the pixel coordinates of all calibration points on the calibration plate after each translation of the robot end effector, and the spatial angle between the camera imaging plane and the calibration plate plane is calculated.

[0047] When the method is applicable to the third case, the spatial angle calculation steps include:

[0048] The camera intrinsic parameter matrix is ​​read from the memory. The projective transformation matrix between the coordinates of the robot end effector and the pixel coordinates of the center points of all calibration points on the calibration plate is decomposed using the projective geometry method based on the camera intrinsic parameter matrix. The spatial angle between the camera imaging plane and the translation plane of the robot end effector is then calculated.

[0049] Preferably, in the robotic arm calibration and spatial angle measurement method for tilt cameras described in this invention, when the method is applicable to the first case, the rotation and translation recording step includes:

[0050] The robotic arm end effector is controlled to rotate freely within the preset calibration plane. After each rotation, the robotic arm end effector is controlled to translate to keep all calibration points on the calibration plate within the calibration plate image captured by the camera, and the coordinates of the robotic arm end effector after each rotation and translation are recorded.

[0051] Based on the calibration board image captured by the camera, edge detection and line detection algorithms are applied to the checkerboard pattern on the calibration board, and lines at different angles are separated by clustering algorithm;

[0052] Find the intersection points of all intersecting lines, and sort all the intersection points into a calibration point array by rows and columns using the convex hull algorithm to obtain the pixel coordinates of all calibration points on the calibration board.

[0053] When the method is applicable to the second case, the rotation and translation recording steps include:

[0054] The robotic arm end effector is controlled to rotate to both sides by a preset angle within the preset calibration plane, and then rotate back to the initial angle. After each rotation, the robotic arm end effector is controlled to translate to keep all calibration points on the calibration plate within the calibration plate image captured by the camera, and the coordinates of the robotic arm end effector after each rotation and translation are recorded.

[0055] The elliptical regions after deformation of all dots on the calibration plate are obtained by the region detection algorithm. The centroid of each elliptical region is used as a temporary calibration point and sorted into a calibration point array by the convex hull algorithm.

[0056] Based on the calibration point array, calculate the horizontal and vertical external tangents of the elliptical regions in each row and column, and use the intersection of the diagonals of the external tangents of each elliptical region as the real calibration point to obtain the pixel coordinates of all calibration points on the calibration board.

[0057] When the method is applicable to the third case, the rotation and translation recording steps include:

[0058] The robotic arm end effector is controlled to rotate to multiple different angles within the preset calibration plane. During the rotation, the pixel coordinates of the calibration point are determined in real time by the image processing algorithm. The calibration point is kept in the center of the image captured by the camera by controlling the robotic arm end effector to follow the translation.

[0059] Record the coordinates of the robot's end effector after each rotation to the target angle;

[0060] The pixel coordinates of the centroid of the region on the calibration board are obtained by the region detection algorithm and used as the pixel coordinates of all calibration points on the calibration board.

[0061] Preferably, in the robotic arm calibration and spatial angle measurement method for tilt cameras described in this invention, when the method is applicable to the first case, the distance calculation step includes:

[0062] The pixel coordinates of the center points of all calibration points on the calibration plate after each rotation and translation are calculated using the least squares method. The actual distance from the center point of all calibration points on the calibration plate to the camera imaging center is calculated by applying the projective transformation matrix between the coordinates of the robot end and the pixel coordinates of the center points of all calibration points on the calibration plate.

[0063] Then, the actual distance from the center point of all calibration points on the calibration plate to the camera imaging center is superimposed with the translation distance of each rotation and translation to calculate the actual distance from the center point of all calibration points on the calibration plate to the camera imaging center before the first rotation and translation after each rotation and translation.

[0064] The actual distance from the center point of all calibration points on the calibration plate to the camera imaging center before the first rotation and translation is obtained by processing the least squares method.

[0065] Add the actual distance from the end of the robotic arm to the imaging center of the camera during the first rotation and translation to the coordinates of the end of the robotic arm obtained in the rotation and translation recording step to obtain the absolute coordinates of the imaging center of the camera in the robotic arm's spatial coordinate system;

[0066] The absolute coordinates of the center points of all calibration points on the calibration plate are calculated based on the absolute coordinates of the camera imaging center. The coordinates of the end effector of the robot are then subtracted to obtain the fixed coordinates of the center points of all calibration points on the calibration plate on the rotating mechanism of the end effector of the robot.

[0067] When the method is applicable to the second and third cases, the distance calculation steps include:

[0068] The actual distance from the center point of all calibration points on the calibration plate to the camera imaging center is calculated by applying the projective transformation matrix between the coordinates of the robot end effector and the pixel coordinates of the center points of all calibration points on the calibration plate;

[0069] Then, after rotating the translation distance in both clockwise and counterclockwise directions according to the rotation angle, add the actual distance from the center point of all calibration points on the calibration plate to the imaging center of the camera, and calculate the two sets of actual distances from the center point of all calibration points on the calibration plate to the imaging center of the camera before the first rotation and translation after each rotation and translation.

[0070] The least squares method is used to process the two sets of actual distances from the center point of all calibration points on the calibration plate to the imaging center of the camera before the first rotation and translation, to obtain the actual distance from the end of the robot arm to the imaging center of the camera before the first rotation and translation, and to determine whether the rotation direction of the robot arm is clockwise.

[0071] Add the actual distance from the end of the robotic arm to the imaging center of the camera before the first rotation and translation to the coordinates of the end of the robotic arm obtained in the rotation and translation recording step to obtain the absolute coordinates of the imaging center of the camera in the robotic arm spatial coordinate system;

[0072] The absolute coordinates of the center points of all calibration points on the calibration plate are calculated based on the absolute coordinates of the camera's imaging center. The coordinates of the robot's end effector are then subtracted to obtain the fixed coordinates of the camera on the rotating mechanism at the end effector of the robot.

[0073] The present invention also constructs a robot including a controller, a manipulator, and a camera, wherein the controller is used to implement the manipulator calibration and spatial angle measurement method for a tilt camera as described in any of the above-described methods.

[0074] By implementing this invention, the following beneficial effects are achieved:

[0075] This invention is applicable to tilted cameras, offering a simple calibration process and high accuracy, eliminating calibration errors caused by the installation angles of the camera and calibration plate. It uses projective transformation to handle the coordinate mapping relationship between non-parallel planes and combines the camera's intrinsic parameter matrix to decompose the projective matrix, calculating the spatial angles between the camera imaging plane, the robot end effector translation plane, and the calibration plate plane. Industrial robots calibrated using this invention can adapt to more complex environments, meet more requirements, and achieve higher working accuracy. Attached Figure Description

[0076] The present invention will be further described below with reference to the accompanying drawings and embodiments. In the accompanying drawings:

[0077] Figure 1 This is a flowchart illustrating the robot arm calibration and spatial angle measurement method of the present invention applicable to tilt cameras;

[0078] Figure 2This is a side view of the first embodiment of the present invention, in which the camera is tilted and mounted in a fixed position, and the calibration plate of the checkerboard pattern is tilted and mounted on the rotating mechanism at the end of the robot arm.

[0079] Figure 3 This is a side view of the second embodiment of the present invention, in which the camera is tilted and mounted on the rotating mechanism at the end of the robotic arm, and the calibration plate of the dot array is tilted and mounted in a fixed position.

[0080] Figure 4 This is a top view of the third embodiment of the present invention, in which the camera is vertically mounted on the rotating mechanism at the end of the robotic arm, and the single-dot calibration plate is horizontally mounted in a fixed position. Detailed Implementation

[0081] To provide a clearer understanding of the technical features, objectives, and effects of the present invention, specific embodiments of the present invention will now be described in detail with reference to the accompanying drawings.

[0082] It should be noted that the flowcharts shown in the accompanying drawings are merely illustrative and do not necessarily include all content and operations / steps, nor do they necessarily have to be performed in the described order. For example, some operations / steps can be broken down, while others can be combined or partially combined; therefore, the actual execution order may change depending on the specific circumstances.

[0083] The block diagrams shown in the accompanying drawings are merely functional entities and do not necessarily correspond to physically independent entities. That is, these functional entities can be implemented in software, in one or more hardware modules or integrated circuits, or in different network and / or processor devices and / or microcontroller devices.

[0084] This invention discloses a method for calibrating a robot arm and measuring spatial angles using a tilted camera. The camera 1 and calibration plate 2 are mounted at an angle. This method is also compatible with situations where the camera 1 is vertically mounted on a rotating mechanism 4 at the end of the robot arm 3, and where the calibration plate 2 is horizontally mounted in a fixed position. For the tilted mounting of the camera 1 and calibration plate 2, this invention uses a calibration plate 2 with a high-precision calibration point array to reduce the impact of the camera 1's tilt angle on the shape distortion of the calibration plate 2 during imaging, thus reducing the accuracy of the calibration results. Each calibration point can be the center of a single shape, such as the center of a circle or rectangle, or certain elements of a complete shape, such as the intersection of lines or the corner of a rectangle. The robot arm has an end cap that can translate arbitrarily within a preset calibration plane, and a rotating mechanism 4 that rotates arbitrarily within the preset calibration plane around this end cap.

[0085] It should be noted that in the following description, the coordinates of the robot's end effector are in the robot's spatial coordinate system, the pixel coordinates are in the camera's pixel plane coordinate system, and the coordinates on the board are in the calibration board coordinate system.

[0086] First embodiment, such as Figure 1 and Figure 2 As shown, this embodiment discloses a method for calibrating a robotic arm and measuring spatial angles using a tilted camera. This method is semi-automatic. The camera 1 is tilted and mounted in a fixed position, where the calibration plate 2 can be clearly imaged within the camera 1. The checkerboard-patterned calibration plate 2 is tilted and mounted on the rotating mechanism 4 of the robotic arm's end effector 3. In other embodiments, when the robotic arm's end effector 3 only performs translational movements, the rotating mechanism 4 may not be present; that is, the checkerboard-patterned calibration plate 2 is tilted and mounted on the robotic arm's end effector 3. Therefore, when the robotic arm's end effector 3 only performs translational movements, the method includes the following steps:

[0087] Setting steps: Set the translation plane of the robot end effector 3 and the rotation plane of the rotation mechanism 4 of the robot end effector 3 as preset calibration planes.

[0088] Translation recording steps: control the end effector 3 of the robot arm to translate to different positions along the preset calibration plane multiple times, record the coordinates of the end effector of the robot arm after each translation, and use image processing algorithms to identify and record the pixel coordinates of all calibration points on the calibration board image captured by camera 1.

[0089] The projective transformation matrix calculation steps are as follows: Based on the coordinates of the robot end effector and the pixel coordinates of all calibration points on the calibration plate in the translation recording step, the projective transformation matrix between the coordinates of the robot end effector and the pixel coordinates of the center point of all calibration points on the calibration plate is calculated using the least squares method.

[0090] Steps for calculating spatial angles: Decompose each projective transformation matrix using the camera intrinsic parameter matrix, and calculate the spatial angles between the camera imaging plane, the robot end-effector translation plane, and the calibration plate plane.

[0091] When the end effector 3 of the robotic arm performs a rotational and translational motion, the method also includes the following steps:

[0092] Rotation and translation recording steps: control the robot end effector 3 to rotate to different angles along the preset calibration plane multiple times and then translate it. Record the coordinates of the robot end effector and the angle of the robot end effector rotation mechanism after each rotation and translation. The pixel coordinates of all calibration points on the calibration board are identified and recorded by the image processing algorithm of the calibration board image captured by camera 1.

[0093] Distance calculation steps: Based on the coordinates of the robot end effector and the pixel coordinates of all calibration points on the calibration board in the rotation and translation recording steps, the actual distance from the robot end effector 3 to the camera imaging center is calculated using the least squares method. This yields the actual coordinates of the camera imaging center and the center points of all calibration points on the calibration board in the robot space coordinate system, thus completing the calibration of the relationship between the camera and the robot.

[0094] Specifically, in this embodiment, the translation recording step includes:

[0095] S1-1a: Control the robotic arm end effector 3 to translate freely within a preset calibration plane, and keep all calibration points on the calibration plate within the calibration plate image captured by camera 1, recording the coordinates of the robotic arm end effector after each translation. Specifically, the robotic arm end effector 3 is controlled to translate multiple times in different directions within the preset calibration plane. After each translation, at least four valid robotic arm end effector coordinates are available in the image where all calibration points can be identified, and any three of these four coordinates are not collinear.

[0096] Specifically, the operator controls the robotic arm end effector 3 to translate, moving the calibration plate 2 to the center of the image captured by the camera 1. Then, the operator controls the robotic arm end effector 3 to translate freely within the preset calibration plane. The operator judges whether all calibration points on the calibration plate are within the image by visually displaying the real-time image. If so, the operator records the coordinates of the robotic arm end effector after each translation. If not, the operator controls the robotic arm end effector 3 to translate within the preset calibration plane, keeping all calibration points on the calibration plate within the calibration plate image captured by the camera 1.

[0097] In some other embodiments, this step also records the angle of the robot end effector rotation mechanism in the robot's spatial coordinate system, but since it is a translation, the angle remains unchanged each time.

[0098] S1-2a: The tilt distortion of camera 1 will change the scale and angle of the calibration pattern in the image, but tilt distortion is a linear projective transformation, and a straight line remains a straight line after tilt distortion. Therefore, points that can be calculated using straight lines are selected as calibration points in the calibration pattern to ensure that the pixel coordinates and the coordinates on the board represent the calibration points. Therefore, based on the calibration board image captured by camera 1, edge detection and line detection algorithms are used on the checkerboard grid on calibration board 2 to first find the straight lines in the grid, and then a clustering algorithm is used to separate the straight lines at different angles, including horizontal straight lines and vertical straight lines.

[0099] S1-3a: Find the intersection points of all intersecting lines, that is, the intersection points between grid boundary lines. Use the convex hull algorithm to sort all intersection points into a calibration point array by rows and columns to obtain the pixel coordinates of all calibration points on the calibration board.

[0100] In other embodiments, when using a calibration board 2 with other rectangular calibration point arrays, the four corner points of each region of the image are obtained through edge detection and line matching, and the intersection of their diagonals corresponds to the centroid of the rectangle on the board, i.e., the calibration point.

[0101] In this embodiment, the steps for calculating the projective transformation matrix include:

[0102] S2-1a: Calculate the projective transformation matrix between the coordinates on the calibration plate and the pixel coordinates of all calibration points on the calibration plate after each translation of the robot end effector 3 in the translation recording step using the least squares method. That is, the projective transformation matrix between the calibration plate coordinate system and the camera pixel plane coordinate system.

[0103] S2-2a: Calculate the pixel coordinates of the center point of all calibration points on the calibration board using the projective transformation matrix between the coordinates on the calibration board and the pixel coordinates of all calibration points on the calibration board. Use this as input data and calculate the projective transformation matrix between the coordinates of the robot end effector and the pixel coordinates of the center point of all calibration points on the calibration board using the least squares method. This matrix is ​​the projective transformation matrix between the robot's spatial coordinate system and the camera's pixel plane coordinate system. This matrix can transform any image distance with the actual distance.

[0104] The total number of calibration points is an array. If both rows and columns are odd, the center point is one of the calibration points. If both rows and columns are even, non-existent center points must be calculated. This embodiment uses the double least squares method to calculate the projective transformation matrix between the coordinates of the robot's end effector and the pixel coordinates of the center points of all calibration points on the calibration plate, which can reduce the impact of image processing errors on calibration accuracy.

[0105] In this embodiment, the spatial angle calculation steps include:

[0106] S3-1a: Use Zhang Zhengyou's calibration method to process the pixel coordinates of all calibration points on the calibration board at different positions and the coordinates on the calibration board of all calibration points on the calibration board obtained in the translation recording step, and calculate the camera intrinsic parameter matrix.

[0107] S3-2a: Combining the camera intrinsic parameter matrix, the singular value decomposition (SVD) method is used to decompose the projective transformation matrix between the coordinates of the robot end effector and the pixel coordinates of the center points of all calibration points on the calibration plate, and the spatial angle between the camera imaging plane and the translation plane of the robot end effector is calculated.

[0108] S3-3a: Combining the camera intrinsic parameter matrix, the projective geometry method is used to decompose the projective transformation matrix between the coordinates on the calibration plate of all calibration points and the pixel coordinates of all calibration points on the calibration plate after each translation of the robot end effector 3, and the spatial angle between the camera imaging plane and the calibration plate plane is calculated.

[0109] S3-4a: Calculate the spatial angle between the calibration plate plane and the robot end translation plane by combining the spatial angle between the camera imaging plane and the robot end translation plane and the spatial angle between the camera imaging plane and the calibration plate plane.

[0110] In this embodiment, the rotation and translation recording step includes:

[0111] S4-1a: Control the robot end effector 3 to rotate freely within the preset calibration plane. After each rotation, control the robot end effector 3 to translate so that all calibration points on the calibration plate are kept within the calibration plate image captured by camera 1, and record the coordinates of the robot end effector after each rotation and translation.

[0112] Specifically, the operator controls the robot end effector 3 to translate, moving the calibration plate 2 to the center of the image captured by the camera 1. Then, the operator controls the rotation mechanism 4 of the robot end effector 3 to rotate freely within the preset calibration plane. The operator judges whether all calibration points on the calibration plate are within the image by visually displaying the real-time image. If so, the operator records the coordinates of the robot end effector and the angle of the rotation mechanism after each rotation and translation. If not, the operator controls the robot end effector 3 to translate within the preset calibration plane, keeping all calibration points on the calibration plate within the calibration plate image captured by the camera 1.

[0113] S4-2a: The tilt distortion of camera 1 will change the scale and angle of the calibration pattern in the image, but tilt distortion is a linear projective transformation, and a straight line remains a straight line after tilt distortion. Therefore, points that can be calculated using straight lines are selected as calibration points in the calibration pattern to ensure that the pixel coordinates and the coordinates on the board represent the calibration points. Therefore, based on the calibration board image captured by camera 1, edge detection and line detection algorithms are used on the checkerboard grid on calibration board 2 to first find the straight lines in the grid, and then a clustering algorithm is used to separate the straight lines at different angles, including one type of horizontal straight lines and one type of vertical straight lines.

[0114] S4-3a: Find the intersection points of all intersecting lines, that is, the intersection points between grid boundary lines. Use the convex hull algorithm to sort all intersection points into a calibration point array by rows and columns to obtain the pixel coordinates of all calibration points on the calibration board.

[0115] Among them, the rotation mechanism 4 of the robot end 3 is controlled to rotate and translate multiple times in different directions. After rotation and translation, the effective robot end coordinates and the angles of the robot end rotation mechanism that can identify all calibration points in the image are at least three sets, and these three sets of angles are not equal to each other.

[0116] In this embodiment, the distance calculation step includes:

[0117] S5-1a: The least squares method is used to calculate the pixel coordinates of the center points of all calibration points on the calibration plate after each rotation and translation. The actual distance from the center points of all calibration points on the calibration plate to the camera imaging center is calculated by applying the projective transformation matrix between the coordinates of the robot end effector and the pixel coordinates of the center points of all calibration points on the calibration plate. Camera 1 has a principal axis, and the intersection of the principal axis and the preset calibration plane is the camera imaging center, whose pixel coordinates can be obtained from the intrinsic parameter matrix; for an ideal camera, this is the center point of the image.

[0118] S5-2a: Then, add the actual distance from the center point of all calibration points on the calibration plate to the camera imaging center to the translation distance of each rotation and translation, and calculate the actual distance from the center point of all calibration points on the calibration plate to the camera imaging center before the first rotation and translation after each rotation and translation.

[0119] S5-3a: The least squares method is used to process the actual distance from the center point of all calibration points on the calibration plate to the camera imaging center before the first rotation and translation, thus obtaining the actual distance from the robot end effector 3 to the camera imaging center before the first rotation and translation. This embodiment uses the double least squares method to reduce the impact of image processing errors on calibration accuracy.

[0120] S5-4a: Add the actual distance from the end effector 3 of the robot arm to the camera imaging center before the first rotation and translation to the coordinates of the end effector of the robot arm obtained in the rotation and translation recording step to obtain the absolute coordinates of the camera imaging center in the robot arm spatial coordinate system.

[0121] S5-5a: Calculate the absolute coordinates of the center points of all calibration points on the calibration plate based on the absolute coordinates of the imaging center, and subtract the coordinates of the robot end to obtain the fixed coordinates of the center points of all calibration points on the calibration plate on the rotating mechanism 4 of the robot end 3.

[0122] Second embodiment, such as Figure 1 and Figure 3 As shown, this embodiment discloses a method for calibrating a robotic arm and measuring spatial angles using a tilted camera. This method is fully automatic. The camera 1 is tilted and mounted on the rotating mechanism 4 of the robotic arm's end effector 3. The calibration plate 2, consisting of a dot array, is tilted and mounted in a fixed position where it can clearly image within the camera 1. In other embodiments, when the robotic arm's end effector 3 only performs translational movements, the rotating mechanism 4 may not be present; that is, the camera 1 is tilted and mounted on the robotic arm's end effector 3. Therefore, when the robotic arm's end effector 3 only performs translational movements, the method includes the following steps:

[0123] Setting steps: Set the translation plane of the robot end effector 3 and the rotation plane of the rotation mechanism 4 of the robot end effector 3 as preset calibration planes.

[0124] Translation recording steps: control the end effector 3 of the robot arm to translate to different positions along the preset calibration plane multiple times, record the coordinates of the end effector of the robot arm after each translation, and use image processing algorithms to identify and record the pixel coordinates of all calibration points on the calibration board image captured by camera 1.

[0125] The projective transformation matrix calculation steps are as follows: Based on the coordinates of the robot end effector and the pixel coordinates of all calibration points on the calibration plate in the translation recording step, the projective transformation matrix between the coordinates of the robot end effector and the pixel coordinates of the center point of all calibration points on the calibration plate is calculated using the least squares method.

[0126] Steps for calculating spatial angles: Decompose each projective transformation matrix using the camera intrinsic parameter matrix, and calculate the spatial angles between the camera imaging plane, the robot end-effector translation plane, and the calibration plate plane.

[0127] When the end effector 3 of the robotic arm performs a rotational and translational motion, the method also includes the following steps:

[0128] Rotation and translation recording steps: control the robot end effector 3 to rotate to different angles along the preset calibration plane multiple times and then translate it. Record the coordinates of the robot end effector and the angle of the robot end effector rotation mechanism after each rotation and translation. The pixel coordinates of all calibration points on the calibration board are identified and recorded by the image processing algorithm of the calibration board image captured by camera 1.

[0129] Distance calculation steps: Based on the coordinates of the robot end effector and the pixel coordinates of all calibration points on the calibration board in the rotation and translation recording steps, the actual distance from the robot end effector 3 to the camera imaging center is calculated using the least squares method. This yields the actual coordinates of the camera imaging center and the center points of all calibration points on the calibration board in the robot space coordinate system, thus completing the calibration of the relationship between the camera and the robot.

[0130] Specifically, in this embodiment, the translation recording step includes:

[0131] S1-1b: After calibration begins, the starting position of the last calibration is read from the memory, and the robot arm is controlled to move to that position.

[0132] S1-2b: Control the end effector 3 of the robot arm to sequentially translate to the position array centered on the current position. For example, the number of rows and columns of the position array is 3x3 and the spacing is 5mm. After traversing all positions in the position array, return to the center position and record the coordinates of the end effector of the robot arm after each translation.

[0133] S1-3b: Since the circle becomes an ellipse after tilting distortion, and the centroid of the ellipse does not correspond to the actual center of the circle, the elliptical regions after the deformation of all the dots on the calibration plate 2 are obtained through the region detection algorithm. The centroid of each elliptical region is used as a temporary calibration point, and the points are sorted into a calibration point array through the convex hull algorithm.

[0134] S1-4b: Since the tangents of circles and ellipses are constant, the horizontal and vertical external tangents of each row and column of the elliptical region are calculated based on the calibration point array. The intersection of the diagonals of the external tangents of each elliptical region is used as the real calibration point to obtain the pixel coordinates of all calibration points on the calibration board.

[0135] In this embodiment, the steps for calculating the projective transformation matrix include:

[0136] S2-1b: Calculate the projective transformation matrix between the coordinates on the calibration plate and the pixel coordinates of all calibration points on the calibration plate after each translation of the robot end effector 3 in the translation recording step using the least squares method. That is, the projective transformation matrix between the calibration plate coordinate system and the camera pixel plane coordinate system.

[0137] S2-2b: Calculate the pixel coordinates of the center point of all calibration points on the calibration board using the projective transformation matrix between the coordinates on the calibration board and the pixel coordinates of all calibration points on the calibration board. Use this as input data to calculate the projective transformation matrix between the coordinates of the robot end effector and the pixel coordinates of the center point of all calibration points on the calibration board using the least squares method. This matrix is ​​the projective transformation matrix between the robot's spatial coordinate system and the camera's pixel plane coordinate system. This matrix can transform any image distance with the actual distance.

[0138] The total number of calibration points is an array. If both rows and columns are odd, the center point is one of the calibration points. If both rows and columns are even, non-existent center points must be calculated. This embodiment uses the double least squares method to calculate the projective transformation matrix between the coordinates of the robot's end effector and the pixel coordinates of the center points of all calibration points on the calibration plate, which can reduce the impact of image processing errors on calibration accuracy.

[0139] In this embodiment, the spatial angle calculation steps include:

[0140] S3-1b: Read the camera intrinsic parameter matrix from the memory, combine the camera intrinsic parameter matrix with the projective geometry method to decompose the projective transformation matrix between the coordinates of the robot end effector and the pixel coordinates of the center points of all calibration points on the calibration plate, and calculate the spatial angle between the camera imaging plane and the translation plane of the robot end effector.

[0141] S3-2b: Combining the camera intrinsic parameter matrix, the singular value decomposition (SVD) method is used to decompose the projective transformation matrix between the coordinates on the calibration plate of all calibration points and the pixel coordinates of all calibration points on the calibration plate after each translation of the robot end effector 3, and the spatial angle between the camera imaging plane and the calibration plate plane is calculated.

[0142] In this embodiment, the rotation and translation recording step includes:

[0143] S4-1b: Control the robotic arm end effector 3 to rotate to both sides by a preset angle within the preset calibration plane, for example, rotate to both sides by 6° and 12°, and then rotate back to the initial angle. After each rotation, control the robotic arm end effector 3 to translate so that all calibration points on the calibration plate are kept within the calibration plate image captured by camera 1, and record the coordinates of the robotic arm end effector after each rotation and translation.

[0144] S4-2b: Since the circle becomes an ellipse after tilting distortion, and the centroid of the ellipse does not correspond to the actual center of the circle, the elliptical regions after the deformation of all the dots on the calibration plate 2 are obtained through the region detection algorithm. The centroid of each elliptical region is used as a temporary calibration point, and the points are sorted into a calibration point array through the convex hull algorithm.

[0145] S4-3b: Since the tangents of circles and ellipses are constant, the horizontal and vertical external tangents of each row and column of the elliptical region are calculated based on the calibration point array. The intersection of the diagonals of the external tangents of each elliptical region is used as the real calibration point to obtain the pixel coordinates of all calibration points on the calibration board.

[0146] Among them, the rotation mechanism 4 of the robot end 3 is controlled to rotate and translate multiple times in different directions. After rotation and translation, the effective robot end coordinates and the angles of the robot end rotation mechanism that can identify all calibration points in the image are at least three sets, and these three sets of angles are not equal to each other.

[0147] In this embodiment, the distance calculation step includes:

[0148] S5-1b: The least squares method is used to calculate the pixel coordinates of the center points of all calibration points on the calibration plate after each rotation and translation. The actual distance from the center points of all calibration points on the calibration plate to the camera imaging center is calculated by applying the projective transformation matrix between the coordinates of the robot end effector and the pixel coordinates of the center points of all calibration points on the calibration plate. Camera 1 has a principal axis, and the intersection of the principal axis and the preset calibration plane is the camera imaging center, whose pixel coordinates can be obtained from the intrinsic parameter matrix; for an ideal camera, this is the center point of the image.

[0149] S5-2b: Then, rotate the translation distance in both clockwise and counterclockwise directions according to the rotation angle, and add it to the actual distance from the center point of all calibration points on the calibration plate to the camera imaging center. Calculate the two sets of actual distances from the center point of all calibration points on the calibration plate to the camera imaging center before the first rotation and translation after each rotation and translation.

[0150] S5-3b: Use the least squares method to process the two sets of actual distances from the center point of all calibration points on the calibration plate to the imaging center of the camera before the first rotation and translation, obtain the actual distance from the end effector 3 of the robot to the imaging center of the camera before the first rotation and translation, and determine whether the rotation direction of the robot is clockwise.

[0151] S5-4b: Add the actual distance from the end effector 3 of the robot arm to the camera imaging center before the first rotation and translation to the coordinates of the end effector of the robot arm obtained in the rotation and translation recording step to obtain the absolute coordinates of the camera imaging center in the robot arm spatial coordinate system.

[0152] S5-5b: Calculate the absolute coordinates of the center points of all calibration points on the calibration plate based on the absolute coordinates of the camera imaging center, and subtract the coordinates of the robot end to obtain the fixed coordinates of the camera 1 on the rotating mechanism 4 of the robot end 3.

[0153] The third embodiment, such as Figure 1 and Figure 4 As shown, this embodiment discloses a method for calibrating a robotic arm and measuring spatial angles using a tilted camera. This method is semi-automatic. The camera 1 is vertically mounted on the robotic arm end effector 3 or its rotating mechanism 4. The vertical mounting of the camera 1 is also a form of tilting, except the tilt angle is 0. A single-dot calibration plate 2 is horizontally mounted in a fixed position, where the calibration plate 2 can clearly image within the camera 1. In other embodiments, when the robotic arm end effector 3 only performs translational movements, the robotic arm end effector 3 may not have a rotating mechanism 4, meaning the camera 1 is tilted and mounted on the robotic arm end effector 3. Therefore, when the robotic arm end effector 3 only performs translational movements, the method includes the following steps:

[0154] Setting steps: Set the translation plane of the robot end effector 3 and the rotation plane of the rotation mechanism 4 of the robot end effector 3 as preset calibration planes.

[0155] Translation recording steps: control the end effector 3 of the robot arm to translate to different positions along the preset calibration plane multiple times, record the coordinates of the end effector of the robot arm after each translation, and use image processing algorithms to identify and record the pixel coordinates of all calibration points on the calibration board image captured by camera 1.

[0156] The projective transformation matrix calculation steps are as follows: Based on the coordinates of the robot end effector and the pixel coordinates of all calibration points on the calibration plate in the translation recording step, the projective transformation matrix between the coordinates of the robot end effector and the pixel coordinates of the center point of all calibration points on the calibration plate is calculated using the least squares method.

[0157] Steps for calculating the spatial angle: Decompose each projective transformation matrix using the camera intrinsic parameter matrix, and calculate the spatial angle between the camera imaging plane and the translation plane of the robot end effector.

[0158] When the end effector 3 of the robotic arm performs a rotational and translational motion, the method also includes the following steps:

[0159] Rotation and translation recording steps: control the robot end effector 3 to rotate to different angles along the preset calibration plane multiple times and then translate it. Record the coordinates of the robot end effector and the angle of the robot end effector rotation mechanism after each rotation and translation. The pixel coordinates of all calibration points on the calibration board are identified and recorded by the image processing algorithm of the calibration board image captured by camera 1.

[0160] Distance calculation steps: Based on the coordinates of the robot end effector and the pixel coordinates of all calibration points on the calibration board in the rotation and translation recording steps, the actual distance from the robot end effector 3 to the camera imaging center is calculated using the least squares method. This yields the actual coordinates of the camera imaging center and the center points of all calibration points on the calibration board in the robot space coordinate system, thus completing the calibration of the relationship between the camera and the robot.

[0161] Specifically, in this embodiment, the translation recording step includes:

[0162] S1-1c: Control the end effector 3 of the robot arm to translate sequentially in eight directions within the preset calibration plane. During each translation, the image processing algorithm determines and controls the calibration point on the calibration plate to be within the calibration plate image captured by the camera 1 in real time, and records the coordinates of the end effector of the robot arm after each translation.

[0163] Specifically, the operator controls the robot end effector 3 to translate, moving the calibration plate 2 to the center of the image captured by the camera 1. The operator controls the robot end effector 3 to translate sequentially in eight directions within the preset calibration plane. During each translation, the image processing algorithm determines in real time whether the unique calibration point on the calibration plate is within the image captured by the camera 1. The translation in that direction is stopped when the calibration point is about to exceed the image range. The coordinates of the robot end effector after each translation are recorded.

[0164] S1-2c: Obtain the pixel coordinates of the centroid of the region on calibration board 2 through the region detection algorithm, and use them as the pixel coordinates of all calibration points on calibration board 2, that is, the pixel coordinates of the unique calibration point on the calibration board.

[0165] In this embodiment, the steps for calculating the projective transformation matrix include:

[0166] S2-1c: Since there is only one calibration point on calibration plate 2, the double least squares method cannot be applied to reduce the error. Instead, the single circle on calibration plate 2 is used as the center point of the calibration point, and the least squares method is used to calculate the projective transformation matrix between the coordinates of the robot end effector and the pixel coordinates of the center point of all calibration points on the calibration plate.

[0167] In this embodiment, the spatial angle calculation steps include:

[0168] S3-1c: The camera intrinsic parameter matrix is ​​read from memory. Combined with the camera intrinsic parameter matrix, the projective geometry method is used to decompose the projective transformation matrix between the coordinates of the robot's end effector and the pixel coordinates of the center points of all calibration points on the calibration board. The spatial angle between the camera imaging plane and the robot's end effector translation plane is calculated. The calculation results show that a straight line at infinity remains a straight line at infinity after the projective transformation, indicating that the projective transformation is an affine transformation, meaning the camera imaging plane and the robot's end effector translation plane are parallel. Since there is only one calibration point on calibration board 2, the camera intrinsic parameter matrix cannot be recalculated, the tilt angle of the calibration board plane cannot be calculated, and the double least squares method cannot be used to reduce errors. The double least squares method means that the first least squares method calculates the center point, and the second least squares method calculates the mapping matrix or fitted circle.

[0169] In this embodiment, the rotation and translation recording step includes:

[0170] S4-1c: Controls the end effector 3 of the robotic arm to rotate to multiple different angles within a preset calibration plane, such as 5 different angles. During the rotation, the pixel coordinates of the calibration point are determined in real time through an image processing algorithm. The calibration point is kept in the center of the image captured by the camera 1 by controlling the translation of the end effector 3 of the robotic arm.

[0171] S4-2c: Record the coordinates of the robot's end effector after each rotation to the target angle.

[0172] S4-3c: The pixel coordinates of the centroid of the region on calibration board 2 are obtained through the region detection algorithm and used as the pixel coordinates of all calibration points on calibration board 2, that is, the pixel coordinates of the unique calibration point on the calibration board.

[0173] Among them, the rotation mechanism 4 of the robot end 3 is controlled to rotate and translate multiple times in different directions. After rotation and translation, the effective robot end coordinates and the angle of the robot end rotation mechanism that can identify a unique calibration point in the image are at least three sets, and the angles of these three sets are not equal to each other.

[0174] In this embodiment, the distance calculation step includes:

[0175] S5-1c: The actual distance from the center point of all calibration points on the calibration plate to the camera imaging center is calculated using the projective transformation matrix between the coordinates of the robot end effector and the pixel coordinates of the center point of all calibration points on the calibration plate. Camera 1 has a principal axis, and the intersection of the principal axis and the preset calibration plane is the camera imaging center, whose pixel coordinates can be obtained from the intrinsic parameter matrix; for an ideal camera, this is the center point of the image.

[0176] S5-2c: Then, rotate the translation distance in both clockwise and counterclockwise directions according to the rotation angle, and add it to the actual distance from the center point of all calibration points on the calibration plate to the camera imaging center. Calculate the two sets of actual distances from the center point of all calibration points on the calibration plate to the camera imaging center before the first rotation and translation after each rotation and translation.

[0177] S5-3c: Use the least squares method to process the two sets of actual distances from the center point of all calibration points on the calibration plate to the imaging center of the camera before the first rotation and translation, obtain the actual distance from the end effector 3 of the robot to the imaging center of the camera before the first rotation and translation, and determine whether the rotation direction of the robot is clockwise.

[0178] S5-4c: Add the actual distance from the end effector 3 of the robot arm to the camera imaging center before the first rotation and translation to the coordinates of the end effector of the robot arm obtained in the rotation and translation recording step to obtain the absolute coordinates of the camera imaging center in the robot arm spatial coordinate system.

[0179] S5-5c: Calculate the absolute coordinates of the center points of all calibration points on the calibration plate based on the absolute coordinates of the camera imaging center, and subtract the coordinates of the robot end to obtain the fixed coordinates of the camera on the rotating mechanism 4 of the robot end 3.

[0180] The fourth embodiment discloses a robot, such as an industrial robot, including a controller, a manipulator, and a camera. The controller is used to implement the manipulator calibration and spatial angle measurement method for a tilting camera as described in any one of the first, second, and third embodiments above, which will not be repeated here.

[0181] By implementing this invention, the following beneficial effects are achieved:

[0182] This invention is applicable to tilted cameras, offering a simple calibration process and high accuracy, eliminating calibration errors caused by the installation angles of the camera and calibration plate. It uses projective transformation to handle the coordinate mapping relationship between non-parallel planes and combines the camera's intrinsic parameter matrix to decompose the projective matrix, calculating the spatial angles between the camera imaging plane, the robot end effector translation plane, and the calibration plate plane. Industrial robots calibrated using this invention can adapt to more complex environments, meet more requirements, and achieve higher working accuracy.

[0183] It is understood that the above embodiments only illustrate some implementation methods of the present invention, and their descriptions are relatively specific and detailed, but they should not be construed as limiting the scope of the present invention. It should be noted that for those skilled in the art, without departing from the concept of the present invention, the above embodiments or technical features can be freely combined, and several modifications and improvements can be made. These all fall within the protection scope of the present invention. That is, the embodiments described "in some embodiments" can be freely combined with any of the embodiments above and below. Therefore, all equivalent transformations and modifications made within the scope of the claims of the present invention should fall within the scope of the claims of the present invention.

Claims

1. A method for calibrating a robotic arm and measuring spatial angles for a tilting camera, characterized in that, Includes the following steps: Setting steps: Set the translation plane of the robot end effector and the rotation plane of the rotation mechanism of the robot end effector as preset calibration planes; Translation recording steps: control the end effector of the robot arm to translate to different positions along the preset calibration plane multiple times, record the coordinates of the end effector of the robot arm after each translation, and use image processing algorithms to identify and record the pixel coordinates of all calibration points on the calibration board image captured by the camera; Projective transformation matrix calculation steps: Based on the coordinates of the robot end point and the pixel coordinates of all calibration points on the calibration plate in the translation recording step, the projective transformation matrix between the coordinates of the robot end point and the pixel coordinates of the center point of all calibration points on the calibration plate is calculated using the least squares method; Spatial angle calculation steps: Decompose each of the projective transformation matrices using the camera intrinsic parameter matrix, and calculate the spatial angle between the camera imaging plane and the translation plane of the robot end effector, or between the camera imaging plane, the translation plane of the robot end effector, and the calibration plate plane. The method further includes: Rotation and translation recording steps: control the rotation mechanism of the end effector of the robot arm to rotate to different angles along the preset calibration plane multiple times and then translate it. Record the coordinates of the end effector of the robot arm after each rotation and translation. Use image processing algorithms to identify and record the pixel coordinates of all calibration points on the calibration board image captured by the camera. Distance calculation step: Based on the coordinates of the robot end effector and the pixel coordinates of all calibration points on the calibration board in the rotation and translation recording step, the actual distance from the robot end effector to the camera imaging center is calculated using the least squares method. This yields the actual coordinates of the camera imaging center and the center points of all calibration points on the calibration board in the robot space coordinate system, thus completing the calibration of the relationship between the camera and the robot.

2. The method for calibrating a robotic arm and measuring spatial angles for a tilting camera according to claim 1, characterized in that, The method is applicable to the first case: the camera is tilted and mounted in a fixed position, and the calibration plate of the checkerboard pattern is tilted and mounted on the end of the robot or on the rotating mechanism of the end of the robot; The method is also applicable to a second case: the camera is tilted and mounted on the end of the robotic arm or on the rotating mechanism of the end of the robotic arm, and the calibration plate of the dot array is tilted and mounted in a fixed position. The method is also applicable to a third case: the camera is vertically mounted on the end of the robotic arm or on the rotating mechanism of the end of the robotic arm, and the calibration plate with a single dot is horizontally mounted in a fixed position.

3. The method for calibrating a robotic arm and measuring spatial angles for a tilting camera according to claim 2, characterized in that, When the method is applicable to the first case, the translation recording step includes: The robotic arm end effector is controlled to translate freely within the preset calibration plane, and all calibration points on the calibration plate are kept within the calibration plate image captured by the camera. The coordinates of the robotic arm end effector after each translation are recorded. Based on the calibration board image captured by the camera, edge detection and line detection algorithms are applied to the checkerboard pattern on the calibration board, and lines at different angles are separated by clustering algorithm; Find the intersection points of all intersecting lines, and sort all the intersection points into a calibration point array by rows and columns using the convex hull algorithm to obtain the pixel coordinates of all calibration points on the calibration board. When the method is applicable to the second case, the translation recording step includes: After calibration begins, the starting position of the last calibration is read from the memory, and the robot arm is controlled to move to that position. The robotic arm end effector is controlled to sequentially translate to a position array centered on the current position, traversing all positions in the position array and returning to the center position, and the coordinates of the robotic arm end effector after each translation are recorded; The elliptical regions after deformation of all dots on the calibration plate are obtained by the region detection algorithm. The centroid of each elliptical region is used as a temporary calibration point and sorted into a calibration point array by the convex hull algorithm. Based on the calibration point array, calculate the horizontal and vertical external tangents of the elliptical regions in each row and column, and use the intersection of the diagonals of the external tangents of each elliptical region as the real calibration point to obtain the pixel coordinates of all calibration points on the calibration board. When the method is applicable to the third case, the translation recording step includes: The robotic arm end effector is controlled to translate sequentially in eight directions within the preset calibration plane. During each translation, an image processing algorithm is used to determine and control the calibration points on the calibration plate within the calibration plate image captured by the camera, and the coordinates of the robotic arm end effector after each translation are recorded. The pixel coordinates of the centroid of the region on the calibration board are obtained by the region detection algorithm and used as the pixel coordinates of all calibration points on the calibration board.

4. The method for calibrating a robotic arm and measuring spatial angles for a tilting camera according to claim 2, characterized in that, When the method is applicable to both the first and second cases, the projective transformation matrix calculation steps include: The least squares method is used to calculate the projective transformation matrix between the coordinates on the calibration plate of all calibration points and the pixel coordinates of all calibration points on the calibration plate after each translation of the robot end in the translation recording step. The pixel coordinates of the center point of all calibration points on the calibration board are calculated using the projective transformation matrix between the coordinates on the calibration board and the pixel coordinates of all calibration points on the calibration board. This matrix is ​​then used as input data to calculate the projective transformation matrix between the coordinates of the robot end effector and the pixel coordinates of the center point of all calibration points on the calibration board using the least squares method. When the method is applicable to the third case, the steps for calculating the projective transformation matrix include: Using a single dot on the calibration plate as the center point of the calibration point, the least squares method is used to calculate the projective transformation matrix between the coordinates of the robot end effector and the pixel coordinates of the center point of all calibration points on the calibration plate.

5. The method for calibrating a robotic arm and measuring spatial angles for a tilting camera according to claim 4, characterized in that, When the method is applicable to the first case, the spatial angle calculation steps include: The Zhang Zhengyou calibration method is used to process the pixel coordinates of all calibration points on the calibration board at different positions and the coordinates on the calibration board of all calibration points on the calibration board obtained in the translation recording step, and the camera intrinsic parameter matrix is ​​calculated. By combining the camera intrinsic parameter matrix, the singular value decomposition method is used to decompose the projective transformation matrix between the coordinates of the robot end effector and the pixel coordinates of the center points of all calibration points on the calibration plate, and the spatial angle between the camera imaging plane and the translation plane of the robot end effector is calculated. Then, by combining the camera intrinsic parameter matrix, the projective geometry method is used to decompose the projective transformation matrix between the coordinates on the calibration plate of all calibration points and the pixel coordinates of all calibration points on the calibration plate after each translation of the robot end effector, and the spatial angle between the camera imaging plane and the calibration plate plane is calculated. By combining the spatial angle between the camera imaging plane and the translation plane of the robot end effector and the spatial angle between the camera imaging plane and the calibration plate plane, the spatial angle between the calibration plate plane and the translation plane of the robot end effector is calculated; When the method is applicable to the second case, the spatial angle calculation steps include: The camera intrinsic parameter matrix is ​​read from the memory. The projective transformation matrix between the coordinates of the robot end effector and the pixel coordinates of the center points of all calibration points on the calibration plate is decomposed using the projective geometry method based on the camera intrinsic parameter matrix. The spatial angle between the camera imaging plane and the translation plane of the robot end effector is calculated. Then, by combining the camera intrinsic parameter matrix, the singular value decomposition method is used to decompose the projective transformation matrix between the coordinates on the calibration plate of all calibration points and the pixel coordinates of all calibration points on the calibration plate after each translation of the robot end effector, and the spatial angle between the camera imaging plane and the calibration plate plane is calculated. When the method is applicable to the third case, the spatial angle calculation steps include: The camera intrinsic parameter matrix is ​​read from the memory. The projective transformation matrix between the coordinates of the robot end effector and the pixel coordinates of the center points of all calibration points on the calibration plate is decomposed using the projective geometry method based on the camera intrinsic parameter matrix. The spatial angle between the camera imaging plane and the translation plane of the robot end effector is then calculated.

6. The method for calibrating a robotic arm and measuring spatial angles for a tilting camera according to claim 2, characterized in that, When the method is applicable to the first case, the rotation and translation recording step includes: The robotic arm end effector is controlled to rotate freely within the preset calibration plane. After each rotation, the robotic arm end effector is controlled to translate to keep all calibration points on the calibration plate within the calibration plate image captured by the camera, and the coordinates of the robotic arm end effector after each rotation and translation are recorded. Based on the calibration board image captured by the camera, edge detection and line detection algorithms are applied to the checkerboard pattern on the calibration board, and lines at different angles are separated by clustering algorithm; Find the intersection points of all intersecting lines, and sort all the intersection points into a calibration point array by rows and columns using the convex hull algorithm to obtain the pixel coordinates of all calibration points on the calibration board. When the method is applicable to the second case, the rotation and translation recording steps include: The robotic arm end effector is controlled to rotate to both sides by a preset angle within the preset calibration plane, and then rotate back to the initial angle. After each rotation, the robotic arm end effector is controlled to translate to keep all calibration points on the calibration plate within the calibration plate image captured by the camera, and the coordinates of the robotic arm end effector after each rotation and translation are recorded. The elliptical regions after deformation of all dots on the calibration plate are obtained by the region detection algorithm. The centroid of each elliptical region is used as a temporary calibration point and sorted into a calibration point array by the convex hull algorithm. Based on the calibration point array, calculate the horizontal and vertical external tangents of the elliptical regions in each row and column, and use the intersection of the diagonals of the external tangents of each elliptical region as the real calibration point to obtain the pixel coordinates of all calibration points on the calibration board. When the method is applicable to the third case, the rotation and translation recording steps include: The robotic arm end effector is controlled to rotate to multiple different angles within the preset calibration plane. During the rotation, the pixel coordinates of the calibration point are determined in real time by the image processing algorithm. The calibration point is kept in the center of the image captured by the camera by controlling the robotic arm end effector to follow the translation. Record the coordinates of the robot's end effector after each rotation to the target angle; The pixel coordinates of the centroid of the region on the calibration board are obtained by the region detection algorithm and used as the pixel coordinates of all calibration points on the calibration board.

7. The method for calibrating a robotic arm and measuring spatial angles for a tilting camera according to claim 2, characterized in that, When the method is applicable to the first case, the distance calculation step includes: The pixel coordinates of the center points of all calibration points on the calibration plate after each rotation and translation are calculated using the least squares method. The actual distance from the center point of all calibration points on the calibration plate to the camera imaging center is calculated by applying the projective transformation matrix between the coordinates of the robot end and the pixel coordinates of the center points of all calibration points on the calibration plate. Then, the actual distance from the center point of all calibration points on the calibration plate to the camera imaging center is superimposed with the translation distance of each rotation and translation to calculate the actual distance from the center point of all calibration points on the calibration plate to the camera imaging center before the first rotation and translation after each rotation and translation. The actual distance from the center point of all calibration points on the calibration plate to the camera imaging center before the first rotation and translation is obtained by processing the least squares method. Add the actual distance from the end of the robotic arm to the imaging center of the camera before the first rotation and translation to the coordinates of the end of the robotic arm obtained in the rotation and translation recording step to obtain the absolute coordinates of the imaging center of the camera in the robotic arm spatial coordinate system; The absolute coordinates of the center points of all calibration points on the calibration plate are calculated based on the absolute coordinates of the camera imaging center. The coordinates of the end effector of the robot are then subtracted to obtain the fixed coordinates of the center points of all calibration points on the calibration plate on the rotating mechanism of the end effector of the robot. When the method is applicable to the second and third cases, the distance calculation steps include: The actual distance from the center point of all calibration points on the calibration plate to the camera imaging center is calculated by applying the projective transformation matrix between the coordinates of the robot end effector and the pixel coordinates of the center points of all calibration points on the calibration plate; Then, after rotating the translation distance in both clockwise and counterclockwise directions according to the rotation angle, add the actual distance from the center point of all calibration points on the calibration plate to the imaging center of the camera, and calculate the two sets of actual distances from the center point of all calibration points on the calibration plate to the imaging center of the camera before the first rotation and translation after each rotation and translation. The least squares method is used to process the two sets of actual distances from the center point of all calibration points on the calibration plate to the imaging center of the camera before the first rotation and translation, to obtain the actual distance from the end of the robot arm to the imaging center of the camera before the first rotation and translation, and to determine whether the rotation direction of the robot arm is clockwise. Add the actual distance from the end of the robotic arm to the imaging center of the camera before the first rotation and translation to the coordinates of the end of the robotic arm obtained in the rotation and translation recording step to obtain the absolute coordinates of the imaging center of the camera in the robotic arm spatial coordinate system; The absolute coordinates of the center points of all calibration points on the calibration plate are calculated based on the absolute coordinates of the camera's imaging center. The coordinates of the robot's end effector are then subtracted to obtain the fixed coordinates of the camera on the rotating mechanism at the end effector of the robot.

8. A robot, comprising a controller, a robotic arm, and a camera, characterized in that, The controller is used to implement the robotic arm calibration and spatial angle measurement method for tilt cameras as described in any one of claims 1 to 7.

Images