A method for camera calibration using a calibrated robot

Abstract

The present invention relates to a method for calibrating a camera using a calibrated robot. First, the robot is pre-calibrated using a calibrated camera to obtain calibration parameters of the robot. Then, the camera to be calibrated is replaced with the calibrated camera to collect images for calibration, and the parameters of the camera to be calibrated are solved. By using this method, the result of calibration using a long focal length module is sufficiently accurate, and the accuracy of image recognition for large distortion effects is improved.

Description

Technical Field

[0001] The present invention relates to the technical field of camera calibration, and in particular to a method for performing camera calibration by using a calibrated robot. Background Art

[0002] In the modern industrial field, with the development of technologies such as artificial intelligence and autonomous driving, the requirements for a series of technical indicators such as the recognition accuracy of machine vision and the accuracy of optical measurement have become extremely high. As an important component of the image acquisition system, the camera module needs to be accurately calibrated to ensure the accuracy of the subsequent visual system.

[0003] At present, the commonly used camera calibration method is the "Zhang Zhengyou calibration method", which requires a moving target to take multiple pictures in different postures relative to the camera module to ensure that the imaging has sufficient perspective distortion effect. For long focal length modules, the perspective distortion effect of the imaging calibrated by Zhang Zhengyou's calibration method is not obvious enough, resulting in insufficient constraints on the optimization parameters and inaccurate calibration results. For images with large distortion effects, the images calibrated by Zhang Zhengyou's calibration method are severely distorted. In the process of overall optimization of internal and external parameters, the calculation results have large errors, and the calibration results are unstable, which is not conducive to the subsequent processing of the images generated by the camera. In order to achieve higher recognition accuracy and precision, it is necessary to improve the existing camera calibration method and propose a new camera calibration method. Summary of the invention

[0004] The purpose of the present invention is to design a method for calibrating a camera using a calibrated robot, so that the result of calibration using a long focal length module is sufficiently accurate and the accuracy of image recognition for large distortion effects is improved.

[0005] To achieve the above object, the present invention provides the following technical solutions:

[0006] A method for calibrating a camera using a calibrated robot. First, the robot is pre-calibrated using a calibrated camera to obtain calibration parameters of the robot. Then, a camera to be calibrated is used to replace the calibrated camera to collect images for calibration, and the parameters of the camera to be calibrated are solved.

[0007] Furthermore, in the above-mentioned method of calibrating a camera using a calibrated robot, the pre-calibration of the robot includes the following steps: modeling the robot; establishing and solving the transformation relationship between points on the calibration plate, the robot and the camera; normalizing the point set in the camera coordinate system to the ideal image coordinate system; correcting the distortion and solving the transformation parameters from the anti-distortion point to the image coordinate system; calibrating the image, acquiring multiple pixel points and obtaining the total error optimization objective function of all pixel points; solving the parameters to complete the pre-calibration of the robot.

[0008] Furthermore, in the above-mentioned method of calibrating a camera using a calibrated robot, the modeling of the robot includes establishing a robot base coordinate system, establishing an X-axis rotation joint coordinate system, establishing a Y-axis rotation joint coordinate system, and establishing a gripper coordinate system.

[0009] Furthermore, in the above-mentioned method of calibrating a camera using a calibrated robot, the conversion relationship T0 between the calibration plate coordinate system and the robot base coordinate system is

[0010] The transformation relationship T1 between the gripper coordinate system and the camera coordinate system is:

[0011] The transformation relationship between the robot base coordinate system and the camera coordinate system is P c =T1·DH x DH y ·P robot .

[0012] Furthermore, in the above-mentioned method of calibrating the camera using the calibrated robot, the transformation formula for normalizing the point set in the camera coordinate system to the ideal image coordinate system is: Z c Not 0.

[0013] Furthermore, in the above-mentioned method of camera calibration using a calibrated robot, the radial distortion forward model considered in the distortion correction is: In the formula (x′ n ,y′ n ) is the anti-distortion point, denoted as point P′ n ;

[0014] Anti-distortion point P n The transformation formula from ′ to the image coordinate system is P′ u =A·P′ n , the internal parameter conversion matrix

[0015] Furthermore, in the above-mentioned method of performing camera calibration using a calibrated robot, the total error optimization objective function is:

[0016] Furthermore, in the above-mentioned method of calibrating the camera using a calibrated robot, the DH parameter is the conversion relationship between the x-axis link and the y-axis link of the robot's mechanical arm:

[0017]

[0018]

[0019] Furthermore, in the above-mentioned method of calibrating a camera by using a calibrated robot, the camera to be calibrated is used to replace the calibrated camera for image acquisition for calibration, and solving the parameters of the camera to be calibrated includes the following steps: installing the camera to be calibrated and rotating it for image acquisition; detecting corner point information to obtain the corner points of the image coordinate system and the corresponding calibration plate coordinate system points; calibrating the image acquisition to obtain multiple pixel points and obtain the total error optimization objective function of all pixel points; solving the parameters to complete the calibration of the camera.

[0020] Furthermore, in the above-mentioned method of performing camera calibration using a calibrated robot, the total error optimization objective function is:

[0021] Compared with the prior art, the present invention has the following beneficial effects: the present invention provides a method for calibrating a camera using a calibrated robot, firstly, the robot is pre-calibrated using a calibrated camera, and the robot DH is solved by modeling. x and DH y Parameters, as well as the T0 parameters of the posture relationship between the robot and the calibration plate, and then the external parameters of the camera to be calibrated are precisely constrained based on these pre-calibrated parameters, reducing the parameter freedom in the process of solving the internal and external parameters of the camera calibration, so that the calibration results are more accurate, fast and stable. Compared with traditional algorithms such as the "Zhang Zhengyou calibration method", the requirements for calibration image acquisition are simplified. By introducing the robot DH parameters to constrain the external parameters, the number of parameters that need to be solved during the camera calibration process is reduced, and there is no need to move the calibration plate to generate sufficient perspective distortion. At the same time, it can solve the problem of not being able to obtain sufficient perspective distortion in the calibration of the telephoto module, resulting in the equivalent focal length and image principal point C x and C y The problem of inaccurate estimation has been solved. The calibration results of the long focal length module are sufficiently accurate, and the accuracy of image recognition for large distortion effects is improved. BRIEF DESCRIPTION OF THE DRAWINGS

[0022] Figure 1 It is a diagram illustrating the pre-calibrated robot DH parameters of the present invention;

[0023] Figure 2 This is a calibration effect diagram using the method of the present invention;

[0024] Figure 3 This is a corner point finding effect diagram using the method of the present invention;

[0025] Figure 4 This is an anti-distortion effect diagram using the method of the present invention;

[0026] Figure 5 This is a parameter flow chart of the camera calibration performed by a robot according to the present invention. DETAILED DESCRIPTION

[0027] In order to further explain the technical means and effects adopted by the present invention to achieve the predetermined invention purpose, the specific implementation mode, structure, characteristics and effects of the present invention are described in detail below in combination with the accompanying drawings and preferred embodiments.

[0028] Example: Please refer to Figure 1-5 A method for calibrating a camera by using a calibrated robot, using a high-precision chessboard as a calibration plate, conveniently locating accurate sub-pixel image coordinates. In practical applications, the calibration plate is not limited to the chessboard form, and any pattern similar to the chessboard that can find collinear points can be used as a calibration plate. The relationship between the robot, the camera, and the calibration plate is as follows: a gripper is provided at the end of the robot's mechanical arm, the gripper grips the camera, and the camera head is aligned with the calibration plate. The robot's mechanical arm has multiple rotation joints. In this embodiment, the mechanical arm has an X-axis rotation joint and a Y-axis rotation joint, and the X-axis rotation joint and the Y-axis rotation joint each have their rotation centers.

[0029] First, the robot is pre-calibrated. To pre-calibrate the robot, a calibrated camera is required, that is, the camera that the gripper holds when the robot needs to be pre-calibrated has been calibrated.

[0030] Pre-calibrating the robot includes the following steps:

[0031] (1) Model the robot, specifically: establish the robot base coordinate system with the robot base as the origin; establish the X-axis rotation joint coordinate system with the X-axis rotation center as the origin; establish the Y-axis rotation joint coordinate system with the Y-axis rotation center as the origin; and establish the gripper coordinate system with the center point of the gripper end as the origin.

[0032] (2) Solve the DH parameters of the conversion relationship between the x-axis link and the y-axis link of the robot arm.

[0033] The link refers to the part of the robot arm separated by two adjacent joints. It is a rigid body used to maintain a fixed relationship between each joint. For the previous coordinate system and the current coordinate system of the link, the conversion relationship between the two is: Among them, Figure 1 As shown, d n is the joint offset, defined as the distance from x n-1 Move to x n The distance along z n-1 The direction is positive; θ n is the joint angle, defined as the angle from x n-1 Rotate to x n The angle around z n Positive rotation is positive; a n is the connecting rod length, defined as the distance from z n-1 Move to z n The distance along x nThe axis is pointing positively; α n is the connecting rod rotation angle, defined as the rotation from z n-1 Rotate to z n The angle around x n The positive rotation of the axis is positive; Rot represents rotation transformation, and Trans represents translation transformation. When n=1, it represents the transformation relationship between the Y-axis rotation joint coordinate system and the robot base coordinate system; when n=2, it represents the transformation relationship between the X-axis rotation joint coordinate system and the Y-axis rotation joint coordinate system and the robot base coordinate system. n has a maximum value, and the maximum value of n is related to the number of connecting rods. In this embodiment, the robot arm can rotate only in the x-axis and y-axis directions, that is, there are only two connecting rods, the x-axis connecting rod and the y-axis connecting rod, and the maximum value n=2.

[0034] The DH parameters of the x-axis connecting rod and the y-axis connecting rod are calculated as follows:

[0035]

[0036]

[0037] For DH x Parameters, "Current Coordinate System" is the X-axis rotation joint coordinate system, "Previous Coordinate System" is the Y-axis rotation joint coordinate system; for DH y Parameters: "Current Coordinate System" is the Y-axis rotation joint coordinate system, and "Previous Coordinate System" is the robot base coordinate system.

[0038] (3) Establish the transformation relationship T0 between the calibration plate coordinate system and the robot base coordinate system, and use three-dimensional rigid body transformation to model. Formula 3, Among them, P w =[X w ,Y w ,Z w ] T Indicates the calibration plate coordinate system point, P robot =[X robot ,Y robot ,Z robot ] T Represents the robot base coordinate system point, R represents the rotation transformation matrix from the calibration plate coordinate system to the robot base coordinate system, and T represents the translation transformation matrix from the calibration plate coordinate system to the robot base coordinate system.

[0039] (4) Establish the transformation relationship T1 between the gripper coordinate system and the camera coordinate system, and use three-dimensional rigid body transformation to model. Formula 4, Among them, P Fixture =[X Fixture ,Y Fixture ,ZFixture ] T Represents the gripper coordinate system point, P c =[X c ,Y c ,Z c ] T represents the camera coordinate system point, R′ represents the rotation transformation matrix from the gripper coordinate system to the camera coordinate system, and T′ represents the translation transformation matrix from the gripper coordinate system to the camera coordinate system. In the subsequent actual calibration process, the relative position of the gripper and the camera module changes very little, so the translation transformation matrix T′ can be considered to remain unchanged, and the rotation transformation matrix R′ can be fine-tuned.

[0040] (5) Establish the transformation relationship between the robot base coordinate system and the camera coordinate system, P c =T1·DH x DH y ·P robot , Formula 5, where P C =(X C ,Y C ,Z C ) represents a point set in the camera coordinate system.

[0041] (6) When Z c When it is not 0, the point set in the camera coordinate system is normalized to the ideal image coordinate system: Formula 6, where P n =[X c / Z c ,Y c / Z c ,1] T =[X n ,Y n , 1] T represents the normalized ideal image coordinate system, Z c Indicates the Z-axis coordinate value in the camera coordinate system.

[0042] (7) Distortion correction. According to Zhang Zhengyou’s calibration method, the forward model considering radial distortion is: Formula 7, where (x′ n ,y′ n ) is the anti-distortion point, denoted as point P′ n ,(x n ,y n ) is the normalized ideal image coordinate point Pn in (6), Represents the Euclidean distance from the distortion point to the distortion center. This formula represents the optical lens distortion model, which uses polynomial fitting to approximate the optical distortion characteristics. For ordinary pinhole models, k1, k2, and k3 items can meet the requirements; for lenses with severe distortion, high-order polynomial models or division models need to be considered. Here, ellipsis is used to represent the general fitting polynomial.

[0043] (8) Solve for the anti-distortion point P′ n The intrinsic transformation matrix A, P′ to the image coordinate system u =A·P′ n , formula 8, Where A represents the current camera intrinsic conversion matrix, C x and C y represents the coordinates of the distortion center, P′ u Represents the point projected from the calibration plate coordinate system to the image coordinate system after a series of coordinate transformations.

[0044] (9) Calibrate the image, obtain multiple pixels and get the total error optimization objective function of all pixels. The pixel point is the point in the image coordinate system, and P u Indicates. For the jth coordinate point of the i-th image The final pixel reprojection error is Formula 9, where Formula 10. T intrinsic is the camera’s intrinsic parameter transformation parameter, which is equivalent to the current camera’s intrinsic parameter transformation matrix A, and can be obtained by equation 8; T dist is the distortion parameter, which can be obtained by equation 7; T proj is the normalized transformation parameter, which can be obtained by equation 6. The calibration sample is m images (m>0) and n points on these images. According to equations 1 to 10, the total error optimization objective function of all pixels can be obtained as: Equation 11. Since the pre-calibrated robot uses a calibrated camera, T intrinsic , T dist is a known quantity, T proj For fixed operation. x and DH y The calculation is performed based on the actual position of the robot arm.

[0045] (10) Solve for the parameters T1 and T0. Use a nonlinear solution method to iteratively optimize and solve Equation 11. Commonly used optimization algorithms include the Gauss-Newton method and the LM method.

[0046] The robot is pre-calibrated by steps (1) to (10), and then the calibrated robot is used to calibrate the camera to be calibrated, including the following steps:

[0047] (11) Install the camera to be calibrated, and let the robotic arm drive the gripper and the camera to rotate, and take multiple pictures.

[0048] (12) Detect corner point information, such as Figure 3 As shown, the image coordinate system corner point P is obtained u " and the corresponding calibration plate coordinate system point P w ". Image coordinate system corner point P u There are many detection methods, such as manual corner point selection method, Harris corner point detection method, Opencv findChessboardCorners() function method, etc. In this embodiment, Harris corner point detection + edge-based region growing method is used to realize the image coordinate system corner point P u "Detection. Calibration plate coordinate system point P w The detection of " needs to be combined with the actual calibration board chessboard grid length and width, with the upper left corner of the calibration board as the origin, to establish the calibration board coordinate system to measure point P w ”.

[0049] (13) Online calibration takes k images (k>0) and h points on the image. According to equations 1 to 10, the total error optimization objective function of all pixel points is obtained as: Equation 12. Since a calibrated robot is used to calibrate the camera, the robot’s pose is known for each calibration image, i.e., DH x and DH y The parameter is a constant; and the position value of the robot relative to the calibration plate is pre-calibrated, that is, the conversion relationship between the calibration plate coordinate system and the robot base coordinate system T0 parameter is a constant; and T proj is a fixed normalized transformation parameter.

[0050] (14) Solve the camera’s intrinsic transformation parameter T′ intrinsic , distortion parameter T′ dist And the transformation relationship between the gripper coordinate system and the camera coordinate system T′1. Use a nonlinear solution method to iteratively optimize and solve Equation 12. Commonly used optimization algorithms include Gauss-Newton method and LM method. After the camera is fixed, the relative position of the camera and the gripper remains unchanged during the calibration process, that is, multiple calibration images correspond to the same set of T′1 rigid body transformations.

[0051] Steps (11)-(14) can be used to calibrate the camera using a calibrated robot.

[0052] The present invention provides a method for calibrating a camera using a calibrated robot. First, the robot is pre-calibrated using a calibrated camera, and a robot DH is obtained by modeling. x and DH yParameters, as well as the T0 parameters of the posture relationship between the robot and the calibration plate, and then the external parameters of the camera to be calibrated are precisely constrained based on these pre-calibrated parameters, reducing the parameter freedom in the process of solving the internal and external parameters of the camera calibration, so that the calibration result is more accurate, fast and stable.

[0053] Compared with traditional algorithms such as the Zhang Zhengyou calibration method, this method of camera calibration using a calibrated robot has the following significant advantages:

[0054] 1. Simplified the requirements for calibration and image acquisition. The mobile gripper drives the camera module to acquire images, and there is no need to move the calibration plate to different positions for acquisition, which reduces the amount of data and improves the efficiency of the algorithm. Preferably, calibration takes nine images of a nine-square grid that cover a sufficient field of view.

[0055] 2. The robot DH parameters are introduced for calibration, which reduces the number of parameters to be solved. In the traditional calibration method, each calibration image has an external parameter relative to the camera module. When taking multiple images for calibration, multiple sets of external parameters need to be calculated, and the number of parameters to be solved is large. By pre-calibrating the robot, accurate external parameter constraints are provided for the camera to be calibrated. Only the camera's internal parameter transformation parameter T′ needs to be solved. intrinsic , distortion parameter T′ dist And fine-tune a small number of parameters. Each time the camera to be calibrated is replaced, since the position of the camera relative to the gripper changes very little, T′1 can be fine-tuned on the basis of T1 (the translation transformation matrix T′ remains unchanged, and the rotation transformation matrix R′ is fine-tuned). In addition, multiple calibration images share a set of T1′ parameters, which reduces the number of parameters to be solved and increases the robustness of the algorithm. Telephoto lenses have high requirements for the external environment (calibration external parameters), mainly because they require a large focusing space. Wide-angle and large-distortion modules have high requirements for FOV (field of view) coverage. This method is suitable for ordinary camera modules, and improves the calibration accuracy of telephoto lens modules and wide-angle and large-distortion modules, solving the industry difficulties in module calibration. The calibration effect is as follows: Figure 2 shown.

[0056] 3. The robot DH parameters are introduced to constrain the external parameters. There is no need to move the calibration plate to produce sufficient perspective distortion. At the same time, it can solve the problem of insufficient perspective distortion in the calibration of the telephoto module, which leads to the equivalent focal length and the image principal point C. x and C y The problem of inaccurate estimation makes the calibration result of the long focal length module accurate enough and improves the accuracy of image recognition for large distortion effects. Figure 4 shown.

[0057] The above description is only a preferred embodiment of the present invention and does not limit the present invention in any form. Although the present invention has been disclosed as a preferred embodiment as above, it is not used to limit the present invention. Any technical personnel in this field can make some changes or modify the technical contents disclosed above into equivalent embodiments without departing from the scope of the technical solution of the present invention. However, any brief modifications, equivalent changes and modifications made to the above embodiments based on the technical essence of the present invention without departing from the content of the technical solution of the present invention are still within the scope of the technical solution of the present invention.

Claims

1. A method for calibrating a camera using a calibrated robot, characterized in that: First, the robot is pre-calibrated with a calibrated camera to obtain the calibration parameters of the robot. Then, the camera to be calibrated is used to replace the calibrated camera to collect images for calibration, and the parameters of the camera to be calibrated are solved. The robot pre-calibration comprises the following steps: modeling the robot; establishing and solving the conversion relationship between the calibration plate, the robot and the camera; normalizing the point set in the camera coordinate system to the ideal image coordinate system; correcting the distortion and solving the conversion parameters from the anti-distortion point to the image coordinate system; calibrating the image, obtaining multiple pixel points and obtaining the total error optimization objective function of all pixel points; solving the parameters and completing the robot pre-calibration; The total error optimization objective function in the pre-calibration step is , where T intrinsic is the camera’s intrinsic transformation parameter, T dist is the distortion parameter, T proj is the normalized transformation parameter; m is the number of pre-calibrated images, n is the number of points on the image, is the jth coordinate point of the i-th picture, represents the jth calibration plate coordinate system point of the i-th image, T0 is the conversion relationship between the calibration plate coordinate system and the robot base coordinate system, and T1 is the conversion relationship between the gripper coordinate system and the camera coordinate system; for DH x Parameters, "Current Coordinate System" is the X-axis rotation joint coordinate system, "Previous Coordinate System" is the Y-axis rotation joint coordinate system; for DH y Parameters, "Current Coordinate System" is the Y-axis rotation joint coordinate system, and "Previous Coordinate System" is the robot base coordinate system.

2. The method for camera calibration using a calibrated robot according to claim 1, characterized in that: The modeling of the robot includes establishing a robot base coordinate system, an X-axis rotation joint coordinate system, a Y-axis rotation joint coordinate system and a gripper coordinate system.

3. The method for camera calibration using a calibrated robot according to claim 2, characterized in that: The transformation relationship T0 between the calibration plate coordinate system and the robot base coordinate system is: , , ; The transformation relationship T1 between the gripper coordinate system and the camera coordinate system is: , , ; The transformation relationship between the robot base coordinate system and the camera coordinate system is: , Among them, P robot represents the robot base coordinate system point, P w represents the calibration plate coordinate system point; R represents the rotation transformation matrix from the calibration plate coordinate system to the robot base coordinate system, r1-r9 represents the value of each element of the R matrix; T represents the translation transformation matrix from the calibration plate coordinate system to the robot base coordinate system, t x , t y , t z Respectively represent the value of each element of the T matrix; P c Represents the camera coordinate system point, P Fixture Represents the gripper coordinate system point; represents the rotation transformation matrix from the gripper coordinate system to the camera coordinate system, - express The value of each element of the matrix; represents the translation transformation matrix from the gripper coordinate system to the camera coordinate system, , , Respectively The value of each element of the matrix.

4. The method for camera calibration using a calibrated robot according to claim 1, characterized in that: The transformation formula for normalizing the point set in the camera coordinate system to the ideal image coordinate system is: , Z c Not 0, Z c Indicates the Z-axis coordinate value in the camera coordinate system, P c Represents a point set in the camera coordinate system.

5. The method for camera calibration using a calibrated robot according to claim 1, characterized in that: The radial distortion forward model considered in distortion correction is: , where is the anti-distortion point, denoted as point ; r d Represents the Euclidean distance from the distortion point to the distortion center; is the normalized ideal image coordinate point P n ; k1, k2, k3 represent the polynomial coefficients of the distortion model; Anti-distortion point The transformation to the image coordinate system is , the internal parameter conversion matrix ; Where A represents the current camera internal parameter conversion matrix, C x and C y represents the coordinates of the distortion center, Represents the point projected from the calibration plate coordinate system to the image coordinate system after a series of coordinate transformations.

6. The method for camera calibration using a calibrated robot according to claim 1, characterized in that: The method of replacing a calibrated camera with a camera to be calibrated and collecting images for calibration, and solving the parameters of the camera to be calibrated includes the following steps: installing the camera to be calibrated and rotating it to collect images; detecting corner point information to obtain corner points of the image coordinate system and corresponding points of the calibration plate coordinate system; calibrating the image to obtain multiple pixel points and obtaining a total error optimization objective function for all pixel points; solving the parameters to complete the calibration of the camera.

7. The method for camera calibration using a calibrated robot according to claim 6, characterized in that: The total error optimization objective function in the step of solving the parameters of the camera to be calibrated is ,in, is the intrinsic transformation parameter of the camera to be calibrated, is the distortion parameter of the camera to be calibrated, is the conversion relationship between the gripper coordinate system and the camera coordinate system to be calibrated, k is the number of images collected for online calibration, h is the number of points on the image, represents the jth calibration plate coordinate system point of the i-th image, is the jth coordinate point of the i-th image.

Images