A camera calibration method using a calibrated auxiliary camera

Abstract

The present invention relates to a camera calibration method using a calibrated auxiliary camera. An industrial camera with pre-calibrated internal parameters is firstly used as an auxiliary camera, and the external parameters of a camera to be tested (or DUT camera) are precisely constrained by the auxiliary camera, thereby reducing the parameter degrees of freedom in the process of solving the internal and external parameters of the camera calibration, thereby making the calibration result more accurate, fast and stable.

Description

Technical Field

[0001] The present invention relates to the technical field of camera calibration, and in particular to a camera calibration method using a calibrated auxiliary camera. Background Art

[0002] In the modern industrial field, with the development of technologies such as artificial intelligence and autonomous driving, the requirements for a series of technical indicators such as the recognition accuracy of machine vision and the accuracy of optical measurement have become extremely high. As an important component of the image acquisition system, the camera module needs to be accurately calibrated to ensure the accuracy of the subsequent visual system.

[0003] At present, the commonly used camera calibration method is the "Zhang Zhengyou calibration method", which requires moving the calibration plate to take multiple pictures in different postures relative to the camera module to ensure that the imaging has sufficient perspective distortion effect. For long focal length modules, the perspective distortion effect of the imaging calibrated by Zhang Zhengyou's calibration method is not obvious enough, resulting in insufficient constraints on the optimization parameters and inaccurate final calibration results; for images with large distortion effects, the images calibrated by Zhang Zhengyou's calibration method are severely distorted. In the process of overall optimization of internal and external parameters, the calculation results have large errors, and the calibration results are unstable, which is not conducive to the subsequent processing of the images generated by the camera. In order to achieve higher recognition accuracy and precision, it is necessary to improve the existing camera calibration method and propose a new camera calibration method. Summary of the invention

[0004] The purpose of the present invention is to design a camera calibration method using a calibrated auxiliary camera, so that the calibration result using a long focal length module is sufficiently accurate and the accuracy of image recognition for large distortion effects is improved.

[0005] To achieve the above object, the present invention provides a camera calibration method using a calibrated auxiliary camera, including offline calibration of a test environment and online calibration of a DUT camera, specifically comprising the following steps:

[0006] Offline calibration of the test environment includes:

[0007] Step 1: Calibrate the auxiliary camera and the DUT camera. Use the Zhang Zhengyou calibration method to pre-calibrate the internal parameters of the auxiliary camera and the DUT camera.

[0008] Step 2: Install the auxiliary camera and the DUT camera on the fixture in the robotic arm, and rotate the fixture through the robotic arm to make the auxiliary camera and the DUT camera take multiple calibration pictures. The position relationship model between the calibration auxiliary camera and the DUT camera is , where D, R, and C represent the coordinate system of the DUT camera, the coordinate system of the auxiliary camera, and the coordinate system of the calibration plate, respectively. It represents the coordinate system transformation matrix from the auxiliary camera to the DUT camera, and the coordinate system of the calibration board is the world coordinate system;

[0009] For the online calibration of DUT camera:

[0010] Step 3: Replace the DUT camera position with the actual product to be tested, rotate the camera module fixture through the robotic arm, take multiple calibration pictures and calculate the corner point coordinates;

[0011] Step 4: Solve the position relationship of the calibration plate relative to the auxiliary camera According to the solvePnP function provided by opencv, solve the external parameter conversion matrix of the auxiliary camera and ;

[0012] Step 5: Solve the transformation relationship from the current world coordinate system to the DUT camera coordinate system ;

[0013] Step 6: Normalize the DUT camera coordinate system to the ideal image coordinate system;

[0014] Step 7: According to Zhang Zhengyou's calibration method, establish a radial distortion forward model, and record the anti-distortion point in the model as point ;

[0015] Step 8: Solving The intrinsic conversion matrix A to the image coordinate system; and record It is the point projected onto the image coordinate system after a series of coordinate transformations in the world coordinate system;

[0016] Step 9: The coordinate point of the jth point in the i-th image is , and calculate the final pixel reprojection error ;

[0017] Step 10: Calculate the total error optimization objective function for all pixels ;

[0018] Step 11: Use a nonlinear solution method to iteratively optimize and solve the total error optimization objective function in step 10.

[0019] Furthermore, in step 3, the sub-pixel point set of the image corner points is obtained, and the corner detection function of OpenCV is called to perform sub-pixel corner detection and collinear point detection; more generally, the Harris method can be used to perform corner detection, and then the collinear detection of the corner points can be performed along the edge direction.

[0020] Furthermore, for the world coordinate point P of the calibration plate W , its coordinates in the DUT camera and auxiliary camera coordinate systems can be expressed as:

[0021] ; Formula (1)

[0022] ; Formula (2)

[0023] ; Formula (3)

[0024] Among them, R D Rotation matrix and T D The translation matrix represents the point set P w The transformation matrix from the world coordinate system of the calibration board to the DUT camera coordinate system; R R Rotation matrix and T R The translation matrix represents the point set P w The transformation matrix from the world coordinate system to the auxiliary camera coordinate system; R rotation matrix and T translation matrix represent the point set P D The transformation matrix from the auxiliary camera coordinate system to the DUT camera coordinate system;

[0025] ; Formula (4)

[0026] ; Formula (5)

[0027] Among them, the camera intrinsic parameters are pre-calibrated offline, and the external parameter matrix R can be easily solved through the solvePnP function of opencv D , T D , R R , T R , and according to equations (4) and (5), the transformation R from the auxiliary camera coordinate system to the DUT camera coordinate system can be solved T matrix.

[0028] Furthermore, in step 5, the following relationship exists:

[0029] ; Formula (6)

[0030] ; Formula (7)

[0031] ; Formula (8)

[0032] in, and They represent the world coordinate points during online calibration and the points in the DUT camera coordinate system respectively. ; The rotation matrix and The rotation matrices are represented by arrive The transformation matrix.

[0033] Further, remember Z c is the Z-axis coordinate value in the DUT camera coordinate system; when Z c When it is not 0, the camera coordinate system is normalized to the ideal image coordinate system:

[0034] ; Formula (9)

[0035] in, Represents the normalized image coordinate system.

[0036] Furthermore, the radial distortion forward model is:

[0037] ; Formula (10)

[0038] in, The anti-distortion point is denoted as point , Represents the Euclidean distance from the distortion point to the distortion center.

[0039] Furthermore, the anti-distortion point The relationship with the internal parameter conversion matrix A is:

[0040]

[0041] ; Formula (11)

[0042] Among them, A represents the current DUT camera internal parameter matrix; Cx and Cy represent the coordinates of the distortion center, Represents the point projected onto the image coordinate system after a series of coordinate transformations in the world coordinate system.

[0043] Furthermore, the pixel reprojection error is:

[0044] ; Formula (12)

[0045] Furthermore, for the m actually calibrated images (m>0) and the n points on the images, the total error optimization objective function of all pixels can be obtained as:

[0046] ; Formula (13)

[0047] ; Formula (14)

[0048] in, The normalized transformation relationship of the coordinate system is the projection model of formula (9), The distortion transformation relationship is the distortion model of formula (10), It represents the projection relationship from the anti-distortion point to the image coordinate system, that is, the camera intrinsic parameter matrix of formula (11).

[0049] Furthermore, for the nonlinear solution method, the optimization algorithms used include Gauss-Newton method, LM method, etc., to iteratively optimize and solve equation (14).

[0050] Compared with the prior art, the present invention has the following beneficial effects: the present invention provides a camera calibration method using a calibrated auxiliary camera, which uses an industrial camera with pre-calibrated internal parameters as an auxiliary camera, and uses the auxiliary camera to accurately constrain the external parameters of the camera to be tested (or DUT camera), thereby reducing the parameter freedom in the process of solving the internal and external parameters of the camera calibration, thereby making the calibration result more accurate, fast and stable. BRIEF DESCRIPTION OF THE DRAWINGS

[0051] Figure 1 Schematic diagram of a camera calibration method using an auxiliary camera;

[0052] Figure 2 This is a flow chart of camera calibration parameters using an auxiliary camera;

[0053] Figure 3 This is the calibration effect diagram;

[0054] Figure 4 To find the corner point effect diagram;

[0055] Figure 5 This is the anti-distortion effect diagram.

[0056] Among them: 1. Calibration board; 2. DUT camera; 3. Auxiliary camera; 4. Fixture. DETAILED DESCRIPTION

[0057] In order to further explain the technical means and effects adopted by the present invention to achieve the predetermined invention purpose, the specific implementation mode, structure, characteristics and effects of the present invention are described in detail below in combination with the accompanying drawings and preferred embodiments.

[0058] Example: Please refer to Figure 1-5 A camera calibration method using a calibrated auxiliary camera includes offline calibration of a test environment and online calibration of a DUT camera 2, specifically comprising the following steps:

[0059] Offline calibration of the test environment includes:

[0060] Step 1: Calibrate the auxiliary camera 3 and the DUT camera 2. Use the Zhang Zhengyou calibration method to pre-calibrate the internal parameters of the auxiliary camera 3 and the DUT camera 2. In addition, in this method, a high-precision chessboard is used as the calibration plate 1. Its function is to locate the precise sub-pixel image coordinates. The calibration plate 1 is not limited to the chessboard form. Similar patterns that can find collinear points can be used as the calibration plate 1.

[0061] Step 2: Use the robot arm to rotate the fixture 4 of the camera module, take multiple calibration pictures, and calibrate the positional relationship model between the auxiliary camera 3 and the DUT as follows: , where D, R, and C represent the coordinate system of the DUT camera 2, the coordinate system of the auxiliary camera 3, and the coordinate system of the calibration plate 1, respectively. represents the coordinate system transformation matrix from the auxiliary camera 3 to the DUT camera 2, and the coordinate system of the calibration plate 1 is the world coordinate system; the specific derivation is as follows;

[0062] For the world coordinate point P of calibration plate 1 W , its coordinates in the coordinate system of DUT camera 2 and auxiliary camera 3 can be expressed as:

[0063] ; Formula (1)

[0064] ; Formula (2)

[0065] ; Formula (3)

[0066] Among them, R D Rotation matrix and T D The translation matrix represents the point set P w The transformation matrix from the world coordinate system of calibration plate 1 to the coordinate system of DUT camera 2; R R Rotation matrix and T R The translation matrix represents the point set P w The transformation matrix from the world coordinate system to the auxiliary camera 3 coordinate system; R rotation matrix and T translation matrix represent the point set P D The transformation matrix from the auxiliary camera 3 coordinate system to the DUT camera 2 coordinate system;

[0067] ; Formula (4)

[0068] ; Formula (5)

[0069] Among them, the camera intrinsic parameters are pre-calibrated offline, and the external parameter matrix R can be easily solved through the solvePnP function of opencv D , T D , R R , T R, and according to equations (4) and (5), the transformation R from the auxiliary camera 3 coordinate system to the DUT camera 2 coordinate system can be solved T matrix.

[0070] For the online calibration of DUT camera 2, there are:

[0071] Step 3: Replace the position of DUT camera 2 with the actual product to be tested, rotate the fixture 4 of the camera module through the robotic arm, take multiple calibration pictures and calculate the coordinates of the corner points; this step can obtain the sub-pixel point set of the image corner points; call the corner point detection function of opencv to perform sub-pixel corner point detection and collinear point detection. More generally, the Harris method can be used for corner point detection, and then the collinear detection of the corner points can be performed along the edge direction.

[0072] Step 4: Solve the position relationship between calibration plate 1 and auxiliary camera 3 According to the solvePnP function provided by opencv, solve the external parameter conversion matrix of auxiliary camera 3 and ;

[0073] Step 5: Solve the transformation relationship from the current world coordinate system to the DUT camera 2 coordinate system ;have

[0074] ; Formula (6)

[0075] ; Formula (7)

[0076] ; Formula (8)

[0077] in, and They represent the world coordinate points during online calibration and the points in the DUT camera 2 coordinate system, respectively. ; The rotation matrix and The rotation matrices are represented by arrive The transformation matrix.

[0078] Step 6: Normalize the DUT camera 2 coordinate system to the ideal image coordinate system; and record Z c is the Z-axis coordinate value in the DUT camera 2 coordinate system; when Z c When it is not 0, the camera coordinate system is normalized to the ideal image coordinate system:

[0079] ; Formula (9)

[0080] in, Represents the normalized image coordinate system.

[0081] Step 7: According to Zhang Zhengyou's calibration method, the radial distortion forward model is established as follows:

[0082] ; Formula (10)

[0083] in, is the anti-distortion point, denoted as point , Represents the Euclidean distance from the distortion point to the distortion center.

[0084] Step 8: Solving The intrinsic transformation matrix to the image coordinate system;

[0085]

[0086] ; Formula (11)

[0087] Where A represents the internal reference matrix of the current DUT camera 2; Cx and Cy represent the coordinates of the distortion center. Represents the point projected onto the image coordinate system after a series of coordinate transformations in the world coordinate system.

[0088] Step 9: The coordinate point of the jth point in the i-th image is , and calculate the final pixel reprojection error ,in;

[0089] ; Formula (12)

[0090] Step 10: For the actual calibrated m images (m>0) and n points on the image, the total error optimization objective function of all pixels can be obtained as:

[0091] ; Formula (13)

[0092] ; Formula (14)

[0093] in, The normalized transformation relationship of the coordinate system is the projection model of formula (9), The distortion transformation relationship is the distortion model of formula (10), It represents the projection relationship from the anti-distortion point to the image coordinate system, that is, the camera intrinsic parameter matrix of formula (11).

[0094] Step 11: Use nonlinear solution methods, such as Gauss-Newton method, LM method, etc. to iteratively optimize and solve equation (14).

[0095] In particular, in the nonlinear optimization solution process, the pose of each image calibration plate 1 to the auxiliary camera 3 is accurately solved in advance by the PnP method. It can be used as a known quantity. After each replacement of the camera to be calibrated, the position and posture of the DUT camera 2 relative to the auxiliary camera 3 changes very little. It can be considered that the translation matrix T parameters remain unchanged, and the rotation matrix can be fine-tuned based on the pre-calibrated R parameters. In addition, the camera intrinsic parameters need to be solved.

[0096] Therefore, during the entire camera calibration process, a high-precision industrial camera is introduced as an auxiliary camera 3 to effectively constrain the calibration external parameters of the camera to be tested (or DUT camera 2); the entire process only requires solving the internal parameters of DUT camera 2 and a small amount of external parameters, which improves the parameter constraints and reduces the amount of parameter solutions, greatly improving the stability, accuracy, and speed of the final calibration results.

[0097] The camera calibration method using the calibrated auxiliary camera 3 has the following significant advantages for telephoto module calibration compared with traditional algorithms such as the “Zhang Zhengyou calibration method”:

[0098] 1. Simplified the requirements for calibration and image acquisition. The mobile mechanical gripper camera module is used to acquire images without the need for rotation. This reduces the number of images acquired, reduces the amount of data, and improves algorithm efficiency. Preferably, calibration takes nine images of a nine-square grid that cover a sufficient field of view.

[0099] 2. Auxiliary camera 3 is introduced for calibration. The function of auxiliary camera 3 is to provide accurate external parameter constraints for the camera DUT to be calibrated. It only needs to solve the calibration internal parameters of the DUT and fine-tune a small number of parameters. The calibration accuracy of telephoto lens modules and wide-angle and large-distortion modules is improved.

[0100] 3. There is no need to solve external parameters, which increases the robustness of the algorithm.

[0101] 4. The auxiliary camera 3 is introduced to constrain the external parameters. There is no need to move the calibration plate 1 to produce sufficient perspective distortion. At the same time, it can solve the problem that the equivalent focal length and Cx, Cy cannot be accurately estimated due to insufficient perspective distortion in the calibration of the telephoto module.

[0102] The above are only preferred embodiments of the present invention, and are not intended to limit the present invention in any form. Although the present invention has been disclosed as above in the form of preferred embodiments, it is not intended to limit the present invention. Any technical personnel in the field can make some changes or modify the technical contents disclosed above into equivalent embodiments without departing from the scope of the technical solution of the present invention. However, any brief modifications, equivalent changes and modifications made to the above embodiments based on the technical essence of the present invention without departing from the content of the technical solution of the present invention are still within the scope of the technical solution of the present invention.

Claims

1. A camera calibration method using a calibrated auxiliary camera, characterized in that: The following steps are included: Step 1: Calibrate the auxiliary camera (3) and the DUT camera (2), and use the Zhang Zhengyou calibration method to pre-calibrate the internal parameters of the auxiliary camera (3) and the DUT camera (2); Step 2: Install the auxiliary camera (3) and the DUT camera (2) on the fixture (4) in the robot arm, and rotate the fixture (4) through the robot arm to make the auxiliary camera (3) and the DUT camera (2) take multiple calibration pictures. The position relationship model between the calibration auxiliary camera (3) and the DUT camera (2) is: , where D, R, and C represent the coordinate system of the DUT camera (2), the coordinate system of the auxiliary camera (3), and the coordinate system of the calibration plate (1), respectively. represents the coordinate system transformation matrix from the auxiliary camera (3) to the DUT camera (2), and the coordinate system of the calibration plate (1) is the world coordinate system; For the DUT camera (2) online calibration: Step 3: Replace the position of the DUT camera (2) with the actual product to be tested, rotate the fixture (4) through the robotic arm, take multiple calibration images and calculate the corner point coordinates; Step 4: Solve the position relationship between the calibration plate (1) and the auxiliary camera (3) According to the solvePnP function provided by opencv, the external parameter transformation matrix of the auxiliary camera (3) is solved and ; Step 5: Solve the transformation relationship from the current world coordinate system to the DUT camera (2) coordinate system ; Step 6: Normalize the DUT camera (2) coordinate system to the ideal image coordinate system; Step 7: According to Zhang Zhengyou's calibration method, establish a radial distortion forward model, and record the anti-distortion point in the model as point ; Step 8: Solving The intrinsic conversion matrix A to the image coordinate system; and record It is the point projected onto the image coordinate system after a series of coordinate transformations in the world coordinate system; Step 9: The coordinate point of the jth point in the i-th image is , and calculate the final pixel reprojection error ; Step 10: Calculate the total error optimization objective function for all pixels ; Step 11: Use a nonlinear solution method to iteratively optimize and solve the total error optimization objective function in step 10.

2. The camera calibration method using a calibrated auxiliary camera according to claim 1, characterized in that: In step 3, the sub-pixel point set of the image corner is obtained, and the corner detection function of OpenCV is called to perform sub-pixel corner detection and collinear point detection.

3. The camera calibration method using a calibrated auxiliary camera according to claim 1, characterized in that: For the world coordinate point P of the calibration plate (1) W , its coordinates in the coordinate systems of the DUT camera (2) and the auxiliary camera (3) can be expressed as: ; Formula (1) ; Formula (2) ; Formula (3) Among them, R D Rotation matrix and T D The translation matrix represents the point set P w Transformation matrix from the world coordinate system to the DUT camera (2) coordinate system; R R Rotation matrix and T R The translation matrix represents the point set P w The transformation matrix from the world coordinate system to the auxiliary camera (3) coordinate system; R rotation matrix and T translation matrix represent the point set P respectively. D The transformation matrix from the auxiliary camera (3) coordinate system to the DUT camera (2) coordinate system; ; Formula (4) ; Formula (5) Among them, the camera intrinsic parameters are pre-calibrated offline, and the external parameter matrix R can be easily solved through the solvePnP function of opencv D , T D , R R , T R , and according to equations (4) and (5), the transformation R from the auxiliary camera (3) coordinate system to the DUT camera (2) coordinate system can be solved T matrix.

4. The camera calibration method using a calibrated auxiliary camera according to claim 3, characterized in that: In step 5, the following relationship exists: ; Formula (6) ; Formula (7) ; Formula (8) in, and They represent the world coordinate points and the points in the DUT coordinate system during online calibration, ; The rotation matrix and The rotation matrices are represented by arrive The transformation matrix.

5. The camera calibration method using a calibrated auxiliary camera according to claim 4, characterized in that: Remember Z c is the Z-axis coordinate value in the DUT camera (2) coordinate system; when Z c When it is not 0, the camera coordinate system is normalized to the ideal image coordinate system: ; Formula (9) in, Represents the normalized image coordinate system.

6. The camera calibration method using a calibrated auxiliary camera according to claim 5, characterized in that: The radial distortion forward model is: ; Formula (10) in; The anti-distortion point is denoted as point , Represents the Euclidean distance from the distortion point to the distortion center.

7. The camera calibration method using a calibrated auxiliary camera according to claim 6, characterized in that: Anti-distortion point The relationship with the internal parameter conversion matrix A is: ; Formula (11) Where A represents the internal parameter matrix of the current DUT camera (2); Cx and Cy represent the coordinates of the distortion center, Represents the point projected onto the image coordinate system after a series of coordinate transformations in the world coordinate system.

8. The camera calibration method using a calibrated auxiliary camera according to claim 7, characterized in that: The pixel reprojection error is: ; Formula (12).

9. The camera calibration method using a calibrated auxiliary camera according to claim 8, characterized in that: For the actual calibrated m images (m>0) and n points on the image, the total error optimization objective function of all pixels can be obtained as: ; Formula (13) ; Formula (14) in, The normalized transformation relationship of the coordinate system is the projection model of formula (9), The distortion transformation relationship is the distortion model of formula (10), It represents the projection relationship from the anti-distortion point to the image coordinate system, that is, the camera intrinsic parameter matrix of formula (11).

10. The camera calibration method using a calibrated auxiliary camera according to claim 9, characterized in that: For the nonlinear solution method, the optimization algorithms used are Gauss-Newton method or LM method to iteratively optimize and solve equation (14).

Images