Camera calibration method and device based on two-dimensional calibration object and iteration

Abstract

The invention relates to a camera calibration method and device based on a two-dimensional calibration object and iteration, and the method comprises the steps: a to-be-calibrated camera shoots an image of a calibration plate at multiple angles, and the calibration plate is provided with a plurality of circular patterns which reflect infrared light; extracting circle center coordinates of the circular pattern on the calibration plate in each image; performing initial calibration on the camera by using the extracted circle center coordinate and the actual size of the calibration plate; converting the calibration plate image into a front view by using an initial calibration result, re-extracting a circle center coordinate on the front view, and back-projecting the extracted coordinate to an original image coordinate system; recalibrating the camera, and binding and adjusting the parameters; and circularly executing the coordinate extraction, the back projection and the parameter binding adjustment until the calibration result reaches the set precision requirement. Compared with the prior art, the problem that an infrared camera cannot use a common calibration plate is solved, and the calibration precision of the camera is improved through the iteration method of extracting the circle center in the front view.

Description

Technical Field

[0001] The invention relates to the field of optical cameras and camera calibration, and in particular to a method for calibrating multiple infrared cameras by using a circular calibration plate. Background Art

[0002] Camera calibration is an important step in computer vision, especially in the fields of 3D reconstruction and motion capture. Camera calibration is an inevitable first step. The purpose of calibration is to build a mathematical model of the multi-camera system, that is, to calculate the internal and external parameters of the camera, including the focal length, principal point offset, distortion coefficient, and the position and posture of the camera in the world coordinate system. Camera calibration requires the use of external tools, among which two-dimensional calibration objects are more commonly used because they are simple to make and can achieve higher accuracy. In simple terms, a two-dimensional calibration object is a board. The camera is calibrated using the measurement information of the marking points on the board. The board pattern is generally a checkerboard or a number of regularly arranged circles.

[0003] For a long time, infrared cameras have not been able to achieve high accuracy in photographing calibration objects, because calibration plates are mostly made of ceramics or glass. However, infrared cameras cannot clearly capture the patterns on calibration plates made of ordinary materials, resulting in inaccurate extraction of calibration points on the calibration plates, resulting in poor calibration accuracy. Summary of the invention

[0004] The purpose of the present invention is to overcome the defect that the conventional infrared camera does not clearly photograph the common calibration plate, resulting in poor calibration accuracy, and to provide a camera calibration device and method based on two-dimensional calibration objects and iterations.

[0005] The purpose of the present invention can be achieved by the following technical solutions:

[0006] As a first aspect of the present invention, a camera calibration method based on a two-dimensional calibration object and iteration is provided, the method steps comprising:

[0007] The camera to be calibrated takes images of a calibration plate at multiple angles, where a plurality of circular patterns reflecting infrared light are arranged on the calibration plate;

[0008] Extract the center coordinates of the circular pattern on the calibration plate in each image;

[0009] The camera is initially calibrated using the extracted circle center coordinates and the actual size of the calibration plate;

[0010] The calibration plate image is converted into a front view using the initial calibration result, the coordinates of the circle center are re-extracted on the front view, and the extracted coordinates are back-projected into the original image coordinate system;

[0011] Recalibrate the camera using the new image coordinates and adjust the parameters in a bundled manner;

[0012] The above-mentioned coordinate extraction, back-projection and parameter bundling adjustment are executed repeatedly until the calibration result reaches the set accuracy requirement.

[0013] As a preferred technical solution, the photographing of the calibration plate image is specifically as follows:

[0014] With the camera fixed, place the calibration plate in different postures to ensure that each camera can capture a complete image of the calibration plate and multiple cameras capture images at the same time.

[0015] As a preferred technical solution, the coordinates of the center of the circular pattern on the calibration plate are extracted as follows:

[0016] On the photographed calibration plate pattern, the pixel coordinates of the center of each circular pattern on the image are identified; the pixel coordinates of the edge of each circle are detected, and a circle is fitted using the edge pixels, with the center of the circle as the center coordinate.

[0017] As a preferred technical solution, the initial calibration of the camera specifically includes calculating the intrinsic parameters and extrinsic parameters of the camera;

[0018] The intrinsic parameters include camera focal length, principal point offset and distortion coefficient;

[0019] The extrinsic parameters include the position and orientation of the camera relative to the world coordinate system.

[0020] As a preferred technical solution, the method of converting the calibration plate image into a front view using the initial calibration result is specifically as follows:

[0021] Use the position relationship of the calibration plate relative to the camera to convert the image from the pixel coordinate system to the calibration plate coordinate system;

[0022] The calibration plate image is converted into a front view using the perspective relationship, and the homography matrix H from the original image to the front view is obtained.

[0023] As a preferred technical solution, the method of re-extracting the coordinates of the circle center on the front view and then inversely mapping them back to the original image coordinate system is specifically as follows:

[0024] The position of the circle center is extracted on the front view, and after the center of the circle is extracted, the inverse matrix of the homography matrix H is used to map the center of the circle back to the original image coordinate system.

[0025] As a preferred technical solution, the bundling adjustment is specifically as follows:

[0026] Taking the calibration result as the initial value, all parameters are put together for nonlinear iterative optimization with the purpose of minimizing the reprojection error. The cost function is:

[0027]

[0028] Among them, x ij is the projection of the jth point on the calibration plate on the i-th camera, K i is the intrinsic parameter matrix of the i-th camera, R i and T i is the external parameter matrix of the i-th camera, X j is the jth point on the calibration plate; error is the reprojection error, which refers to the error between the coordinates of the circle on the calibration plate in the 3D space, after being converted to the image coordinate system through the camera model, and the center value directly extracted from the image.

[0029] As an optimal technical solution, the parameters of the bundling adjustment include: the intrinsic parameter matrix of each camera includes: the focal length and principal point of the camera; the distortion coefficient of each camera includes: the distortion coefficient includes radial distortion and tangential distortion; the external parameters of each camera include: rotation matrix and displacement matrix; the homography matrix of each image mapped from the original image to the front view.

[0030] As a preferred technical solution, the step of until the calibration result reaches the set accuracy requirement is specifically:

[0031] Set the reprojection error threshold. Calculate the reprojection error of the calibration plate each time after the bundle adjustment. If it is greater than the threshold, return to step 4 and continue to iterate and loop to perform the above coordinate extraction and back projection and parameter bundle adjustment until the iterative result is less than the set threshold.

[0032] As a second aspect of the present invention, a camera calibration device based on a two-dimensional calibration object and iteration is provided, and the device is used to implement the camera calibration method based on a two-dimensional calibration object and iteration as described above. The calibration device consists of a calibration plate with multiple circular patterns regularly arranged on a plane, and the circular patterns are made of a material that can reflect infrared light.

[0033] Compared with the prior art, the present invention has the following beneficial effects:

[0034] The present invention makes a calibration plate that can reflect infrared light, which greatly improves the shooting clarity of the infrared camera and ensures the accuracy of extracting the center of the circle. In addition, an iterative optimization algorithm is provided to convert the captured image into a front view to extract the center of the circle, and then inversely map it back to the original coordinate system. Through repeated iterations, the accuracy of camera calibration is further improved. BRIEF DESCRIPTION OF THE DRAWINGS

[0035] Figure 1 It is a schematic diagram of the process of the online continuous calibration method of the multi-camera system of the present invention;

[0036] Figure 2 It is a circular calibration plate made of reflective material. DETAILED DESCRIPTION

[0037] The present invention is described in detail below in conjunction with the accompanying drawings and specific embodiments. This embodiment is implemented based on the technical solution of the present invention, and provides a detailed implementation method and specific operation process, but the protection scope of the present invention is not limited to the following embodiments.

[0038] Example 1

[0039] like Figure 1 As shown, the embodiment of the present invention provides a camera calibration method based on a two-dimensional calibration object and iteration, and the specific steps are as follows:

[0040] Step 1: The camera to be calibrated takes images of the calibration plate. The calibration plate needs to be placed in different positions, and each camera needs to take pictures at the same time, which requires about 15 to 30 pictures.

[0041] The camera takes pictures of the calibration plate at different angles, which means that the calibration plate is placed in different postures while the camera is fixed, so as to ensure that each camera can capture the complete image of the calibration plate and take the image at the same time.

[0042] The two-dimensional calibration object is a calibration plate composed of circular patterns. The general calibration plate is made of ceramic or glass, but for infrared cameras, the pattern on the calibration plate made of ordinary materials cannot be clearly photographed, thus reducing the calibration accuracy. Therefore, in order to meet the high-precision calibration of infrared cameras, a new type of calibration plate is made.

[0043] like Figure 2 The figure shows a new type of calibration plate made by the present invention. The circular pattern of the calibration plate can reflect infrared light, and the image taken by the infrared camera is very clear. The specific production steps of the calibration plate are as follows:

[0044] Step 101: Cut out circular patterns on a flat piece of iron sheet. The radius of each circular pattern and the distance between them are set manually.

[0045] Step 102: Stick a layer of reflective sticker on a flat board.

[0046] Step 103: Cover the prepared iron pattern on the board to produce a circular pattern calibration plate that reflects infrared light.

[0047] Step 2: Extract the center coordinates of the circular pattern on each image. On the captured calibration plate pattern, identify the pixel coordinates of the center of each circular pattern on the image. The method used is to detect the pixel coordinates of the edge of each circle, fit a circle using the edge pixels, and use the center of the circle as the center coordinates.

[0048] Step 3: Use the center coordinates extracted in step 2 and the actual size of the calibration plate to calibrate the camera, including the camera's intrinsic and extrinsic parameters. Intrinsic parameters are the inherent properties of the camera, which are generally considered to not change significantly after leaving the factory, including the camera's focal length, principal point offset, and distortion coefficient. Extrinsic parameters refer to the position and direction relative to the world coordinate system. Calculate the focal length, principal point offset, distortion coefficient, and position and direction of all cameras to the world coordinate system.

[0049] Step 4: Use the calibration result to convert the captured calibration plate image into a front view. Use the position relationship of the calibration plate relative to the camera to convert the image from the pixel coordinate system to the calibration plate coordinate system, and then use the perspective relationship to convert the calibration plate image into a front view. The homography matrix from the original image to the front view is recorded as H.

[0050] Step 5: Re-extract the coordinates of the circle center on the front view, and then back-project the extracted coordinates to the original image coordinate system.

[0051] After converting the calibration plate image into a front view using the perspective relationship, extract the position of the circle center on the front view, and then use the inverse matrix of H to map the position of the circle center back to the original image coordinate system. This is because the standard circular pattern will be photographed as an ellipse by the camera from different angles, and the center position of the ellipse cannot accurately represent the center position of the real circular pattern, so the original image is first converted into a front view, the center position is extracted, and then mapped back. This will improve the accuracy of circular position extraction.

[0052] Step 6: Use the new image coordinate system obtained in step 5 to recalibrate the camera and adjust various parameters.

[0053] Recalibrating the camera specifically involves recalculating the internal and external parameters of the camera using the data obtained in step 4.

[0054] The parameters for bundling adjustment include:

[0055] The intrinsic parameter matrix K of each camera i (i=1,2...n) includes: focal length and principal point of the camera.

[0056] Distortion coefficient D of each camera i (i=1,2...n) includes: The distortion coefficients include radial distortion and tangential distortion.

[0057] The external parameters of each camera include:

[0058] R i (i=1,2...n)

[0059] T i (i=1,2...n)

[0060] Among them, R i is the rotation matrix, T i is the displacement matrix.

[0061] The homography matrix H of each image, where H is the homography matrix mapping the original image to the front view.

[0062] Bundling adjustment means taking the calibration result as the initial value, putting all parameters together for nonlinear iterative optimization with the purpose of minimizing the reprojection error. The cost function is:

[0063]

[0064] Among them, x ij is the projection of the jth point on the calibration plate on the i-th camera, K i is the intrinsic parameter matrix of the i-th camera, R i and T i is the external parameter matrix of the i-th camera, X j is the jth point on the calibration plate.

[0065] Error is the reprojection error, which refers to the error between the coordinates of the circle on the fixed plate in the 3D space, after being converted to the image coordinate system through the camera model, and the center value directly extracted from the image.

[0066] Step 7: Determine whether the quality of camera calibration meets the requirements. If not, loop through steps 4 and 5 until the calibration result meets the set accuracy requirements. This means setting a reprojection error threshold in advance. Each time after the bundle adjustment, calculate the reprojection error of the calibration plate. If it is greater than the threshold, continue to iterate and return to step 4 until the result of the iteration is less than the set threshold. In each iteration, each parameter will be closer to the true value, and ultimately the camera calibration will reach a higher level of accuracy.

[0067] The preferred specific embodiments of the present invention are described in detail above. It should be understood that a person skilled in the art can make many modifications and changes based on the concept of the present invention without creative work. Therefore, any technical solution that can be obtained by a person skilled in the art through logical analysis, reasoning or limited experiments based on the concept of the present invention on the basis of the prior art should be within the scope of protection determined by the claims.

Claims

1. A camera calibration method based on two-dimensional calibration objects and iterations, characterized in that: The method steps include: The camera to be calibrated takes images of a calibration plate at multiple angles, wherein the calibration plate is provided with a plurality of circular patterns reflecting infrared light; Extract the center coordinates of the circular pattern on the calibration plate in each image; The camera is initially calibrated using the extracted circle center coordinates and the actual size of the calibration plate; The calibration plate image is converted into a front view using the initial calibration result, the coordinates of the circle center are re-extracted on the front view, and the extracted coordinates are back-projected into the original image coordinate system; Recalibrate the camera using the new image coordinates and adjust the parameters in a bundled manner; The above-mentioned coordinate extraction, back-projection and parameter bundling adjustment are executed repeatedly until the calibration result reaches the set accuracy requirement.

2. The camera calibration method based on two-dimensional calibration objects and iteration according to claim 1, characterized in that: The photographing of the calibration plate image is specifically as follows: With the camera fixed, place the calibration plate in different postures to ensure that each camera can capture a complete image of the calibration plate and that multiple cameras capture images at the same time.

3. The camera calibration method based on two-dimensional calibration objects and iteration according to claim 1, characterized in that: The coordinates of the center of the circular pattern on the calibration plate are specifically: On the photographed calibration plate pattern, the pixel coordinates of the center of each circular pattern on the image are identified; the pixel coordinates of the edge of each circle are detected, and a circle is fitted using the edge pixels, with the center of the circle as the center coordinate.

4. The camera calibration method based on two-dimensional calibration objects and iteration according to claim 1, characterized in that: The initial calibration of the camera specifically includes calculating the intrinsic parameters and extrinsic parameters of the camera; The intrinsic parameters include camera focal length, principal point offset and distortion coefficient; The extrinsic parameters include the position and orientation of the camera relative to the world coordinate system.

5. The camera calibration method based on two-dimensional calibration objects and iteration according to claim 1, characterized in that: The method of converting the calibration plate image into a front view using the initial calibration result is specifically as follows: Use the position relationship of the calibration plate relative to the camera to convert the image from the pixel coordinate system to the calibration plate coordinate system; The calibration plate image is converted into a front view using the perspective relationship, and the homography matrix H from the original image to the front view is obtained.

6. The camera calibration method based on two-dimensional calibration objects and iteration according to claim 1, characterized in that: The process of re-extracting the coordinates of the circle center on the front view and then inversely mapping them back to the original image coordinate system is as follows: The position of the circle center is extracted on the front view, and after the center of the circle is extracted, the inverse matrix of the homography matrix H is used to map the center of the circle back to the original image coordinate system.

7. The camera calibration method based on two-dimensional calibration objects and iteration according to claim 1, characterized in that: The bundle adjustment is specifically as follows: Taking the calibration result as the initial value, all parameters are put together for nonlinear iterative optimization with the purpose of minimizing the reprojection error. The cost function is: Among them, x ij is the projection of the jth point on the calibration plate on the i-th camera, K i is the intrinsic parameter matrix of the i-th camera, R i and T i is the external parameter matrix of the i-th camera, X j is the jth point on the calibration plate; error is the reprojection error, which refers to the error between the coordinates of the circle on the calibration plate in the 3D space, after being converted to the image coordinate system through the camera model, and the center value directly extracted from the image.

8. The camera calibration method based on two-dimensional calibration objects and iteration according to claim 7, characterized in that: The parameters of the bundle adjustment include: the intrinsic parameter matrix of each camera includes: the focal length and principal point of the camera; the distortion coefficient of each camera includes: the distortion coefficient includes radial distortion and tangential distortion; the external parameters of each camera include: rotation matrix and displacement matrix; the homography matrix of each image mapped from the original image to the front view.

9. The camera calibration method based on two-dimensional calibration objects and iteration according to claim 1, characterized in that: The calibration result reaches the set accuracy requirement specifically as follows: Set the reprojection error threshold. Calculate the reprojection error of the calibration plate each time after the bundle adjustment. If it is greater than the threshold, return to step 4 and continue to iterate and loop to perform the above coordinate extraction and back projection and parameter bundle adjustment until the iterative result is less than the set threshold.

10. A camera calibration device based on a two-dimensional calibration object and iteration, characterized in that: The device is used to implement the camera calibration method based on two-dimensional calibration objects and iterations as described in any one of claims 1-9, and the calibration device consists of a calibration plate with multiple circular patterns regularly arranged on a plane, and the circular patterns are made of a material that can reflect infrared light.

Images