Lens distortion calibration method and calibration system based thereon

Abstract

The present invention provides a method for calibrating lens distortion in a stereo vision system comprising a left camera and a right camera. In one embodiment, the method includes the steps of: a) acquiring images of corner points using the stereo vision system, wherein the corner points are formed by the intersection of m horizontal lines and n vertical lines arranged on a plane, and the actual distance between adjacent corner points is accurately known; b) extracting the image coordinates of the corner points from the set of images and calculating a set of intrinsic parameters, extrinsic parameters, and distortion parameters for the left camera and the right camera through a homography matrix; and c) obtaining a set of corrected image coordinates (I) and (II) using the set of intrinsic parameters, extrinsic parameters, and distortion parameters from step (b) and a lens distortion model.

Description

TECHNICAL FIELD

[0001] The present invention relates to computer vision measurement technology, and in particular to a depth-dependent lens distortion model and a calibration method thereof. BACKGROUND ART

[0002] Stereo vision measurement technology obtains 3D information of an object by mapping image information and imaging model parameters. Compared with conventional measurement methods, this technology has the advantages of wide measurement range, high versatility, high precision and non-contact property, and is widely used in various fields such as 3D measurement. Calibration of a stereo vision measurement system refers to calibrating the intrinsic parameters and extrinsic parameters of a camera, as well as the distortion parameters of a lens. Lens distortion parameters are closely related to the depth information of an object, but existing depth-dependent lens distortion models are very complex, which consequently complicates the calibration process and reduces practicality. In the present invention, the problem is overcome by a simplified depth-dependent lens distortion model, and a high-precision and high-efficiency calibration method is implemented. The present invention is of great significance for improving the accuracy and practicality of vision measurement systems.

[0003] In 2021, Li Xiao et al. from China University of Petroleum (East China) disclosed "calibration method of depth of field distortion model based on target loose attitude Constraint" (CN113781581A). The depth of field distortion model of Li Xiao et al. is very complex and has many unknown parameters, which consequently results in a complicated calibration process. Furthermore, depth information of a cooperative target is also required. Therefore, this method has low practicability.

[0004] Furthermore, in 2017, Liu Wei et al. from Dalian University of Technology disclosed "A high-precision stereo vision measurement method based on depth of field compensation" (CN107144241A). Liu Wei et al. developed a radial distortion compensation model in the depth of field direction. In order to obtain distortion parameters at different depths using this model, the optical axis must be perpendicular to the plane of reference. Therefore, this measurement method is difficult to implement and has low practical applicability. [Overview of the project] [Means for solving the problem]

[0005] The present invention provides a method for calibrating lens distortion in a stereo vision system comprising a left camera and a right camera. In one embodiment, the method comprises: a) acquiring images of corner points using the stereo vision system, wherein the corner points are formed by the intersection of m horizontal lines and n vertical lines arranged on a plane, and the actual distance between adjacent corner points is accurately known; b) extracting the image coordinates of the corner points from the set of images and calculating a set of intrinsic parameters, extrinsic parameters, and distortion parameters for the left camera and the right camera through a homography matrix; and c) the set of intrinsic parameters, extrinsic parameters, and distortion parameters from step (b) and the following equation (2).

[0006]

number

[0007]

number

[0008]

number

[0009]

number

[0010] The present invention also provides a stereo vision system comprising a computing device for carrying out the method of the present invention; a computer vision measurement method using the stereo vision system of the present invention; a non-temporary computer-readable medium storing program instructions that, when executed by the computing device, cause the computing device to perform an operation for lens distortion calibration of the stereo vision system, including an operation of the method of the present invention; and a computing device comprising a) a processor; b) memory; and c) program instructions stored in memory that, when executed by the processor, cause the computing device to perform an operation for lens distortion calibration of the stereo vision system, including an operation of the method of the present invention. [Brief explanation of the drawing]

[0011] [Figure 1] This invention demonstrates that the simplified depth-dependent lens distortion model calibration method employs the most commonly used checkerboard. [Modes for carrying out the invention]

[0012] Existing depth-dependent lens distortion models are complex, involving many unknown parameters, which complicates the calibration process. The simplified depth-dependent lens distortion model and calibration method of the present invention successfully overcome this problem. The simplified model in the present invention has a simple structure, fewer parameters, and is easy to calibrate. The calibration method in the present invention operates in the following four steps: (1) A stereo vision system is used to photograph checkerboards in various orientations. (2) Zhang's calibration method is used to calibrate the intrinsic and extrinsic parameters of the camera. (3) The parameters of the simplified depth-dependent lens distortion model are preliminaryly optimized based on linear constraints on the checkerboard plane. (4) Using the parameters preliminaryly optimized in the above steps as initial values, the intrinsic and extrinsic parameters and distortion parameters of the camera are overall optimized with the aim of optimizing the minimum average RMSE between the actual distance and the reconstructed distance of the corner points on the checkerboard plane. Then, the optimal values ​​for all parameters are obtained. This calibration method improves practicality by ensuring the calibration accuracy of a simplified depth-dependent lens distortion model and simplifying the calibration process.

[0013] This invention provides a simplified depth-dependent lens distortion model and a calibration method thereof, belonging to the field of computer vision measurement technology. Lens distortion is one of the main causes of image distortion. Most lens distortion models ignore the effect of depth, resulting in low measurement accuracy in the depth of field direction. Lens distortion models that take the effect of depth into account are extremely complex. This makes the calibration process complex, resulting in reduced practicality. This invention provides a simplified depth-dependent lens distortion model and a simple method for calibrating its parameters. The calibration method consists of the following four steps: First, a stereo vision system is used to photograph a checkerboard in various poses. Second, Zhang's calibration method is used to calibrate the intrinsic and extrinsic parameters of the camera. Third, the parameters of the simplified depth-dependent lens distortion model are preliminaryly optimized based on linear constraints on the checkerboard plane. Fourth, by (1) using the average RMSE (root mean square error) between the actual distance and the reconstructed distance of the corner points on the checkerboard plane as the objective function, and (2) employing multi-objective optimization, more accurate intrinsic and extrinsic parameters, as well as simplified depth-dependent lens distortion parameters, are obtained.

[0014] The present invention includes a simplified depth-dependent lens distortion model and a method for calibrating the same.

[0015] 1. Simplified depth-dependent lens distortion model

[0016] The simplified depth-dependent lens distortion model in this invention was developed by simplifying the distortion model proposed by Fraser and Shortis in "Variation of distortion within the photographic field" in 1992. The equation for this simplified depth-dependent lens distortion model is as follows:

[0017]

number

[0018] [Num] is a radial lens distortion parameter,

[0019] [Num] is a decentering lens distortion parameter,

[0020] [Num] is an image coordinate before correction,

[0021] [Num] is a corrected image coordinate,

[0022] [Num] is

[0023] [Num] is a distortion function of an image point in each direction,

[0024] [Num] is a coordinate of a distortion center,

[0025] [Num] is a distortion radius of an image point,

[0026] [Num] is depth information of an image point.

[0027] 2. Calibration of a Simplified Depth-Dependent Lens Distortion Model

[0028] (1) Image acquisition

[0029] A stereo vision system typically consists of two cameras and two lenses. The simplified depth-dependent lens distortion model calibration method in this invention employs the most commonly used checkerboard, as shown in Figure 1. The checkerboard has m horizontal lines and n vertical lines. Two lines intersect to form corner points. The distance between adjacent corner points is precisely known. In the image acquisition process, the lenses and cameras are first fixed to an optical platform, and the checkerboard is randomly placed at multiple positions within a certain depth range. In each pose, the two cameras simultaneously capture images of the checkerboard. A total of 2 × w checkerboard images were collected.

[0030] (2) Calibration of the camera's intrinsic and extrinsic parameters

[0031] The intrinsic and extrinsic parameters of two cameras are calibrated using Zhang's calibration method, proposed in 2000 in "A flexible new technique for camera calibration." Image coordinates of corner points in a checkerboard image are extracted, and initial values ​​for the camera's intrinsic and extrinsic parameters are calculated through homography matrices.

[0032] (3) Preliminary optimization of simplified depth-dependent lens distortion model parameters

[0033]

number

[0034]

number

[0035]

number

[0036]

number

[0037]

number

[0038]

number

[0039]

number

[0040]

number

[0041] As shown in Figure 1, if the corner points on the same straight line in the left and right checkerboard images are known, the equation of the straight line determined by the corner points in the left and right image coordinate systems is as follows:

[0042]

number

[0043]

number

[0044]

number

[0045]

number

[0046]

number

[0047]

number

[0048]

number

[0049]

number

[0050]

number

[0051]

number

[0052]

number

[0053] If an image (left or right image) contains i lines and j observation points are extracted from each line, then there can be i·j equations. These equations have i·2+8 variables (i·2 line parameters and 8 strain parameters). If i·j > i·2+8, the optimal solution for the strain parameters is obtained.

[0054]

number

[0055]

number

[0056]

number

[0057]

number

[0058]

number

[0059]

number

[0060]

number

[0061]

number

[0062]

number

[0063]

number

[0064]

number

[0065] (4) Overall optimization of the camera's intrinsic and extrinsic parameters, and the simplified depth-dependent lens distortion model parameters.

[0066] First, the corrected points are reconstructed in 3D using the following formula in image coordinates.

[0067]

number

[0068]

number

[0069] Equation (7) below is obtained by combining equations (5) and (6).

[0070]

number

[0071]

number

[0072] In equations (5), (6), and (7),

[0073]

number

[0074]

number

[0075]

number

[0076]

number

[0077]

number

[0078]

number

[0079]

number

[0080]

number

[0081]

number

[0082]

number

[0083]

number

[0084]

number

[0085] Next, the objective function is proposed as shown in equation (9).

[0086]

number

[0087] Finally, the Levenberg-Marquardt optimization method is employed to optimize the intrinsic and extrinsic parameters of the camera, as well as the simplified depth-dependent lens distortion model parameters. The optimized initial values ​​are the intrinsic and extrinsic parameters of the camera obtained by Zhang's calibration method, and the simplified depth-dependent lens distortion model parameters obtained by linear constraints. According to the optimization variables, optimized initial values, and optimization objective function described above, the optimized values ​​of the intrinsic and extrinsic parameters of the camera, and the simplified depth-dependent lens distortion model parameters are obtained.

[0088] Examples

[0089] The following embodiment is used to illustrate a simplified calibration method for the depth-dependent lens distortion model in the present invention.

[0090] 1. Calibration of a simplified depth-dependent lens distortion model

[0091] (1) Image acquisition

[0092] The simplified depth-dependent lens distortion model calibration system consists of two cameras, two 12mm prime lenses, and a 12×9 (150mm×105mm) checkerboard. Eight horizontal and eleven vertical lines are distributed across the checkerboard, and these lines intersect to form a total of 88 corner points. The distance between adjacent corner points is 15mm. The camera resolution is 2464 pixels × 2056 pixels. The image acquisition process begins by fixing the lenses and cameras onto an optical platform. The checkerboard is then randomly placed in multiple poses within a given depth range. The checkerboard and its corner points only need to cover the widest possible depth, while simultaneously ensuring that all corner points on the checkerboard plane are clearly imaged. For each pose, the two cameras simultaneously capture images of the checkerboard. Finally, 26 images of the checkerboard in various poses are collected.

[0093] (2) Calibration of the camera's intrinsic and extrinsic parameters

[0094] The intrinsic and extrinsic parameters are calibrated using Zhang's calibration method. First, the image coordinates of the corner points in the checkerboard image are extracted, and then the initial values ​​of the camera's intrinsic and extrinsic parameters are calculated through the homography matrix. The parameters are shown in Table 1.

[0095] [Table 1]

[0096] (3) Preliminary optimization of simplified depth-dependent lens distortion model parameters

[0097] Firstly, according to equation (2), the corrected image coordinates of the i-th row and j-th column corner point in the checkerboard image are obtained in the left and right image coordinate systems. Secondly, using the intrinsic and extrinsic parameters in Table 1 and the image coordinates in the checkerboard image corrected by the simplified depth-dependent lens distortion model, the 3D information is reconstructed by employing equation (8). Finally, as shown in equation (4), a linear constraint relationship is established between the corner points on the same line in the left and right images in the left and right image coordinate systems, thereby achieving a simplified depth-dependent preliminary optimization of the lens distortion model parameters, as shown in Table 2.

[0098] [Table 2]

[0099] (4) Overall optimization of the camera's intrinsic and extrinsic parameters, and the simplified depth-dependent lens distortion model parameters.

[0100] The objective of optimization is to minimize the mean RMSE between the actual distance and the reconstructed distance of the corner points on the checkerboard plane, and the objective function is given by equation (9).

[0101] The Levenberg-Marquardt optimization method is employed to optimize the intrinsic and extrinsic parameters of the camera, as well as the simplified depth-dependent lens distortion model parameters. The intrinsic and extrinsic parameters of the camera and the simplified depth-dependent lens distortion model parameters obtained by preliminary optimization, as shown in Tables 1 and 2, are used as optimized initial values. As shown in Tables 3 and 4, the optimized values ​​of the intrinsic and extrinsic parameters of the camera and the simplified depth-dependent lens distortion model parameters are obtained according to the optimization variables, optimized initial values, and optimization objective function described above.

[0102] [Table 3]

[0103] [Table 4]

[0104] To verify the validity of the calibration method, first, five checkerboard images are collected sequentially at different poses and depths without changing the positions of the left and right cameras. Next, the image coordinates of the corner points in the checkerboard images are extracted. In this invention, the optimized values ​​in Tables 3 and 4 are substituted into a simplified depth-dependent lens distortion model. Then, image coordinates considering lens distortion are obtained. Finally, the 3D information of the collinear corner points in the checkerboard image is obtained by equation (6). The average RMSE between the actual distance of the corner points on the line on the checkerboard plane and various geometric distances is used as the error evaluation criterion.

[0105] Select different corner points on the same line on the checkerboard plane such that their actual geometric distances are 15 mm, 75 mm, and 150 mm. Calibrate the lens distortion parameters using Zhang's calibration method and the calibration method of the present invention, respectively. Reconstruct the corner points of the checkerboard image in 3D using equation (6). Calculate the 3D reconstructed geometric distance between two corner points, and the result is as follows:

[0106] When the geometric distance between two corner points is 15 mm and the average depth change is 8.75 mm, the difference between the geometric distance optimized by the calibration method of the present invention and the actual distance between the two corner points is 0.0057 mm, and the difference between the geometric distance optimized by Zhang's calibration method and the actual distance between the two corner points is 0.0170 mm.

[0107] When the geometric distance between two corner points is 75 mm and the average depth change is 43.77 mm, the difference between the geometric distance optimized by the calibration method of the present invention and the actual distance between the two corner points is 0.0255 mm, and the difference between the geometric distance optimized by Zhang's calibration method and the actual distance between the two corner points is 0.0315 mm.

[0108] When the geometric distance between two corner points is 150 mm and the average depth change is 87.51 mm, the difference between the geometric distance optimized by the calibration method of the present invention and the actual distance between the two corner points is 0.0315 mm, and the difference between the geometric distance optimized by Zhang's calibration method and the actual distance between the two corner points is 0.0625 mm.

[0109] The results above demonstrate that a simplified depth-dependent lens distortion model can ensure accuracy in camera calibration, simplify the calibration process, and improve practicality.

[0110] The present invention provides a simplified depth-dependent radial lens distortion model. In one embodiment, the above radial lens distortion model is shown below, and the simplified depth-dependent radial lens distortion model parameters have a first-order linear relationship with depth information.

[0111]

number

[0112]

number

[0113]

number

[0114]

number

[0115]

number

[0116]

number

[0117]

number

[0118]

number

[0119] The present invention also provides a simplified depth-dependent eccentric lens distortion model. In one embodiment, the parameters of the simplified depth-dependent eccentric lens distortion model have a nonlinear relationship with depth information, as shown below.

[0120]

number

[0121]

number

[0122]

number

[0123]

number

[0124]

number

[0125]

number

[0126]

number

[0127]

number

[0128] The present invention also provides a simplified calibration method for a depth-dependent lens distortion model. In one embodiment, the above calibration method includes the following steps: Step 1: Image acquisition A stereo vision system typically consists of two cameras and two lenses. The simplified depth-dependent lens distortion model calibration method in the present invention employs the most commonly used checkerboard, as shown in Figure 1. On the checkerboard, m horizontal lines and n vertical lines are arranged. Two lines intersect to form corner points. The distance between adjacent corner points is precisely known. In the image acquisition process, the lenses and cameras are first fixed to an optical platform, and the checkerboard is randomly placed at multiple positions within a certain depth range. In each pose, the two cameras simultaneously capture images of the checkerboard. A total of 2 × w checkerboard images were collected. Step 2: Calibration of intrinsic and extrinsic parameters of the cameras The intrinsic and extrinsic parameters of the two cameras are calibrated using Zhang's calibration method, proposed in "A flexible new technique for camera calibration" in 2000. Step 3: Extract the image coordinates of the corner points in the checkerboard image and calculate the initial values ​​of the camera's intrinsic and extrinsic parameters through the homography matrix. Step 3: Preliminary optimization of the simplified depth-dependent lens distortion model parameters.

[0129]

number

[0130]

number

[0131]

number

[0132]

number

[0133]

number

[0134]

number

[0135]

number

[0136]

number

[0137] As shown in Figure 1, if the corner points on the same straight line in the left and right checkerboard images are known, the equation of the straight line determined by the corner points in the left and right image coordinate systems is as follows:

[0138]

number

[0139]

number

[0140]

number

[0141]

number

[0142]

number

[0143]

number

[0144]

number

[0145]

number

[0146]

number

[0147]

number

[0148]

number

[0149] If an image (left or right image) contains i lines and j observation points are extracted from each line, then there can be i·j equations. These equations have i·2+8 variables (i·2 line parameters and 8 strain parameters). If i·j > i·2+8, the optimal solution for the strain parameters is obtained.

[0150]

number

[0151]

number

[0152]

number

[0153]

number

[0154]

number

[0155]

Math.

[0156]

Math.

[0157]

Math.

[0158]

Math.

[0159]

Math.

[0160]

Math.

[0161] Step 4: Global optimization of camera intrinsic parameters, extrinsic parameters, and simplified depth-dependent lens distortion model parameters First, 3D reconstruction of the corrected points in image coordinates is performed using the expression shown below.

[0162]

Math.

[0163]

Math.

[0164] [Math.]] and eliminating [[INDENT]].

[0165] [Math.]]

[0166] In formulas (15), (16) and (17),

[0167] [Math.]] are a rotation matrix and a translation matrix, representing the transformation between the left image coordinate system and the world coordinate system,

[0168] [Math.]] are a rotation matrix and a translation matrix, representing the transformation between the right image coordinate system and the world coordinate system,

[0169] [Math.]] is the intrinsic parameter matrix of the left and right cameras,

[0170] [Math.]] is 3D reconstruction information of the corner point at the i-th row and j-th column in the checkerboard image in the world coordinate system,

[0171] [Math.]] is a scaling factor,

[0172] [Math.]] These are the values ​​in the i-th row and j-th column of the left and right camera projection matrices, respectively.

[0173]

number

[0174]

number

[0175]

number

[0176]

number

[0177]

number

[0178]

number

[0179] Next, the objective function is proposed as shown in equation (19).

[0180]

number

[0181] Finally, the Levenberg-Marquardt optimization method is employed to optimize the intrinsic and extrinsic parameters of the camera, as well as the simplified depth-dependent lens distortion model parameters. The optimized initial values ​​are the intrinsic and extrinsic parameters of the camera obtained by Zhang's calibration method, and the simplified depth-dependent lens distortion model parameters obtained by linear constraints. According to the optimization variables, optimized initial values, and optimization objective function described above, the optimized values ​​of the intrinsic and extrinsic parameters of the camera, and the simplified depth-dependent lens distortion model parameters are obtained.

[0182] The present invention provides a method for calibrating lens distortion in a stereo vision system comprising a left camera and a right camera. In one embodiment, the method comprises: a) acquiring images of corner points using the stereo vision system, wherein the corner points are formed by the intersection of m horizontal lines and n vertical lines arranged on a plane, and the actual distance between adjacent corner points is accurately known; b) extracting the image coordinates of the corner points from the set of images and calculating a set of intrinsic parameters, extrinsic parameters, and distortion parameters for the left camera and the right camera through a homography matrix; and c) the set of intrinsic parameters, extrinsic parameters, and distortion parameters from step (b) and the following equation (2).

[0183]

number

[0184]

number

[0185]

number

[0186]

number

[0187] In one embodiment, step (b) above is achieved using Zhang's method.

[0188] In one embodiment, step (e) above is achieved using the Levenberg-Marquardt optimization method.

[0189] In one embodiment, the above image includes images of corner points at different poses and / or depths.

[0190] In one embodiment, the above image includes images of corner points randomly placed in multiple poses within a specific depth range.

[0191] In one embodiment, the above-mentioned corner points are located on a checkerboard.

[0192] In one embodiment, the left and right cameras are fixed on an optical platform.

[0193] In one embodiment, the strain parameters include eccentric strain parameters and radial strain parameters.

[0194] The present invention also provides a stereo vision system comprising a computing device for carrying out the method of the present invention, and a method for computer vision measurement using the stereo vision system of the present invention.

[0195] In one embodiment, the computer vision measurement described above includes depth-dependent measurement.

[0196] The present invention further provides a non-temporary computer-readable medium on which, when executed by a computing device, program instructions are stored causing a computer device to perform an operation for lens distortion calibration of a stereo vision system, which includes the method of the present invention.

[0197] The present invention also provides a computing device comprising: a) a processor; b) memory; and c) program instructions stored in the memory, which, when executed by the processor, cause the computing device to perform an operation for lens distortion calibration of a stereo vision system, including an operation by the method of the present invention.

[0198] In one embodiment, the above-described computing device is connected to a stereo vision system.

[0199] In one embodiment, the above-described arithmetic unit includes a data input unit for transferring images of corner points.

Claims

1. A method for calibrating lens distortion in a stereo vision system, wherein the stereo vision system comprises a left camera and a right camera, and the method is a. A step of acquiring an image of a corner point using the stereo vision system, wherein the corner point is formed by the intersection of m horizontal lines and n vertical lines arranged on a plane, and the actual distance between adjacent corner points is accurately known. b. A step of extracting the image coordinates of the corner points from the image set and calculating a set of intrinsic parameters, extrinsic parameters, and distortion parameters for the left camera and the right camera through a homography matrix, c. The set of intrinsic parameters, extrinsic parameters, and strain parameters of step (b) and the following equation (2) [Math 1] A set of corrected image coordinates using the lens distortion model defined by [Math 2] The steps to obtain and d. Equations (8) and (9) below [Math 3] [Math 4] Using the set of initial intrinsic parameters, extrinsic parameters, and distortion parameters from step (b), and the set of corrected image coordinates from step (c), the step of reconstructing a set of 3D information, e. A method comprising the step of optimizing the lens distortion model by minimizing the RMSE between the actual distance and the corresponding distance reconstructed from the set of 3D information.

2. The method according to claim 1, wherein step (b) is achieved using Zhang's method.

3. The method according to claim 1, wherein step (e) is achieved using the Levenberg-Marquardt optimization method.

4. The method according to claim 1, wherein the image includes images of corner points at different poses and / or depths.

5. The method according to claim 4, wherein the image includes images of corner points randomly placed in multiple poses within a specific depth range.

6. The method according to claim 1, wherein the corner point is located on a checkerboard.

7. The method according to claim 1, wherein the left camera and the right camera are fixed on an optical platform.

8. The method according to claim 1, wherein the strain parameters include an eccentric strain parameter and a radial strain parameter.

9. A stereo vision system comprising a computing device for carrying out the method according to any one of claims 1 to 8.

10. A method for computer vision measurement using the stereo vision system described in claim 9.

11. The method according to claim 10, wherein the computer vision measurement includes depth-dependent measurement.

12. A non-temporary computer-readable medium storing program instructions that, when executed by a computing device, cause a computer device to perform an operation for lens distortion calibration of a stereo vision system, the operation comprising the method according to any one of claims 1 to 8.

13. A computing device, a. Processor and b. Memory and, c. A computing device comprising: a program instruction stored in the memory and, when executed by the processor, causing the computing device to perform an operation for lens distortion calibration of a stereo vision system, the operation comprising the method according to any one of claims 1 to 8.

14. The computing device according to claim 13, wherein the computing device is connected to a stereo vision system.

15. The arithmetic device according to claim 13, further comprising a data input unit for transferring images of corner points.

Images