A binocular camera calibration method, system, and device

Abstract

The present application belongs to the field of machine vision, and particularly relates to a binocular camera calibration method, system and device, aiming at solving the problem of uneven refraction caused by adding a transparent baffle, thus leading to the difficulty in binocular camera calibration. The method comprises: acquiring images of calibration boards placed in different postures by a left camera and a right camera of a binocular camera; combining the coordinates of the center of each effective circle spot on each calibration board in its corresponding calibration board coordinate system with the corresponding four-dimensional image coordinates to form a control point set; constructing an intersection point set and fitting to obtain a corresponding intersection line equation; taking any calibration board coordinate system as a reference coordinate system, and obtaining the pose transformation matrix of other calibration board coordinate systems relative to the reference coordinate system; taking the pixel projection straight line of all pixels in the left camera and the right camera as the calibration result of the binocular camera, and realizing the calibration of the binocular camera. The present application can effectively solve the problem of binocular camera calibration difficulty caused by the addition of a transparent baffle.

Description

Technical Field

[0001] This invention belongs to the field of machine vision, specifically relating to a binocular camera calibration method, system, and device. Background Technology

[0002] The development of inspection technology has significantly improved the quality of industrial production, especially the surface quality inspection of precision workpieces. With the development of computer vision technology, binocular camera measurement has become an important method of non-contact measurement. The measurement accuracy of a binocular camera depends on its calibration accuracy. Extensive research has been conducted on binocular camera calibration both domestically and internationally. A binocular camera consists of two monocular cameras with a fixed pose relationship, and its calibration often requires the calibration of the monocular cameras as a basis. For monocular cameras, existing technologies typically represent the imaging model as a pinhole imaging model, orthogonal projection model, etc. To improve calibration accuracy, researchers also calibrate distortion parameters related to lens distortion based on the imaging model. For binocular cameras, a common calibration method is to first calibrate each of its two monocular cameras separately, and then further determine the pose relationship between the two monocular cameras. This staged binocular camera calibration method can lead to the calibration error of the monocular cameras affecting the determination of the pose relationship between the two monocular cameras, thereby reducing the calibration accuracy of the binocular camera. Furthermore, binocular cameras operating in complex industrial environments such as oil and gas spills and dust typically require the addition of a transparent shield for protection. This transparent shield causes light refraction, which is often non-uniform, and this alteration to the camera's imaging model leads to calibration difficulties.

[0003] Therefore, it is necessary for those skilled in the art to calibrate the binocular camera when the imaging model changes, in order to solve the problem of uneven refraction caused by the addition of a transparent baffle, which makes the calibration of the binocular camera difficult. Summary of the Invention

[0004] To address the aforementioned problems in the prior art, namely, to resolve the issue of uneven refraction caused by adding a transparent baffle, which leads to difficulties in calibrating a binocular camera, this invention provides a binocular camera calibration method, comprising:

[0005] Step S10: Images of the calibration board placed in different postures are acquired by the left and right cameras of the binocular camera and used as input images;

[0006] Step S20: Establish calibration board coordinate systems for calibration boards in different poses, obtain the coordinates of the center of all effective circular spots on each calibration board in the corresponding calibration board coordinate system, and extract the four-dimensional image coordinates corresponding to the center of each effective circular spot in each pair of input images; combine the coordinates of the center of each effective circular spot on each calibration board in its corresponding calibration board coordinate system with the corresponding four-dimensional image coordinates to form a set of control points, wherein the effective circular spots are those that can be observed simultaneously by the left and right cameras of the binocular camera;

[0007] Step S30: Search for two sets of control points in each pair of control point sets whose Euclidean distance between their four-dimensional pixel coordinates is less than a set distance threshold, and form a set of intersection points for the intersection lines between the calibration plates corresponding to these two control point sets; fit each set of intersection points to obtain the corresponding intersection line equation.

[0008] Step S40: Combining the equations of each intersection line, and taking any calibration plate coordinate system as the reference coordinate system, calculate the representation of the normal vector of the XY plane of the other calibration plate coordinate systems in the reference coordinate system, and then obtain the pose transformation matrix of the other calibration plate coordinate systems relative to the reference coordinate system.

[0009] Step S50: Based on the control point set corresponding to the calibration board in each attitude, obtain the image coordinates of the centers of the four nearest valid circular spots of each pixel in the left and right cameras of the binocular camera on each calibration board image. Calculate the coordinates of the intersection points of the pixel projection lines of each pixel in the left camera with the plane where the calibration board is located in each attitude using interpolation in the corresponding calibration board coordinate system. Transform these coordinates to the reference coordinate system to obtain the first set of intersection point coordinates corresponding to each pixel in the left camera. Then, calculate the coordinates of the intersection points of the pixel projection lines of each pixel in the right camera with the plane where the calibration board is located in each attitude using interpolation in the corresponding calibration board coordinate system. The coordinates of the first intersection point coordinate set corresponding to each pixel in the left camera are obtained by transforming them into the reference coordinate system. Linear fitting is then performed on the coordinates of the first intersection point coordinate set corresponding to each pixel in the left camera, and the fitted line is used as the pixel projection line of each pixel in the left camera. Similarly, linear fitting is performed on the coordinates of the second intersection point coordinate set corresponding to each pixel in the right camera, and the fitted line is used as the pixel projection line of each pixel in the right camera. The pixel projection lines of all pixels in the left camera and the pixel projection lines of all pixels in the right camera are used as the calibration results of the stereo camera, thus achieving the calibration of the stereo camera.

[0010] In some preferred embodiments, images of the calibration plate placed in different orientations are acquired by the left and right cameras of a binocular camera and used as input images. The method is as follows:

[0011] Images of a calibration board in different poses are captured using the left and right cameras of a binocular camera. For each pose of the calibration board, the left and right cameras of the binocular camera each capture an image and combine them into a pair to obtain a set pair of images, which are used as input images.

[0012] In some preferred embodiments, the four-dimensional image coordinates are obtained by:

[0013] The `findCirclesGrid` function from the OpenCV library is used to obtain the image coordinates corresponding to the centers of all valid circles in each image of the input image. The image coordinates corresponding to the centers of each valid circle in each pair of input images are combined to form four-dimensional image coordinates.

[0014] In some preferred embodiments, taking any calibration plate coordinate system as the reference coordinate system, the representation of the normal vector of the XY plane of other calibration plate coordinate systems in the reference coordinate system is calculated, and then the pose transformation matrix of the other calibration plate coordinate systems relative to the reference coordinate system is obtained. The method is as follows:

[0015] The calibration plate coordinate system corresponding to the first orientation of the calibration plate As the reference coordinate system, other calibration plate coordinate systems XY plane The normal vector in the reference coordinate system The following indicates 1 n m The calculation is as follows:

[0016] 1 n2 = 1 I 12 ∧ 1 I 23 / | 1 I 12 ∧ 1 I 23 |

[0017] 1 n3 = 1 I 13 ∧ 1 I 23 / | 1 I 13 ∧ 1 I 23 |

[0018] Where ∧ represents antisymmetric operation, | 1 I 12 ∧ 1 I 23 | is a vector 1 I 12 ∧ 1 I 23The norm of, | 1 I 13 ∧ 1 I 23 | is the vector 1 I 13 ∧ 1 I 23 's norm, 1 I 23 is l 23 's direction vector in the calibration board coordinate system and is solved according to . T represents the transpose operation. m I ij represents the intersection line l ij 's direction vector in the calibration board coordinate system, i, j = 1, 2, 3, i < j, m represents the number of calibration board coordinate systems, m = 1, 2, 3;

[0019] The calibration board coordinate system with respect to 's pose transformation matrix 1 T2 is:

[0020] 1 T2 = 1 I 12 , 1 n2∧ 1 I 12 , 1 n2, 1 O] 2 I 12 ,e z ∧ 2 I 12 ,e z , 2 O] -1

[0021] where 2 I 12 ,e z ∧2 I 12 ,e z , 2 O] -1 is the inverse operation of the matrix 2 I 12 ,e z ∧ 2 I 12 ,e z , 2 O]. e z is the unit vector [0, 0, 1] T ; 1 O is l 12 and l 13The coordinates of the intersection point on the plane are obtained by solving. m A ij , m B ij , m C ij are the coefficients of the x-term, y-term, and constant term of the intersection line equation respectively, i, j = 1, 2, 3, and i < j, m = i or m = j; 2 O is the coordinates of the intersection point of l 12 and l 23 on the plane and is obtained by solving;

[0022] The pose transformation matrix of the calibration plate coordinate system relative to 1 T3 is:

[0023] 1 T3 = 1 I 13 , 1 n3 ∧1 I 13 , 1 n3, 1 O]<o000098>I 13 , e z ∧ 3 I 13 , e z , 3 O] -1

[0024] where 3 I 13 , e z ∧ 3 I[[ID=SO]] 13 , e z , 3 O] -1 is the inverse operation of the matrix 3 I 13 , e z ∧ 3 I 13 , e z , 3 O]; 3 [[ID=1OA]]O is the coordinates of the intersection point of l 13 and l 23 on the plane and is obtained by solving.

[0025] In some preferred embodiments, the coordinates of the intersection point of the pixel projection line of any pixel in the left camera and the plane where the calibration board is located under each pose, in the corresponding calibration board coordinate system, are calculated as follows:

[0026] Based on the set of control points corresponding to the calibration board in each pose, the pixel value (px) of the left camera of the stereo camera is obtained respectively. l Image coordinates of the centers of the four nearest valid circular spots on each calibration plate image ( m u lb , m v lb ), b=1,2,3,4, m=1,2,3;

[0027] Obtain from the set of control points ( m u lb , m v lb The center of the corresponding effective circular spot is in the calibration plate coordinate system. coordinates below ( m x b , m y b ,0);

[0028] Based on pixels (px) l pixel coordinates (u l ,v l )as well as( m u lb , m v lb )and( m x b , m y b The difference (0) is used to calculate the pixel value (px) in the left camera. l The coordinates of the intersection points of the pixel projection lines and the plane where the calibration board is located under each attitude in the corresponding calibration board coordinate system.

[0029] Based on the set of control points corresponding to the calibration board in each pose, the pixel value (px) of the right camera of the stereo camera is obtained. r Image coordinates of the centers of the four nearest valid circular spots on each calibration plate image ( m u rc , m v rc c = 1, 2, 3, 4;

[0030] Obtain from the set of control points ( m u rc , m v rc The center of the corresponding effective circular spot is in the calibration plate coordinate system. coordinates below ( m x c , m y c ,0);

[0031] Based on pixels (px) r pixel coordinates (u r ,v r )as well as( m u rc , m v rc )and( m x c , m y c The difference (px) in the right camera is calculated. r The coordinates of the intersection points of the pixel projection lines and the plane where the calibration board is located under each attitude in the corresponding calibration board coordinate system.

[0032] A second aspect of the present invention provides a binocular camera calibration system, comprising:

[0033] The image acquisition module is configured to acquire images of the calibration board placed in different postures through the left and right cameras of the binocular camera, and use them as input images;

[0034] The control point extraction module is configured to establish calibration board coordinate systems for calibration boards of different poses, obtain the coordinates of the center of all effective circular spots on each calibration board in the corresponding calibration board coordinate system, and extract the four-dimensional image coordinates corresponding to the center of each effective circular spot in each pair of input images; combine the coordinates of the center of each effective circular spot on each calibration board in its corresponding calibration board coordinate system with the corresponding four-dimensional image coordinates to form a control point set, wherein the effective circular spots are those that can be observed simultaneously by the left and right cameras of the binocular camera;

[0035] The intersection extraction module is configured to search for two sets of control points in each pair of control point sets whose Euclidean distance between their four-dimensional pixel coordinates is less than a set distance threshold, and to form the intersection point set of the intersection line between the calibration plates corresponding to these two control point sets; and to fit each intersection point set to obtain the corresponding intersection line equation.

[0036] The pose matrix acquisition module is configured to combine the equations of each intersection line, take any calibration plate coordinate system as the reference coordinate system, calculate the representation of the normal vector of the XY plane of other calibration plate coordinate systems in the reference coordinate system, and then obtain the pose transformation matrix of other calibration plate coordinate systems relative to the reference coordinate system.

[0037] The camera calibration module is configured to, based on the control point set corresponding to the calibration board at each attitude, obtain the image coordinates of the centers of the four nearest valid circular spots of each pixel in the left and right cameras of the binocular camera on each calibration board image; calculate the coordinates of the intersection points of the pixel projection lines of each pixel in the left camera with the plane where the calibration board is located at each attitude in the corresponding calibration board coordinate system using interpolation; transform the coordinates to obtain the first set of intersection point coordinates corresponding to each pixel in the left camera in the reference coordinate system; and calculate the coordinates of the intersection points of the pixel projection lines of each pixel in the right camera with the plane where the calibration board is located at each attitude in the corresponding calibration board coordinate system using interpolation. The coordinates of the first intersection point in the left camera are obtained by transforming them to the reference coordinate system to obtain the second intersection point coordinate set corresponding to each pixel in the right camera; the coordinates of the first intersection point coordinate set corresponding to each pixel in the left camera are fitted with a straight line, and the fitted straight line is used as the pixel projection line of each pixel in the left camera; the coordinates of the second intersection point coordinate set corresponding to each pixel in the right camera are fitted with a straight line, and the fitted straight line is used as the pixel projection line of each pixel in the right camera; the pixel projection lines of all pixels in the left camera and the pixel projection lines of all pixels in the right camera are used as the calibration results of the stereo camera, thus realizing the calibration of the stereo camera.

[0038] In a third aspect, the present invention provides a storage device storing a plurality of programs adapted to be loaded and executed by a processor to implement the above-described binocular camera calibration method.

[0039] In a fourth aspect, the present invention provides a processing apparatus, including a processor and a storage device; the processor is adapted to execute various programs; the storage device is adapted to store multiple programs; the programs are adapted to be loaded and executed by the processor to implement the above-described binocular camera calibration method.

[0040] The beneficial effects of this invention are:

[0041] The present invention can effectively solve the problem of difficult binocular camera calibration caused by the introduction of transparent baffles, and provide technical support for the calibration of binocular cameras in complex industrial scenarios. Attached Figure Description

[0042] Other features, objects, and advantages of this application will become more apparent from the following detailed description of non-limiting embodiments with reference to the accompanying drawings:

[0043] Figure 1 This is a flowchart illustrating a binocular camera calibration method according to the present invention.

[0044] Figure 2 This is a schematic diagram of the framework of a binocular camera calibration system according to the present invention. Detailed Implementation

[0045] The present application will now be described in further detail with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are for illustrative purposes only and are not intended to limit the invention. Furthermore, it should be noted that, for ease of description, only the parts relevant to the invention are shown in the accompanying drawings.

[0046] It should be noted that, unless otherwise specified, the embodiments and features described in this application can be combined with each other. This application will now be described in detail with reference to the accompanying drawings and embodiments.

[0047] The present invention provides a binocular camera calibration method, such as... Figure 1 As shown, the method includes the following steps:

[0048] Step S10: Images of the calibration board placed in different postures are acquired by the left and right cameras of the binocular camera and used as input images;

[0049] Step S20: Establish calibration board coordinate systems for calibration boards in different poses, obtain the coordinates of the center of all effective circular spots on each calibration board in the corresponding calibration board coordinate system, and extract the four-dimensional image coordinates corresponding to the center of each effective circular spot in each pair of input images; combine the coordinates of the center of each effective circular spot on each calibration board in its corresponding calibration board coordinate system with the corresponding four-dimensional image coordinates to form a set of control points, wherein the effective circular spots are those that can be observed simultaneously by the left and right cameras of the binocular camera;

[0050] Step S30: Search for two sets of control points in each pair of control point sets whose Euclidean distance between their four-dimensional pixel coordinates is less than a set distance threshold, and form a set of intersection points for the intersection lines between the calibration plates corresponding to these two control point sets; fit each set of intersection points to obtain the corresponding intersection line equation.

[0051] Step S40: Combining the equations of each intersection line, and taking any calibration plate coordinate system as the reference coordinate system, calculate the representation of the normal vector of the XY plane of the other calibration plate coordinate systems in the reference coordinate system, and then obtain the pose transformation matrix of the other calibration plate coordinate systems relative to the reference coordinate system.

[0052] Step S50: Based on the control point set corresponding to the calibration board in each attitude, obtain the image coordinates of the centers of the four nearest valid circular spots of each pixel in the left and right cameras of the binocular camera on each calibration board image. Calculate the coordinates of the intersection points of the pixel projection lines of each pixel in the left camera with the plane where the calibration board is located in each attitude using interpolation in the corresponding calibration board coordinate system. Transform these coordinates to the reference coordinate system to obtain the first set of intersection point coordinates corresponding to each pixel in the left camera. Then, calculate the coordinates of the intersection points of the pixel projection lines of each pixel in the right camera with the plane where the calibration board is located in each attitude using interpolation in the corresponding calibration board coordinate system. The coordinates of the first intersection point coordinate set corresponding to each pixel in the left camera are obtained by transforming them into the reference coordinate system. Linear fitting is then performed on the coordinates of the first intersection point coordinate set corresponding to each pixel in the left camera, and the fitted line is used as the pixel projection line of each pixel in the left camera. Similarly, linear fitting is performed on the coordinates of the second intersection point coordinate set corresponding to each pixel in the right camera, and the fitted line is used as the pixel projection line of each pixel in the right camera. The pixel projection lines of all pixels in the left camera and the pixel projection lines of all pixels in the right camera are used as the calibration results of the stereo camera, thus achieving the calibration of the stereo camera.

[0053] To more clearly explain the binocular camera calibration method of the present invention, the following is in conjunction with... Figure 1 The steps in the embodiments of the method of the present invention are described in detail below.

[0054] This embodiment represents a preferred implementation. By modeling the imaging process of a binocular camera as two clusters of pixel projection lines, the binocular parameter calibration problem is transformed into a fitting problem of the pixel projection line corresponding to each pixel. For example... Figure 1 As shown, the details are as follows:

[0055] Step S10: Images of the calibration board placed in different postures are acquired by the left and right cameras of the binocular camera and used as input images;

[0056] In this embodiment, the calibration board is placed within the field of view of the binocular camera. Preferably, the left and right cameras of the binocular camera are used to acquire images of the calibration board in three different poses. The calibration board has densely distributed circular spots evenly distributed, and the calibration boards in different poses should be non-parallel to each other. Under each calibration board in each pose, the left and right cameras of the binocular camera each acquire an image and combine them into a pair; considering the three calibration boards in three different poses, a total of three pairs of images are obtained, which serve as the input images.

[0057] Step S20: Establish calibration board coordinate systems for calibration boards in different poses, obtain the coordinates of the center of all effective circular spots on each calibration board in the corresponding calibration board coordinate system, and extract the four-dimensional image coordinates corresponding to the center of each effective circular spot in each pair of input images; combine the coordinates of the center of each effective circular spot on each calibration board in its corresponding calibration board coordinate system with the corresponding four-dimensional image coordinates to form a set of control points, wherein the effective circular spots are those that can be observed simultaneously by the left and right cameras of the binocular camera;

[0058] In this embodiment, the center of the calibration plate is taken as the origin O. P X is defined as the direction parallel to the long side of the calibration plate. P The axis is defined as Y, with the direction parallel to the short side of the calibration plate. P The axis is determined by the right-hand rule. P The axes form the calibration plate coordinate system O. P X P Y P Z P O P X P Y P Let be the plane containing the calibration plate. The three calibration plates in different orientations correspond to three calibration plate coordinate systems, denoted as […]. Given m = 1, 2, 3, obtain the coordinates of the centers of the effective circular spots on the calibration plate in the corresponding calibration plate coordinate system. Effective circular spots refer to those that can be simultaneously observed by the left and right cameras of the stereo camera. The `findCirclesGrid` function from the OpenCV library obtains the image coordinates corresponding to the centers of all effective circular spots in each image of the input image. These image coordinates are then combined to form a four-dimensional image coordinate system. Finally, the coordinates of the centers of each effective circular spot on the calibration plate in the calibration plate coordinate system are combined with the corresponding four-dimensional image coordinates to form the control point set D corresponding to each effective circular spot. m D m ={ m x a , m y a ,0, m u la , m v la , m u ra , m v ra}, a = 1, 2, ..., N m m = 1, 2, 3, where N m Let m be the number of valid circular spots on the calibration plate for the m-th pose. m x a , m y a, 0) is the coordinate of the center of the a-th effective circular spot on the calibration board in the m-th pose in coordinate system, ([[]] m u la , m v la ), and ( m u ra , m v ra ) are the image coordinates of the center of the a-th effective circular spot on the calibration board in the m-th pose in the images captured by the left camera and the right camera of the binocular camera respectively. ( m u la , m v la , m u ra , m v ra ) is the four-dimensional image coordinate corresponding to the center of the a-th effective circular spot on the calibration board in the m-th pose.

[0059] Step S3, search for two sets of control points in each pair of control point sets whose Euclidean distance of four-dimensional pixel coordinates is less than the set distance threshold, and form a set of intersection points of the intersection line between the calibration boards corresponding to these two control point sets; fit each set of intersection points to obtain the corresponding intersection line equation;

[0060] In this embodiment, based on the control point set D corresponding to the calibration board in each pose m , m = 1, 2, 3, search for two sets of control points in every two control point sets whose Euclidean distance of four-dimensional pixel coordinates is less than σ d , and respectively form a set of intersection points of the intersection line between the two calibration boards corresponding to these two control point sets on the corresponding calibration boards, and use the least squares method (https: / / zhuanlan.zhihu.com / p / 38128785) to fit each set of intersection points respectively to obtain the corresponding intersection line equation m A ij x + m B ij y + m C ij = 0, where i, j = 1, 2, 3, and i < j, m = i or m = j, m A ij , m B ij , m C ij are the coefficients of the x-term, the coefficients of the y-term, and the constant term of the intersection line equation respectively; σ dLet \(\delta\) be the distance threshold, which is preferably set to 1 in the present invention. There will be an intersection line between any two calibration plates with different poses. Each intersection line will correspond to an intersection line equation on the two calibration plates associated with it.

[0061] Step S40: Combine the intersection line equations. Taking any calibration plate coordinate system as the reference coordinate system, calculate the representation of the normal vector of the XY plane of other calibration plate coordinate systems in the reference coordinate system, and then obtain the pose transformation matrix of other calibration plate coordinate systems relative to the reference coordinate system.

[0062] In this embodiment, there will be three intersection lines between three calibration plates with different poses, which are respectively denoted as \(l_{ij}\), where \(i,j = 1,2,3\) and \(i < j\). The direction vectors of these three intersection lines in the three calibration plate coordinate systems are denoted as \(\vec{l}_{m}\), \(m = 1,2,3\); when \(m = i\) or \(m = j\), \(\vec{l}_{m}=(\vec{B},-\vec{A},0)\), that is, \(\vec{l}_{ij}=(\vec{B},-\vec{A},0)\), \(\vec{l}_{ik}=(\vec{B},-\vec{A},0)\), \(\vec{l}_{jk}=(\vec{B},-\vec{A},0)\). ij ,其中i,j=1,2,3,且i<j。这三条交线在三个标定板坐标系 中的方向向量记为 m I ij ,m=1,2,3;当m=i或m=j时, m I ij =( m B ij ,- m A ij ,0),即 1 I 12 =( 1 B 12 ,- 1 A 12 ,0), 1 I 13 =( 1 B 13 ,- 1 A 13 ,0), 2 I 12 =( 2 B 12 ,- 2 A 12 ,0), 2 I 23 =( 2 B 23 ,- 2 A 23 ,0),<www. 3 I 13 =( 3 B 13 ,- 3 A 13 ,0), 3 ; I 23 =( 3 B 23 ,- 3 A 23 ,0)。

[0063] Preferred As the reference coordinate system, the XY plane of other calibration plate coordinate systems The normal vector in the reference coordinate system The following indicates 1 n m The calculation is as follows:

[0064]

[0065] Where ∧ represents antisymmetric operation, | 1 I 12 ∧ 1 I 23 | is a vector 1 I 12 ∧ 1 I 23 The modulus length, | 1 I 13 ∧ 1 I 23 | is a vector 1 I 13 ∧ 1 I 23 The length of the mold, 1 I 23 For l 23 In the calibration plate coordinate system The direction vector in, according to The solution is found, where T represents the transpose operation.

[0066] Calibration plate coordinate system Compared to pose transformation matrix 1 T2 is:

[0067] 1 T2 = [ 1 I 12 , 1 n2∧ 1 I 12 , 1 n2, 1 O][ 2 I 12 ,e z ∧ 2 I 12 ,e z , 2 O] -1 (2)

[0068] in[ 2 I 12 ,e z ∧ 2 I 12 ,e z ,2 O] -1 For the matrix [ 2 I 12 ,e z ∧ 2 I 12 ,e z , 2 O] performs the inverse operation, e z The unit vector is [0,0,1]. T ; 1 O is l 12 and l 13 In plane The coordinates of the intersection point on the surface, through The solution is obtained; 2 O is l 12 and l 23 In plane The coordinates of the intersection point on the surface, through The solution is obtained.

[0069] Calibration plate coordinate system Compared to pose transformation matrix 1 T3 is:

[0070] 1 T3 = [ 1 I 13 , 1 n3∧ 1 I 13 , 1 n3, 1 O][ 3 I 13 ,e z ∧ 3 I 13 ,e z , 3 O] -1 (3)

[0071] in[ 3 I 13 ,e z ∧ 3 I 13 ,e z , 3 O] -1 For the matrix [ 3 I 13 ,e z ∧ 3 I 12 ,e z , 3 Perform the inverse operation on O]; 3 O is l 13 and l23 In plane The coordinates of the intersection point on the surface, through The solution is obtained.

[0072] Step S50: Based on the control point set corresponding to the calibration board in each attitude, obtain the image coordinates of the centers of the four nearest valid circular spots of each pixel in the left and right cameras of the binocular camera on each calibration board image. Calculate the coordinates of the intersection points of the pixel projection lines of each pixel in the left camera with the plane where the calibration board is located in each attitude using interpolation in the corresponding calibration board coordinate system. Transform these coordinates to the reference coordinate system to obtain the first set of intersection point coordinates corresponding to each pixel in the left camera. Then, calculate the coordinates of the intersection points of the pixel projection lines of each pixel in the right camera with the plane where the calibration board is located in each attitude using interpolation in the corresponding calibration board coordinate system. The coordinates of the first intersection point coordinate set corresponding to each pixel in the left camera are obtained by transforming them into the reference coordinate system. Linear fitting is then performed on the coordinates of the first intersection point coordinate set corresponding to each pixel in the left camera, and the fitted line is used as the pixel projection line of each pixel in the left camera. Similarly, linear fitting is performed on the coordinates of the second intersection point coordinate set corresponding to each pixel in the right camera, and the fitted line is used as the pixel projection line of each pixel in the right camera. The pixel projection lines of all pixels in the left camera and the pixel projection lines of all pixels in the right camera are used as the calibration results of the stereo camera, thus achieving the calibration of the stereo camera.

[0073] In this embodiment, based on the set D of control points corresponding to the calibration board under each attitude... m ={ m x a , m y a ,0, m u la , m v la , m u ra , m v ra}, a = 1, 2, ..., N m m = 1, 2, 3, respectively, to obtain the pixel values ​​(px) in the left camera of the stereo camera. l Image coordinates of the centers of the four nearest valid circular spots on each calibration plate image ( m u lb , m v lb ), b = 1, 2, 3, 4, further, in D m Obtain ( m u lb , m v lb The center of the corresponding effective circular spot is in Coordinates in a coordinate systemm x b , m y b b = 1, 2, 3, 4, m = 1, 2, 3; based on pixels (px). l pixel coordinates (u l ,v l )as well as( m u lb , m v lb )and( m x b , m y b Given b = 1, 2, 3, 4, we preferably use bilinear interpolation (https: / / blog.csdn.net / qq_37577735 / article / details / 80041586) to calculate the pixel values ​​(px) in the left camera when m = 1, m = 2, and m = 3 respectively. l Pixel projection lines and planes The intersection is at Coordinates of the coordinate system And transform to the reference coordinate system Below, constitutes pixels (px) l The set of coordinates of the first intersection point. The specific transformation process is as follows: When m=1, the pixel px in the left camera l Pixel projection lines and planes The coordinates of the intersection point in the reference coordinate system are: When m=2, combined with Compared to pose transformation matrix 1 T2, calculate the pixels (px) in the left camera. l Pixel projection lines and planes The coordinates of the intersection point in the reference coordinate system in When m=3, combined with Compared to pose transformation matrix 1 T3, calculates the pixels (px) in the left camera. l Pixel projection lines and planes The coordinates of the intersection point in the reference coordinate system in and Pixels (px) l The set of coordinates of the first intersection point. (for pixels px) lThe coordinates of the first intersection point are selected and fitted using the least squares method (https: / / zhuanlan.zhihu.com / p / 38128785) to form a straight line. The fitted line is then used as the pixel value (px) in the left camera. l The pixel projection lines are then calculated. Next, the pixel projection lines for other pixels in the left camera are obtained. The method for obtaining the pixel projection lines for other pixels in the left camera is the same as that for pixels (px). l The method for obtaining the pixel projection line is the same.

[0074] Based on the set of control points D corresponding to the calibration board under each attitude m ={ m x a , m y a ,0, m u la , m v la , m u ra , m v ra}, a = 1, 2, ..., N m m = 1, 2, 3, respectively, to obtain the pixel values ​​(px) in the right camera of the stereo camera. r Image coordinates of the centers of the four nearest valid circular spots on each calibration plate image ( m u rc , m v rc ), c = 1, 2, 3, 4, further, in D m Obtain ( m u rc , m v rc The center of the corresponding effective circular spot is in Coordinates in a coordinate system m x c , m y c c = 1, 2, 3, 4, m = 1, 2, 3; based on pixels (px). r pixel coordinates (u r ,v r )as well as( m u rc , m v rc )and( m x c , m y cGiven c = 1, 2, 3, 4, we preferably use bilinear interpolation (https: / / blog.csdn.net / qq_37577735 / article / details / 80041586) to calculate the pixel values ​​(px) in the right camera when m = 1, m = 2, and m = 3 respectively. r Pixel projection lines and planes The intersection is at Coordinates of the coordinate system And transform to the reference coordinate system Below, constitutes pixels (px) r The set of coordinates of the second intersection point. The specific transformation process is as follows: when m=1, the pixel px in the right camera r Pixel projection lines and planes The coordinates of the intersection point in the reference coordinate system are: At the time, combined Compared to pose transformation matrix 1 T2, calculate the pixels (px) in the right camera. r Pixel projection lines and planes The coordinates of the intersection point in the reference coordinate system in When m=3, combined with Compared to pose transformation matrix 1 T3, calculate pixels (px) in the right camera. r Pixel projection lines and planes The coordinates of the intersection point in the reference coordinate system in and Pixels (px) r The set of coordinates of the second intersection point. For pixels (px). r The coordinates of the second intersection point are preferably fitted using the least squares method (https: / / zhuanlan.zhihu.com / p / 38128785) to form a straight line, and the fitted line is used as the pixel value (px) in the right camera. r The pixel projection lines are then calculated. Next, the pixel projection lines for other pixels in the right camera are obtained. The method for obtaining the pixel projection lines for other pixels in the right camera is the same as that for pixels (px). r The method for obtaining the pixel projection line is the same.

[0075] The pixel projection lines of all pixels in the left camera and the pixel projection lines of all pixels in the right camera are used as the calibration results of the stereo camera to achieve stereo camera calibration.

[0076] A binocular camera calibration system according to a second embodiment of the present invention, such as Figure 2As shown, it includes:

[0077] The image acquisition module 100 is configured to acquire images of the calibration board placed in different postures through the left and right cameras of the binocular camera, and use them as input images;

[0078] The control point extraction module 200 is configured to establish calibration board coordinate systems for calibration boards of different poses, obtain the coordinates of the center of all effective circular spots on each calibration board in the corresponding calibration board coordinate system, and extract the four-dimensional image coordinates corresponding to the center of each effective circular spot in each pair of input images; combine the coordinates of the center of each effective circular spot on each calibration board in its corresponding calibration board coordinate system with the corresponding four-dimensional image coordinates to form a control point set, wherein the effective circular spots are those that can be observed simultaneously by the left and right cameras of the binocular camera;

[0079] The intersection line extraction module 300 is configured to search for two sets of control points in each pair of control point sets whose Euclidean distance between their four-dimensional pixel coordinates is less than a set distance threshold, and to form a set of intersection points for the intersection lines between the calibration plates corresponding to these two sets of control points; and to fit each set of intersection points to obtain the corresponding intersection line equation.

[0080] The pose matrix acquisition module 400 is configured to combine the equations of each intersection line, take any calibration plate coordinate system as the reference coordinate system, calculate the representation of the normal vector of the XY plane of other calibration plate coordinate systems in the reference coordinate system, and then obtain the pose transformation matrix of other calibration plate coordinate systems relative to the reference coordinate system.

[0081] The camera calibration module 500 is configured to, based on the control point set corresponding to the calibration board in each attitude, obtain the image coordinates of the centers of the four nearest valid circular spots of each pixel in the left and right cameras of the binocular camera on each calibration board image; calculate the coordinates of the intersection points of the pixel projection lines of each pixel in the left camera with the plane where the calibration board is located in each attitude in the corresponding calibration board coordinate system using interpolation; transform the coordinates to obtain the first intersection point coordinate set corresponding to each pixel in the left camera in the reference coordinate system; and calculate the coordinates of the intersection points of the pixel projection lines of each pixel in the right camera with the plane where the calibration board is located in each attitude on the corresponding calibration board coordinate system using interpolation. The coordinates in the first intersection point coordinate set corresponding to each pixel in the right camera are obtained by transforming them to the reference coordinate system. Linear fitting is then performed on the coordinates in the first intersection point coordinate set corresponding to each pixel in the left camera, and the fitted line is used as the pixel projection line for each pixel in the left camera. Similarly, linear fitting is performed on the coordinates in the second intersection point coordinate set corresponding to each pixel in the right camera, and the fitted line is used as the pixel projection line for each pixel in the right camera. The pixel projection lines of all pixels in the left camera and the pixel projection lines of all pixels in the right camera are used as the calibration results of the stereo camera, thus achieving stereo camera calibration.

[0082] Those skilled in the art will understand that, for the sake of convenience and brevity, the specific working process and related descriptions of the system described above can be found in the corresponding processes in the foregoing method embodiments, and will not be repeated here.

[0083] It should be noted that the binocular camera calibration system provided in the above embodiments is only an example of the division of the above functional modules. In practical applications, the above functions can be assigned to different functional modules as needed, that is, the modules or steps in the embodiments of the present invention can be further decomposed or combined. For example, the modules in the above embodiments can be merged into one module, or further divided into multiple sub-modules to complete all or part of the functions described above. The names of the modules and steps involved in the embodiments of the present invention are only for distinguishing the various modules or steps and are not considered as an improper limitation of the present invention.

[0084] A storage device according to a third embodiment of the present invention stores a plurality of programs, the programs being adapted to be loaded and executed by a processor to implement the above-described binocular camera calibration method.

[0085] A processing apparatus according to a fourth embodiment of the present invention includes a processor and a storage device; the processor is adapted to execute various programs; the storage device is adapted to store multiple programs; the programs are adapted to be loaded and executed by the processor to implement the above-described binocular camera calibration method.

[0086] Those skilled in the art will understand that, for the sake of convenience and brevity, the specific working process and related descriptions of the storage device and processing device described above can be referred to the corresponding process in the foregoing method examples, and will not be repeated here.

[0087] Those skilled in the art will recognize that the modules and method steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of both. The programs corresponding to the software modules and method steps can be placed in random access memory (RAM), main memory, read-only memory (ROM), electrically programmable ROM, electrically erasable programmable ROM, registers, hard disks, removable disks, CD-ROMs, or any other form of storage medium known in the art. To clearly illustrate the interchangeability of electronic hardware and software, the components and steps of the various examples have been generally described in terms of functionality in the foregoing description. Whether these functions are implemented in electronic hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of the invention.

[0088] The terms “first,” “second,” “third,” etc., are used to distinguish similar objects, not to describe or indicate a specific order or sequence.

[0089] The technical solution of the present invention has been described above with reference to the preferred embodiments shown in the accompanying drawings. However, it will be readily understood by those skilled in the art that the scope of protection of the present invention is obviously not limited to these specific embodiments. Without departing from the principles of the present invention, those skilled in the art can make equivalent changes or substitutions to the relevant technical features, and the technical solutions after these changes or substitutions will all fall within the scope of protection of the present invention.

Claims

1. A method for calibrating a binocular camera, characterized in that, The method includes the following steps: Step S10: Images of the calibration board placed in different postures are acquired by the left and right cameras of the binocular camera and used as input images; Step S20: Establish calibration board coordinate systems for calibration boards in different poses, obtain the coordinates of the center of all effective circular spots on each calibration board in the corresponding calibration board coordinate system, and extract the four-dimensional image coordinates corresponding to the center of each effective circular spot in each pair of input images; combine the coordinates of the center of each effective circular spot on each calibration board in its corresponding calibration board coordinate system with the corresponding four-dimensional image coordinates to form a set of control points, wherein the effective circular spots are those that can be observed simultaneously by the left and right cameras of the binocular camera; Step S30: Search for two sets of control points in each pair of control point sets whose Euclidean distance between their four-dimensional image coordinates is less than a set distance threshold, and form a set of intersection points for the intersection lines between the calibration plates corresponding to these two control point sets; fit each set of intersection points to obtain the corresponding intersection line equation. Step S40: Combining the equations of each intersection line, and taking any calibration plate coordinate system as the reference coordinate system, calculate the representation of the normal vector of the XY plane of the other calibration plate coordinate systems in the reference coordinate system, and then obtain the pose transformation matrix of the other calibration plate coordinate systems relative to the reference coordinate system. Step S50: Based on the control point set corresponding to the calibration board in each attitude, obtain the image coordinates of the centers of the four nearest valid circular spots of each pixel in the left and right cameras of the binocular camera on each calibration board image. Calculate the coordinates of the intersection points of the pixel projection lines of each pixel in the left camera with the plane where the calibration board is located in each attitude using interpolation in the corresponding calibration board coordinate system. Transform these coordinates to the reference coordinate system to obtain the first set of intersection point coordinates corresponding to each pixel in the left camera. Then, calculate the coordinates of the intersection points of the pixel projection lines of each pixel in the right camera with the plane where the calibration board is located in each attitude using interpolation in the corresponding calibration board coordinate system. The coordinates of the first intersection point coordinate set corresponding to each pixel in the left camera are obtained by transforming them into the reference coordinate system. Linear fitting is then performed on the coordinates of the first intersection point coordinate set corresponding to each pixel in the left camera, and the fitted line is used as the pixel projection line of each pixel in the left camera. Similarly, linear fitting is performed on the coordinates of the second intersection point coordinate set corresponding to each pixel in the right camera, and the fitted line is used as the pixel projection line of each pixel in the right camera. The pixel projection lines of all pixels in the left camera and the pixel projection lines of all pixels in the right camera are used as the calibration results of the stereo camera, thus achieving the calibration of the stereo camera.

2. The binocular camera calibration method according to claim 1, characterized in that, Images of the calibration board placed in different orientations are acquired using the left and right cameras of a binocular camera and used as input images. The method is as follows: Images of a calibration board in different poses are captured using the left and right cameras of a binocular camera. For each pose of the calibration board, the left and right cameras of the binocular camera each capture an image and combine them into a pair to obtain a set pair of images, which are used as input images.

3. A binocular camera calibration method based on claim 2, characterized in that, The four-dimensional image coordinates are obtained as follows: The `findCirclesGrid` function from the OpenCV library is used to obtain the image coordinates corresponding to the centers of all valid circles in each image of the input image. The image coordinates corresponding to the centers of each valid circle in each pair of input images are combined to form four-dimensional image coordinates.

4. A binocular camera calibration method based on claim 1, characterized in that, Using any calibration plate coordinate system as the reference coordinate system, calculate the representation of the normal vector of the XY plane of other calibration plate coordinate systems in the reference coordinate system, and then obtain the pose transformation matrix of other calibration plate coordinate systems relative to the reference coordinate system. The method is as follows: The calibration plate coordinate system corresponding to the first orientation of the calibration plate As the reference coordinate system, other calibration plate coordinate systems XY plane The normal vector in the reference coordinate system The following indicates The calculation is as follows: ； in This indicates opposition to asymmetric operations. For vectors The length of the mold, For vectors The length of the mold, for In the calibration plate coordinate system The direction vector in, according to Solve it. This indicates the transpose operation. Indicates the line of intersection The direction vector in the calibration plate coordinate system , , Indicates the number of coordinate systems on the calibration plate. ; Calibration plate coordinate system Compared to pose transformation matrix for: ； in For the matrix Perform the inverse operation. unit vector ; for and In plane The coordinates of the intersection point on the surface, through The solution is obtained. , , The equations of the intersection lines are respectively The coefficient of the term, The coefficients and constant terms of the terms. ,and , or ; for and In plane The coordinates of the intersection point on the surface, through The solution is obtained; Calibration plate coordinate system Compared to pose transformation matrix for: ； in For the matrix Perform the inverse operation; for and In plane The coordinates of the intersection point on the surface, through The solution is obtained.

5. A binocular camera calibration method based on claim 1, characterized in that, The coordinates of the intersection points of the pixel projection lines of any pixel in the left camera and the right camera with the plane where the calibration board is located under each pose, in the corresponding calibration board coordinate system, are calculated as follows: Based on the set of control points corresponding to the calibration board in each pose, the pixels in the left camera of the stereo camera are obtained respectively. Image coordinates of the centers of the four nearest valid circular spots on each calibration plate image , , ; Obtain from the set of control points The center of the corresponding effective circular spot is in the calibration plate coordinate system. coordinates below ; Based on pixels pixel coordinates as well as and The pixels in the left camera are calculated using the difference. The coordinates of the intersection points of the pixel projection lines and the plane where the calibration board is located under each attitude in the corresponding calibration board coordinate system. ; Based on the set of control points corresponding to the calibration board in each posture, the pixels in the right camera of the stereo camera are obtained respectively. Image coordinates of the centers of the four nearest valid circular spots on each calibration plate image , ; Obtain from the set of control points The center of the corresponding effective circular spot is in the calibration plate coordinate system. coordinates below ; Based on pixels pixel coordinates as well as and The pixels in the right camera are calculated using the difference. The coordinates of the intersection points of the pixel projection lines and the plane where the calibration board is located under each attitude in the corresponding calibration board coordinate system. .

6. A binocular camera calibration system, characterized in that, include: The image acquisition module is configured to acquire images of the calibration board placed in different postures through the left and right cameras of the binocular camera, and use them as input images; The control point extraction module is configured to establish calibration board coordinate systems for calibration boards of different poses, obtain the coordinates of the center of all effective circular spots on each calibration board in the corresponding calibration board coordinate system, and extract the four-dimensional image coordinates corresponding to the center of each effective circular spot in each pair of input images; combine the coordinates of the center of each effective circular spot on each calibration board in its corresponding calibration board coordinate system with the corresponding four-dimensional image coordinates to form a control point set, wherein the effective circular spots are those that can be observed simultaneously by the left and right cameras of the binocular camera; The intersection extraction module is configured to search for two sets of control points in each pair of control point sets whose Euclidean distance between their four-dimensional image coordinates is less than a set distance threshold, and to form the intersection point set of the intersection line between the calibration plates corresponding to these two control point sets; and to fit each intersection point set to obtain the corresponding intersection line equation. The pose matrix acquisition module is configured to combine the equations of each intersection line, take any calibration plate coordinate system as the reference coordinate system, calculate the representation of the normal vector of the XY plane of other calibration plate coordinate systems in the reference coordinate system, and then obtain the pose transformation matrix of other calibration plate coordinate systems relative to the reference coordinate system. The camera calibration module is configured to, based on the control point set corresponding to the calibration board at each attitude, obtain the image coordinates of the centers of the four nearest valid circular spots of each pixel in the left and right cameras of the binocular camera on each calibration board image; calculate the coordinates of the intersection points of the pixel projection lines of each pixel in the left camera with the plane where the calibration board is located at each attitude in the corresponding calibration board coordinate system using interpolation; transform the coordinates to obtain the first set of intersection point coordinates corresponding to each pixel in the left camera in the reference coordinate system; and calculate the coordinates of the intersection points of the pixel projection lines of each pixel in the right camera with the plane where the calibration board is located at each attitude in the corresponding calibration board coordinate system using interpolation. The coordinates of the first intersection point in the left camera are obtained by transforming them to the reference coordinate system to obtain the second intersection point coordinate set corresponding to each pixel in the right camera; the coordinates of the first intersection point coordinate set corresponding to each pixel in the left camera are fitted with a straight line, and the fitted straight line is used as the pixel projection line of each pixel in the left camera; the coordinates of the second intersection point coordinate set corresponding to each pixel in the right camera are fitted with a straight line, and the fitted straight line is used as the pixel projection line of each pixel in the right camera; the pixel projection lines of all pixels in the left camera and the pixel projection lines of all pixels in the right camera are used as the calibration results of the stereo camera, thus realizing the calibration of the stereo camera.

7. A storage device storing a plurality of programs, characterized in that, The program is applicable to being loaded and executed by a processor to implement a binocular camera calibration method according to any one of claims 1-5.

8. A processing apparatus, comprising a processor and a storage device; the processor being adapted to execute various programs; the storage device being adapted to store multiple programs; characterized in that, The program is applicable to being loaded and executed by a processor to implement a binocular camera calibration method according to any one of claims 1-5.

Images