A binocular camera calibration method, device, equipment and storage medium

Abstract

The application discloses a binocular camera calibration method, device and equipment and a storage medium, the method comprises the following steps: adjusting a plane target to different rotation angles, using a single-axis high-precision displacement table to generate an affine projection model by translating each angle by a fixed distance in batches; based on the plane projection vanishing line and the translation direction vanishing point, combined with the affine projection model, a non-homogeneous equation is established to link the left and right camera internal parameters; a plurality of direction plane translation movements are collected to solve the non-homogeneous equation to obtain the initial value of the camera parameters, and the non-linear optimization method is used to optimize the camera internal and external parameters. The problem that it is difficult to control the uniformity of the spatial distribution of the plane target in an arbitrary position by human placement, resulting in that the camera parameters are affected by the dense distribution of the plane position in the optimization and fall into a local optimal solution is eliminated. In the depth direction, the data is collected by collecting positions in multiple dimensions, so that the strong correlation between the near and far distribution of the plane target and the calculation of the camera focal length is reflected, and the accurate value of the stable camera focal length can be estimated.

Description

Technical Field

[0001] This invention relates to the field of computer vision 3D measurement technology, and in particular to a binocular camera calibration method, apparatus, device, and storage medium based on affine constraints of planar translational motion. Background Technology

[0002] Computer vision 3D measurement uses two or more cameras. Camera calibration methods determine the intrinsic and extrinsic parameters of each camera. During measurement, the 3D coordinates of target pixels are obtained based on stereo vision principles. The camera's intrinsic parameters include focal length, principal point coordinates, and distortion parameters. The focal length represents the magnification during object imaging, magnifying distant objects to the pixel level of the image sensor. The principal point coordinates represent the intersection of the optical axes of the image sensor and the camera lens, and are also the position parameters of the projection center. Distortion parameters represent the nonlinear distortion during camera projection, divided into radial and eccentric distortion, with eccentric distortion causing both radial and tangential nonlinear distortion. Extrinsic parameters include rotation and translation parameters from the 3D world coordinate system to the camera coordinate system. Therefore, the parameters estimated by the camera calibration method directly determine the spatial transformation and projection relationship from the world coordinate system to the camera image coordinate system, and are a key step affecting the error of the stereo vision system.

[0003] Currently, existing camera calibration techniques rely on the Zhang Zhengyou calibration method based on planar targets. This method typically involves manually placing checkerboard-shaped planar targets facing the camera at arbitrary locations. However, the arbitrary placement makes it difficult to control the uniformity of the planar targets, resulting in camera parameters being significantly affected by the densely distributed planar positions during optimization. Furthermore, since the planar targets face the camera, the depth directions in front and behind the camera are generally nearly perpendicular to the planar, resulting in a lack of depth dimension at each acquisition location. The "nearer objects appear larger, farther objects smaller" mechanism in camera perspective models leads to a strong correlation between the near-far distribution of planar targets and the calculation of camera focal length, making it difficult to estimate a stable and accurate value for the camera focal length. Therefore, during camera calibration, it is necessary to control the acquisition of the calibration area and increase dimensional constraints on the acquisition space to improve the calibration accuracy of camera parameters. Summary of the Invention

[0004] In view of the above problems, the present invention provides a binocular camera calibration method, apparatus, device and storage medium for overcoming the above problems or at least partially solving the above problems.

[0005] This invention provides the following solution:

[0006] A binocular camera calibration method includes:

[0007] The checkerboard-patterned target is adjusted to different rotation angles, and the target is fixedly connected to a single-axis translation stage. The target is translated multiple times at each angle, with each translation involving a fixed distance. A set of affine coordinate data is formed by recording the affine coordinates of the non-orthogonal affine coordinate system points, the left-eye image coordinates, and the right-eye image coordinates after each translation at each angle. The affine coordinates include the coordinates of each corner point of the checkerboard-patterned target and the fixed distance. The left-eye image coordinates are the coordinates of each corner point detected in the left-eye image acquired by the left-eye camera, and the right-eye image coordinates are the coordinates of each corner point detected in the right-eye image acquired by the right-eye camera. An affine coordinate matrix is ​​generated using all the affine coordinate data.

[0008] Using the affine coordinate matrix, based on the affine matrix three-dimensional coordinate transformation and the camera projection model, an affine projection model from the affine coordinate system to the camera image coordinate system is established, and the projection matrix is ​​obtained.

[0009] Based on the plane projection cancellation line and translation direction cancellation point, the initial intrinsic parameters of the left and right cameras are obtained by solving the non-homogeneous equations in conjunction with the affine projection model.

[0010] Using the initial intrinsic parameters and the affine projection model, we can obtain several initial extrinsic parameters for the left and right cameras after each translation at each angle.

[0011] The initial intrinsic parameters, several initial extrinsic parameters, and the translation direction vector for each translation are substituted into a nonlinear optimization method to optimize the intrinsic and extrinsic parameters of the left and right cameras, thereby obtaining the target intrinsic and extrinsic parameters.

[0012] Preferably: the non-orthogonal affine coordinate system is transformed to an orthogonal coordinate system using the affine coordinate matrix based on the affine matrix three-dimensional coordinate transformation; the affine matrix three-dimensional coordinate transformation is represented by the following equation 1:

[0013]

[0014] In the formula: P w Let P be a point in the world coordinate system, A be the affine transformation matrix, P be a point in the affine coordinate system, X and Y be the coordinates of the corner points, and Z be the fixed distance (a1, a2, a3). T This is the unit direction vector in the world coordinate system.

[0015] Preferably, the affine projection model is represented by the following equation 2:

[0016]

[0017] In the formula: α is the scaling factor of the coordinates p of the projected point on the image, p is a point in the homogeneous coordinate system of the left or right image, and M is the projection matrix. Let P be in homogeneous coordinate form, R be a 3×3 rotation matrix, A be an affine transformation matrix, t be a translation vector, R, A, t be the extrinsic parameters of the camera, and K be the intrinsic parameter matrix of the camera, where f x and f y γ represents the focal length in the x and y directions. skew c represents the pixel tilt coefficient. x and c y Indicates the coordinates of the projection center;

[0018] based on The projection matrix M is obtained by using least squares estimation of the affine coordinate matrix composed of p and .

[0019] Preferably: Based on the plane projection cancellation line and translational cancellation point, the initial intrinsic parameters of the left and right cameras are obtained by solving the non-homogeneous equations established by combining the affine projection model, including:

[0020] Expanding the projection matrix M column-wise, M = [m1 m2 m3 m4], the absolute quadratic curve of the camera is represented by the intrinsic parameter matrix: B = K ―T K ―1 , B=K ―T K ―1 Substituting into equation (2), the constraint of the affine coordinate system on the camera parameters is described in equation (3):

[0021]

[0022] In the formula: m1 and m2 represent the disappearance lines of the planar target on the image, and m3 is the disappearance point on the image in the translation direction;

[0023] The disappearance line and disappearance point are mapped to the three-dimensional world coordinate system through the absolute quadratic curve B. The angle between the translation direction and the checkerboard plane target provides two binocular constraints, which are represented by the following equation 4:

[0024]

[0025] The absolute quadratic curve matrix is ​​solved by solving the non-homogeneous equation system, and the matrix is ​​decomposed to obtain the initial intrinsic parameters of the left and right cameras.

[0026] Preferably: B and B′ are determined to be real symmetric matrices, each containing 6 unknown parameters. The unknown parameters of the two matrices are expanded into column vectors b and b′, ​​and b and b′ are combined to form column vector X. During each translation process, a set of non-homogeneous equations is established by combining equations 3 and 4, as expressed by equation 5 below:

[0027]

[0028] In the formula: C is the coefficient matrix for each parameter in b and b′ established according to Equations 3 and 4, and Y is the parameter on the right side of Equations 3 and 4;

[0029] The non-homogeneous constraint equation of Equation 5 is used for one set of affine coordinate matrices, and Equation 3 is transformed into a homogeneous equation, which is expressed by Equation 6 below:

[0030]

[0031] The non-homogeneous equations in Equation 5, which combine to form the complete affine coordinate matrix, are used to solve the non-homogeneous equations for the camera parameters. These equations are expressed in Equation 7.

[0032]

[0033] In the formula: C s The coefficient matrix derived from C in Equation 5 and Equation 6 Y is obtained by concatenating rows. s From Equation 5, Y and the zero vector 0 1×6(N―1) The result is obtained by splicing together N, which is the number of affine coordinate matrices.

[0034] Preferably, the initial intrinsic parameters are substituted into Equation 2 to obtain several initial extrinsic parameters corresponding to the left and right cameras after each translation at each angle.

[0035] Preferably: Substituting the initial intrinsic parameters, several initial extrinsic parameters, and the translation direction vector for each translation into a nonlinear optimization method to optimize the intrinsic and extrinsic parameters of the left and right cameras to obtain the target intrinsic and extrinsic parameters includes:

[0036] Substitute the initial intrinsic parameters, several initial extrinsic parameters, and the translation direction vector for each translation into the nonlinear optimization function, and use the reprojection error in the affine projection model of Equation 2 as the objective function; the reprojection error represents the difference between the reprojected image coordinates after substituting the camera parameters and the actual coordinates on the image;

[0037] The camera parameters are iterated using a nonlinear optimization algorithm to minimize the reprojection error, and the final optimized intrinsic and extrinsic parameters of the camera are output.

[0038] A binocular camera calibration device, comprising:

[0039] An affine coordinate matrix acquisition unit is used to adjust a checkerboard-shaped target to different rotation angles. The checkerboard-shaped target is fixedly connected to a single-axis translation stage. The single-axis translation stage is used to translate the checkerboard-shaped target multiple times at each angle, with each translation being a fixed distance. A set of affine coordinate data is formed by recording the affine coordinates of the corresponding non-orthogonal affine coordinate system points, left-eye image coordinates, and right-eye image coordinates after each translation at each angle. The affine coordinates include the coordinates of each corner point of the checkerboard-shaped target and the fixed distance. The left-eye image coordinates are the coordinates of each corner point detected in the left-eye image acquired by the left-eye camera, and the right-eye image coordinates are the coordinates of each corner point detected in the right-eye image acquired by the right-eye camera. An affine coordinate matrix is ​​generated using all the affine coordinate data.

[0040] The affine projection model establishment unit is used to establish an affine projection model from the affine coordinate system to the camera image coordinate system based on the affine coordinate matrix, the affine matrix three-dimensional coordinate transformation, and the camera projection model, and to obtain the projection matrix.

[0041] The initial intrinsic parameter acquisition unit is used to obtain the initial intrinsic parameters of the left eye camera and the right eye camera by solving non-homogeneous equations based on the plane projection cancellation line and translation direction cancellation point, combined with the affine projection model.

[0042] The initial extrinsic parameter acquisition unit is used to solve for several initial extrinsic parameters of the left eye camera and the right eye camera after each translation at each angle using the initial intrinsic parameters and the affine projection model;

[0043] The target intrinsic and extrinsic parameter acquisition unit is used to substitute the initial intrinsic parameters, several initial extrinsic parameters, and the translation direction vector for each translation into a nonlinear optimization method to optimize the intrinsic and extrinsic parameters of the left eye camera and the right eye camera to obtain the target intrinsic and extrinsic parameters.

[0044] A binocular camera calibration device, the device comprising a processor and a memory:

[0045] The memory is used to store program code and transmit the program code to the processor;

[0046] The processor is used to execute the above-described binocular camera calibration method according to the instructions in the program code.

[0047] A computer-readable storage medium for storing program code for performing the above-described binocular camera calibration method.

[0048] According to specific embodiments provided by the present invention, the present invention discloses the following technical effects:

[0049] This application provides a binocular camera calibration method, apparatus, device, and storage medium. This method eliminates the problem that arbitrarily placed targets are difficult to control in terms of the uniformity of the planar target spatial distribution, leading to camera parameters being affected by densely distributed planar positions and getting trapped in local optima during optimization. By acquiring data through multi-dimensional acquisition positions in the depth direction, the strong correlation between the near and far distribution of the planar targets and the calculation of the camera focal length is reflected, allowing for the estimation of a stable and accurate value for the camera focal length. The calibration accuracy of camera parameters is improved by controlling the acquisition of the calibration area and increasing the dimensional constraints of the acquisition space.

[0050] Of course, any product implementing this invention does not necessarily need to achieve all of the advantages described above at the same time. Attached Figure Description

[0051] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the embodiments will be briefly described below. Obviously, the drawings described below are merely some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without any creative effort.

[0052] Figure 1 This is a flowchart of a binocular camera calibration method provided in an embodiment of the present invention;

[0053] Figure 2 This is a schematic diagram of the calibration scenario configuration provided in an embodiment of the present invention;

[0054] Figure 3 This is a schematic diagram of the calibration process provided in an embodiment of the present invention;

[0055] Figure 4 This is a schematic diagram of a binocular camera calibration device provided in an embodiment of the present invention;

[0056] Figure 5 This is a schematic diagram of a binocular camera calibration device provided in an embodiment of the present invention. Detailed Implementation

[0057] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of the present invention, and not all of them. All other embodiments obtained by those skilled in the art based on the embodiments of the present invention are within the scope of protection of the present invention.

[0058] See Figure 1 This invention provides a binocular camera calibration method, such as... Figure 1 As shown, the method may include:

[0059] S101: Adjust the checkerboard plane target to different rotation angles. The checkerboard plane target is fixedly connected to a single-axis displacement stage. The single-axis displacement stage is used to translate the checkerboard plane target multiple times at each angle, with each translation being a fixed distance. A set of affine coordinate data is formed by recording the affine coordinates of the non-orthogonal affine coordinate system points, left-eye image coordinates, and right-eye image coordinates after each translation at each angle. The affine coordinates include the coordinates of each corner point of the checkerboard plane target and the fixed distance. The left-eye image coordinates are the coordinates of each corner point detected in the left-eye image acquired by the left-eye camera, and the right-eye image coordinates are the coordinates of each corner point detected in the right-eye image acquired by the right-eye camera. An affine coordinate matrix is ​​generated using all the affine coordinate data. Specifically, a corner detection algorithm is used to detect the coordinates of each corner point in the left-eye image acquired by the left-eye camera, and the corner detection algorithm detects the coordinates of each corner point in the right-eye image acquired by the right-eye camera.

[0060] S102: Using the affine coordinate matrix based on the affine matrix three-dimensional coordinate transformation and the camera projection model, establish an affine projection model from the affine coordinate system to the camera image coordinate system, and calculate the projection matrix; specifically, using the affine coordinate matrix based on the affine matrix three-dimensional coordinate transformation, transform the non-orthogonal affine coordinate system to an orthogonal coordinate system; the affine matrix three-dimensional coordinate transformation is represented by the following equation 1:

[0061]

[0062] In the formula: P w Let P be a point in the world coordinate system, A be the affine transformation matrix, P be a point in the affine coordinate system, X and Y be the coordinates of the corner points, and Z be the fixed distance (z1, a2, a3). T This is the unit direction vector in the world coordinate system.

[0063] The affine projection model is represented by the following equation 2:

[0064]

[0065] In the formula: α is the scaling factor of the coordinates p of the projected point on the image, p is a point in the homogeneous coordinate system of the left or right image, and M is the projection matrix. Let P be the homogeneous coordinate form, R be the 3×3 rotation matrix, A be the affine transformation matrix, and t be the translation vector. R, A, and t are related to the initial position of the camera relative to the checkerboard target and are collectively referred to as the camera's extrinsic parameters. K is the camera's intrinsic parameter matrix, where f x and f y γ represents the focal length in the x and y directions. skew c represents the pixel tilt coefficient.x and c y Indicates the coordinates of the projection center;

[0066] based on The projection matrix M is obtained by using least squares estimation of the affine coordinate matrix composed of p and .

[0067] S103: Based on the plane projection cancellation line and translational cancellation points, and combined with the affine projection model, establish and solve non-homogeneous equations to obtain the initial intrinsic parameters of the left and right cameras; specifically implemented, including:

[0068] Expanding the projection matrix M column-wise, M = [m1 m2 m3 m4], the absolute quadratic curve of the camera is represented by the intrinsic parameter matrix: B = K ―T K ―1 , B=K ―T K ―1 Substituting into equation (2), the constraint of the affine coordinate system on the camera parameters is described in equation (3):

[0069]

[0070] In the formula: m1 and m2 represent the disappearance lines of the planar target on the image, and m3 is the disappearance point on the image in the translation direction;

[0071] The disappearance line and disappearance point are mapped to the three-dimensional world coordinate system through the absolute quadratic curve B. The angle between the translation direction and the checkerboard plane target provides two binocular constraints, which are represented by the following equation 4:

[0072]

[0073] We determine that B and B′ are real symmetric matrices, each containing 6 unknown parameters. We expand the unknown parameters of the two matrices into column vectors b and b′, ​​and combine b and b′ to form a column vector X. During each translation process, we establish a set of non-homogeneous equations by combining equations 3 and 4, as expressed in equation 5 below:

[0074]

[0075] In the formula: C is the coefficient matrix for each parameter in b and b′ established according to Equations 3 and 4, and Y is the parameter on the right side of Equations 3 and 4. Since the scaling factor α of the affine coordinate matrix for each different plane rotation direction is different, the non-homogeneous constraint equation of Equation 5 is used for one set of affine coordinate matrices. Equation 3 is transformed into a homogeneous equation as shown in Equation 6 below:

[0076]

[0077] Similarly, Equation 6 can also be used for the coefficient matrix of each parameter in b and b′. Applying homogeneous equation 6 to the remaining affine coordinate matrices, and combining it with the non-homogeneous equations of equation 5, we can form a system of non-homogeneous equations for solving all affine coordinate matrices and simultaneously solving for the camera parameters, as expressed in equation 6 below:

[0078]

[0079] In the formula: C s The coefficient matrix derived from C in Equation 5 and Equation 6 Y is obtained by concatenating rows. s From Equation 5, Y and the zero vector 0 1×6(N―1) The result is obtained by splicing together N, which is the number of affine coordinate matrices.

[0080] The absolute quadratic curve matrix is ​​solved by solving the non-homogeneous equation system, and the matrix is ​​decomposed to obtain the initial intrinsic parameters of the left and right cameras.

[0081] S104: Using the initial intrinsic parameters and the affine projection model, we obtain several initial extrinsic parameters corresponding to the left and right cameras after each translation at each angle; in specific implementation, we substitute the initial intrinsic parameters into Equation 2 to obtain several initial extrinsic parameters corresponding to the left and right cameras after each translation at each angle.

[0082] S105: Substitute the initial intrinsic parameters, several initial extrinsic parameters, and the translation direction vector for each translation into a nonlinear optimization method to optimize the intrinsic and extrinsic parameters of the left and right cameras to obtain the target intrinsic and extrinsic parameters. Specifically, the initial intrinsic parameters, several initial extrinsic parameters, and the translation direction vector for each translation are substituted into a nonlinear optimization function, using the reprojection error in the affine projection model of Equation 2 as the objective function; the reprojection error represents the difference between the reprojected image coordinates after substituting the camera parameters and the actual coordinates on the image.

[0083] The camera parameters are iterated using a nonlinear optimization algorithm to minimize the reprojection error, and the final optimized intrinsic and extrinsic parameters of the camera are output.

[0084] The binocular camera calibration method provided in this application uses a single-axis high-precision displacement stage and a planar target to adjust the plane to different rotation angles. For each angle, the plane is translated by a fixed distance several times to generate a three-dimensional coordinate grid in an affine coordinate system. Based on the affine matrix three-dimensional coordinate transformation and the pinhole camera projection model, an affine projection model from the affine coordinate system to the camera image coordinate system is established. Based on the plane projection cancellation line and the translation direction cancellation point, a non-homogeneous equation is established to connect the intrinsic parameters of the left and right cameras in combination with the affine projection model. The translational motion of multiple plane directions is collected to solve the non-homogeneous equation to obtain the initial values ​​of the camera parameters, which are then substituted into a nonlinear optimization method to optimize the camera's intrinsic and extrinsic parameters.

[0085] The binocular camera calibration method provided in this application embodiment will be described in detail below with reference to the binocular camera calibration framework.

[0086] See Figure 2 The affine-constrained binocular camera calibration framework utilizes the following resources: a single-axis high-precision displacement stage, a binocular camera, a checkerboard planar target, and a computer workstation. See also... Figure 3 The specific implementation steps of this method are as follows:

[0087] Step 1: The binocular camera acquires images of the target plane translated by the single-axis displacement stage.

[0088] This method addresses the problem in the existing Zhang Zhengyou calibration method where the distribution range of the planar target at each position in the front and back depth directions is small, leading to a decrease in focal length calculation accuracy due to the strong correlation with depth. The proposed method extends the measurement of translation distance in the depth direction, and the longer depth range constrains the error in focal length calculation to a smaller range.

[0089] Specifically, the computer workstation 5 controls the single-axis displacement stage 4, the left eye camera 1, and the right eye camera 2. The checkerboard plane target 3 is fixed on the single-axis displacement stage 4 and moves a fixed distance Z. The coordinates (X,Y) of each corner point of the checkerboard plane target 3 form a non-orthogonal affine coordinate system, with the coordinates P = (X,Y,Z). T A corner detection algorithm was used to detect the coordinates p and p′ of the checkerboard corner points on the left and right images. A single-axis translation stage was moved a fixed distance multiple times, and the corresponding affine coordinates and image coordinates were recorded to form an affine coordinate grid. The plane angle was adjusted, the above steps were repeated, and the affine coordinate data was recorded.

[0090] Step 2: Establish a projection model of the affine coordinate system onto the camera image.

[0091] Existing calibration techniques based on planar targets rely on plane-to-plane homography calculations. Each calculation considers only the coordinate homography matrix of a single plane, using only all points on that single plane. This limitation on the number of points leads to significant errors. This new method, however, unfolds a series of translational motions of the plane into an affine coordinate matrix. The matrix calculation comprehensively considers all points on the plane at all translational positions, rather than just points at a single position. Using more points results in higher accuracy. Assume the unit direction vector of the single-axis displacement stage 4 in the world coordinate system is (a1, a2, a3). Ｔ The transformation from a non-orthogonal affine coordinate system to an orthogonal coordinate system can be represented by an affine transformation:

[0092]

[0093] Based on the pinhole camera model, point p in the world coordinate system wUsing AP substitution, the projection relationship between point p in the homogeneous coordinate system of the left or right image and point p in the affine coordinate system is as follows:

[0094]

[0095] In the formula: α is the scale factor of the coordinates p of the projected point on the image, p is a point in the coordinate system of the left or right image, and M is the projection matrix. Let P be the homogeneous coordinate form, R be the 3×3 rotation matrix, A be the affine transformation matrix, and t be the translation vector. R, A, and t are related to the initial position of the camera relative to the checkerboard target and are collectively referred to as the camera's extrinsic parameters. K is the camera's intrinsic parameter matrix, where f x and f y γ represents the focal length in the x and y directions. skew c represents the pixel tilt coefficient. x and c y Indicates the coordinates of the projection center;

[0096] based on The projection matrix M is obtained by using least squares estimation of the affine coordinate matrix composed of p and .

[0097] The affine coordinate system projection model adds an affine transformation matrix A compared to the general camera projection model, which is used as an unknown parameter in the subsequent equation solving.

[0098] Step 3: Solve the non-homogeneous equations to obtain the initial values ​​of the left and right camera parameters.

[0099] Existing planar calibration methods do not consider the relationship between the vanishing points and vanishing lines of the two cameras, and calculate their parameters independently during the solution process. This method uses non-homogeneous equations to establish the connection between the vanishing points and vanishing lines of the two cameras, allowing the absolute quadratic curves of the two cameras to be solved in the same set of non-homogeneous equations. The simultaneous constraint of the two cameras improves the stability of parameter calculation compared to the independent calculation of a single camera.

[0100] Expanding the projection matrix M column-wise, M = [m1 m2 m3 m4], the absolute quadratic curve of the camera can be represented by the intrinsic parameter matrix: B = K ―T K ―1 Substituting into equation (2), the constraints of the affine coordinate system on the camera parameters can be described as follows:

[0101]

[0102] This is a set of non-homogeneous equations. Similarly, the projection matrix M′ and the absolute quadratic curve matrix B′ of the right eye camera are also constrained by the affine coordinate system.

[0103] During camera projection, according to the principle of geometric perspective transformation, m1 and m2 represent the vanishing lines of the planar target on the image, and m3 is the vanishing point of the translation direction on the image. Since the translation direction vector of the planar target in one movement is (a1, a2, a3) T Similarly, both camera projection models share the same affine matrix A. The vanishing lines and vanishing points are mapped to the 3D world coordinate system via an absolute quadratic curve B. The angle between the translation direction and the planar target provides two binocular constraints:

[0104]

[0105] Since B and B′ are real symmetric matrices, each containing 6 unknown parameters, the unknown parameters of the two matrices are expanded into column vectors b and b′, ​​and b and b′ are combined to form column vector X. During each translation process, a set of non-homogeneous equations is established by combining equations 3 and 4, as expressed by equation 5 below:

[0106]

[0107] In the equation: C is the coefficient matrix for each parameter in b and b′ established according to equations 3 and 4, and Y is the parameter on the right side of equations 3 and 4. Since the scaling factor α of the affine coordinate matrix for each different plane rotation direction is different, the non-homogeneous constraint equation of equation 5 is used for one set of affine coordinate matrices, transforming equation 3 into a homogeneous equation as shown in equation 6:

[0108]

[0109] Similarly, Equation 6 can also be used for the coefficient matrix of each parameter in b and b′. Applying homogeneous equation 6 to the remaining affine coordinate matrices, and combining it with the non-homogeneous equations of equation 5, we can form a system of non-homogeneous equations for solving all affine coordinate matrices and simultaneously determining the camera parameters, as expressed in equation 7:

[0110]

[0111] In the formula: C s The coefficient matrix derived from C in Equation 5 and Equation 6 Y is obtained by concatenating rows. s From Equation 5, Y and the zero vector 0 1×6(N―1) The result is obtained by splicing together N, which is the number of affine coordinate matrices.

[0112] The absolute quadratic curve matrix is ​​solved by solving the non-homogeneous equation system, and the matrix is ​​decomposed to obtain the initial intrinsic parameters of the left and right cameras.

[0113] In each translation process, a set of non-homogeneous equations can be established by combining equations (3) and (4), and different plane directions can be collected and translated to generate affine meshes to improve the robustness of the algorithm.

[0114] Solve the absolute quadratic curve matrix according to the non-homogeneous equation system, decompose the matrix to obtain the intrinsic parameters of the left and right cameras, and substitute them into equation (2) to obtain the extrinsic parameters of the cameras.

[0115] Step 4: Nonlinearly optimize the camera's intrinsic and extrinsic parameters.

[0116] Since the non-homogeneous equation in step 3 does not consider the camera distortion coefficient, it solves for approximate values ​​of the camera parameters. This invention incorporates all the camera's intrinsic (same), extrinsic (different), and translation direction vectors into the nonlinear optimization function, using the reprojection error in the affine projection model of equation (2) as the objective function. The reprojection error represents the difference between the reprojected image coordinates after substituting the camera parameters and the actual coordinates on the image. A nonlinear optimization algorithm is used to iterate the camera parameters to minimize the reprojection error, outputting the final optimized camera intrinsic and extrinsic parameters.

[0117] In summary, the binocular camera calibration method provided in this application eliminates the problem that arbitrarily placed targets are difficult to control in terms of the uniformity of the planar target spatial distribution, leading to camera parameters being affected by densely distributed planar positions and getting trapped in local optima during optimization. By acquiring data from multiple depth-direction acquisition locations, the strong correlation between the near and far distribution of the planar targets and the calculation of the camera focal length is revealed, allowing for the estimation of a stable and accurate value for the camera focal length. Furthermore, the calibration accuracy of camera parameters is improved by controlling the acquisition of the calibration area and increasing dimensional constraints on the acquisition space.

[0118] See Figure 4 This application embodiment can also provide a binocular camera calibration device, such as... Figure 4 As shown, the device may include:

[0119] The affine coordinate matrix acquisition unit 401 is used to adjust the checkerboard plane target to different rotation angles. The checkerboard plane target is fixedly connected to a single-axis displacement stage. The single-axis displacement stage is used to translate the checkerboard plane target multiple times at each angle, with each translation being a fixed distance. A set of affine coordinate data is formed by recording the affine coordinate points, left-eye image coordinates, and right-eye image coordinates of the corresponding non-orthogonal affine coordinate system after each translation at each angle. The affine coordinate points include the coordinates of each corner point of the checkerboard plane target and the fixed distance. The left-eye image coordinates are the coordinates of each corner point detected in the left-eye image acquired by the left-eye camera, and the right-eye image coordinates are the coordinates of each corner point detected in the right-eye image acquired by the right-eye camera. An affine coordinate matrix is ​​generated using all the affine coordinate data.

[0120] The affine projection model establishment unit 402 is used to establish an affine projection model from the affine coordinate system to the camera image coordinate system based on the affine coordinate matrix, the affine matrix three-dimensional coordinate transformation and the camera projection model, and to obtain the projection matrix.

[0121] The initial intrinsic parameter acquisition unit 403 is used to obtain the initial intrinsic parameters of the left eye camera and the right eye camera by establishing non-homogeneous equations based on the plane projection cancellation line and translation direction cancellation point, combined with the affine projection model.

[0122] The initial extrinsic parameter acquisition unit 404 is used to solve for several initial extrinsic parameters of the left eye camera and the right eye camera after each translation at each angle using the initial intrinsic parameters and the affine projection model;

[0123] The target intrinsic and extrinsic parameter acquisition unit 405 is used to substitute the initial intrinsic parameters, several initial extrinsic parameters, and the translation direction vector of each translation into a nonlinear optimization method to optimize the intrinsic and extrinsic parameters of the left eye camera and the right eye camera to obtain the target intrinsic and extrinsic parameters.

[0124] This application embodiment can also provide a binocular camera calibration device, the device including a processor and a memory:

[0125] The memory is used to store program code and transmit the program code to the processor;

[0126] The processor is used to execute the steps of the above-described binocular camera calibration method according to the instructions in the program code.

[0127] like Figure 5 As shown in the figure, a binocular camera calibration device provided in this application embodiment may include: a processor 10, a memory 11, a communication interface 12, and a communication bus 13. The processor 10, memory 11, and communication interface 12 all communicate with each other through the communication bus 13.

[0128] In this embodiment, the processor 10 may be a central processing unit (CPU), an application-specific integrated circuit, a digital signal processor, a field-programmable gate array, or other programmable logic devices.

[0129] The processor 10 can call the program stored in the memory 11. Specifically, the processor 10 can execute the operations in the embodiment of the binocular camera calibration method.

[0130] The memory 11 is used to store one or more programs. The programs may include program code, which includes computer operation instructions. In this embodiment, the memory 11 stores at least a program for implementing the following functions:

[0131] The checkerboard-patterned target is adjusted to different rotation angles, and the target is fixedly connected to a single-axis translation stage. The target is translated multiple times at each angle, with each translation involving a fixed distance. A set of affine coordinate data is formed by recording the affine coordinates of the non-orthogonal affine coordinate system points, the left-eye image coordinates, and the right-eye image coordinates after each translation at each angle. The affine coordinates include the coordinates of each corner point of the checkerboard-patterned target and the fixed distance. The left-eye image coordinates are the coordinates of each corner point detected in the left-eye image acquired by the left-eye camera, and the right-eye image coordinates are the coordinates of each corner point detected in the right-eye image acquired by the right-eye camera. An affine coordinate matrix is ​​generated using all the affine coordinate data.

[0132] Using the affine coordinate matrix, based on the affine matrix three-dimensional coordinate transformation and the camera projection model, an affine projection model from the affine coordinate system to the camera image coordinate system is established, and the projection matrix is ​​obtained.

[0133] Based on the plane projection cancellation line and translation direction cancellation point, the initial intrinsic parameters of the left and right cameras are obtained by solving the non-homogeneous equations in conjunction with the affine projection model.

[0134] Using the initial intrinsic parameters and the affine projection model, we can obtain several initial extrinsic parameters for the left and right cameras after each translation at each angle.

[0135] The initial intrinsic parameters, several initial extrinsic parameters, and the translation direction vector for each translation are substituted into a nonlinear optimization method to optimize the intrinsic and extrinsic parameters of the left and right cameras, thereby obtaining the target intrinsic and extrinsic parameters.

[0136] In one possible implementation, the memory 11 may include a program storage area and a data storage area. The program storage area may store the operating system and applications required for at least one function (such as file creation or data read / write). The data storage area may store data created during use, such as initialization data.

[0137] In addition, memory 11 may include high-speed random access memory, and may also include non-volatile memory, such as at least one disk storage device or other volatile solid-state storage device.

[0138] Communication interface 12 can be an interface for the communication module, used to connect with other devices or systems.

[0139] Of course, it should be noted that, Figure 5The structure shown does not constitute a limitation on the binocular camera calibration device in the embodiments of this application. In practical applications, the binocular camera calibration device may include... Figure 5 More or fewer components as shown, or combinations of certain components.

[0140] This application embodiment may also provide a computer-readable storage medium for storing program code for executing the steps of the above-described binocular camera calibration method.

[0141] It should be noted that, in this document, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0142] As can be seen from the above description of the embodiments, those skilled in the art can clearly understand that this application can be implemented by means of software plus necessary general-purpose hardware platforms. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute the methods described in various embodiments or some parts of the embodiments of this application.

[0143] The various embodiments in this specification are described in a progressive manner. Similar or identical parts between embodiments can be referred to mutually. Each embodiment focuses on describing the differences from other embodiments. In particular, for system or system embodiments, since they are basically similar to method embodiments, the description is relatively simple, and relevant parts can be referred to the descriptions in the method embodiments. The systems and system embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. Those skilled in the art can understand and implement this without creative effort.

[0144] The above description is merely a preferred embodiment of the present invention and is not intended to limit the scope of protection of the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention are included within the scope of protection of the present invention.

Claims

1. A method for calibrating a binocular camera, characterized in that, include: The checkerboard-patterned target is adjusted to different rotation angles, and the checkerboard-patterned target is fixedly connected to a single-axis displacement stage. The checkerboard-shaped target is translated multiple times at each angle using the single-axis translation stage, with each translation involving a fixed distance. A set of affine coordinate data is formed by recording the affine coordinates of the corresponding non-orthogonal affine coordinate system points, left-eye image coordinates, and right-eye image coordinates after each translation at each angle. The affine coordinates include the coordinates of each corner point of the checkerboard-shaped target and the fixed distance. The left-eye image coordinates are the coordinates of each corner point detected in the left-eye image acquired by the left-eye camera, and the right-eye image coordinates are the coordinates of each corner point detected in the right-eye image acquired by the right-eye camera. An affine coordinate matrix is ​​generated using all the affine coordinate data. Using the affine coordinate matrix, based on the affine matrix 3D coordinate transformation and the camera projection model, an affine projection model from the affine coordinate system to the camera image coordinate system is established, and the projection matrix is ​​obtained. The non-orthogonal affine coordinate system is then transformed to an orthogonal coordinate system using the affine coordinate matrix based on the affine matrix 3D coordinate transformation. The affine matrix 3D coordinate transformation is represented by Equation 1: In the formula: A point in the world coordinate system. Let be the affine transformation matrix. A point in an affine coordinate system, Let these be the coordinates of the corner point. For a fixed distance, This is the unit direction vector in the world coordinate system; The affine projection model is represented by the following equation 2: In the formula: Coordinates of the projection points on the image The scaling factor, Points in the homogeneous coordinate system of the left or right eye image. For the projection matrix, for homogeneous coordinate form, It is a 3×3 rotation matrix. Let be the affine transformation matrix. It is a translation vector. For the camera's external parameters, Let be the camera intrinsic parameter matrix, where and This represents the focal length in the x and y directions. Indicates the pixel tilt coefficient. and Indicates the coordinates of the projection center; based on and The projection matrix is ​​obtained by using the least squares estimation of the affine coordinate matrix. ; Based on the plane projection cancellation line and translation direction cancellation point, the initial intrinsic parameters of the left and right cameras are obtained by solving the non-homogeneous equations in conjunction with the affine projection model. The method for calculating the initial intrinsic parameters includes: Projection matrix Expand by column, The absolute quadratic curve of the camera is represented by an intrinsic parameter matrix: ,Will Substituting into equation (2), the constraint of the affine coordinate system on the camera parameters is described in equation (3): In the formula: and This represents the disappearance line of a planar target on an image. It is the point on the image where the translation direction disappears; The disappearance line and the disappearance point are passed through an absolute quadratic curve. Mapped to the three-dimensional world coordinate system, the angle between the translation direction and the checkerboard plane target provides two binocular constraints, represented by Equation 4 below: The absolute quadratic curve matrix is ​​solved by solving the non-homogeneous equation system, and the matrix is ​​decomposed to obtain the initial intrinsic parameters of the left and right cameras. Using the initial intrinsic parameters and the affine projection model, we can obtain several initial extrinsic parameters for the left and right cameras after each translation at each angle. The initial intrinsic parameters, several initial extrinsic parameters, and the translation direction vector for each translation are substituted into a nonlinear optimization method to optimize the intrinsic and extrinsic parameters of the left and right cameras, thereby obtaining the target intrinsic and extrinsic parameters.

2. The binocular camera calibration method according to claim 1, characterized in that, Sure and These are real symmetric matrices, each containing 6 unknown parameters. The unknown parameters of the two matrices are expanded into column vectors. and ,Will and Composed of column vectors During each translation process, a set of non-homogeneous equations is established by combining equations 3 and 4, as expressed by equation 5 below: In the formula: It is based on Equations 3 and 4 that the following is established regarding and The coefficient matrix of each parameter. These are the parameters on the right-hand side of equations 3 and 4; The non-homogeneous constraint equation of Equation 5 is used for one set of affine coordinate matrices, and Equation 3 is transformed into a homogeneous equation, which is expressed by Equation 6 below: The non-homogeneous equations in Equation 5, which combine to form the complete affine coordinate matrix, are used to solve the non-homogeneous equations for the camera parameters. These equations are expressed in Equation 7. In the formula: From Equation 5 The coefficient matrix derived from equation 6 Obtained by concatenating lines From Equation 5 and zero vector obtained by piecing together It represents the number of affine coordinate matrices.

3. The binocular camera calibration method according to claim 2, characterized in that, Substituting the initial intrinsic parameters into Equation 2, we obtain several initial extrinsic parameters corresponding to the left and right cameras after each translation at each angle.

4. The binocular camera calibration method according to claim 1, characterized in that, Substituting the initial intrinsic parameters, several initial extrinsic parameters, and the translation direction vector for each translation into a nonlinear optimization method to optimize the intrinsic and extrinsic parameters of the left and right cameras, the target intrinsic and extrinsic parameters are obtained, including: Substitute the initial intrinsic parameters, several initial extrinsic parameters, and the translation direction vector for each translation into the nonlinear optimization function, and use the reprojection error in the affine projection model of Equation 2 as the objective function; the reprojection error represents the difference between the reprojected image coordinates after substituting the camera parameters and the actual coordinates on the image; The camera parameters are iterated using a nonlinear optimization algorithm to minimize the reprojection error, and the final optimized intrinsic and extrinsic parameters of the camera are output.

5. A binocular camera calibration device, characterized in that, The method for performing the binocular camera calibration method according to any one of claims 1-4 includes: An affine coordinate matrix acquisition unit is used to adjust a checkerboard-shaped target to different rotation angles. The checkerboard-shaped target is fixedly connected to a single-axis translation stage. The single-axis translation stage is used to translate the checkerboard-shaped target multiple times at each angle, with each translation being a fixed distance. A set of affine coordinate data is formed by recording the affine coordinates of the corresponding non-orthogonal affine coordinate system points, left-eye image coordinates, and right-eye image coordinates after each translation at each angle. The affine coordinates include the coordinates of each corner point of the checkerboard-shaped target and the fixed distance. The left-eye image coordinates are the coordinates of each corner point detected in the left-eye image acquired by the left-eye camera, and the right-eye image coordinates are the coordinates of each corner point detected in the right-eye image acquired by the right-eye camera. An affine coordinate matrix is ​​generated using all the affine coordinate data. The affine projection model establishment unit is used to establish an affine projection model from the affine coordinate system to the camera image coordinate system based on the affine coordinate matrix, the affine matrix three-dimensional coordinate transformation, and the camera projection model, and to obtain the projection matrix. The initial intrinsic parameter acquisition unit is used to obtain the initial intrinsic parameters of the left eye camera and the right eye camera by solving non-homogeneous equations based on the plane projection cancellation line and translation direction cancellation point, combined with the affine projection model. The initial extrinsic parameter acquisition unit is used to solve for several initial extrinsic parameters of the left eye camera and the right eye camera after each translation at each angle using the initial intrinsic parameters and the affine projection model; The target intrinsic and extrinsic parameter acquisition unit is used to substitute the initial intrinsic parameters, several initial extrinsic parameters, and the translation direction vector for each translation into a nonlinear optimization method to optimize the intrinsic and extrinsic parameters of the left eye camera and the right eye camera to obtain the target intrinsic and extrinsic parameters.

6. A binocular camera calibration device, characterized in that, The device includes a processor and a memory: The memory is used to store program code and transmit the program code to the processor; The processor is used to execute the binocular camera calibration method according to any one of claims 1-4 according to the instructions in the program code.

7. A computer-readable storage medium, characterized in that, The computer-readable storage medium is used to store program code for executing the binocular camera calibration method according to any one of claims 1-4.

Images