Binocular camera calibration method, device and electronic equipment

Abstract

The present application provides a binocular camera calibration method, device and electronic device. The method constructs a distortion basic matrix through the lens distortion information and basic matrix of the binocular camera to be calibrated, and the distortion basic matrix is ​​the quantity to be solved. The binocular camera to be calibrated is used to collect a first binocular image, and the coordinates of the same-name points are extracted to construct a first equation with the distortion basic matrix equation. The estimated value of the distortion basic matrix used to calibrate the binocular camera to be calibrated is calculated by the first equation to construct an objective function for calculating the optimal distortion basic matrix. The objective function is optimized using a least squares optimization algorithm to obtain the optimal distortion basic matrix. The method is divided into two stages of preliminary estimation and optimization in the process of obtaining the optimal distortion basic matrix, which alleviates the problem that the nonlinear optimization method is not easy to obtain results. At the same time, the description of radial distortion parameters is added in the process of obtaining the optimal distortion basic matrix to improve the calibration efficiency and accuracy of the binocular camera to be calibrated.

Description

Technical Field

[0001] The present application relates to the field of machine vision technology, and in particular to a binocular camera calibration method, device and electronic equipment. Background Art

[0002] Binocular cameras used in the field of machine vision need to be calibrated before use to determine the model parameters of the binocular camera imaging model, and then more accurately establish the mapping relationship between the camera image and the physical world, thereby improving the imaging accuracy of the binocular camera.

[0003] In combination with actual needs, some binocular cameras use short focal length lenses with large lens angle and wide field of view. This type of binocular camera will produce radial distortion when acquiring images, which will easily produce large errors during image processing. Therefore, when calibrating a binocular camera using a short focal length lens, it is necessary to calibrate the radial distortion parameters to correct the image during image processing.

[0004] In the process of calibrating a binocular camera, it is necessary to calibrate the basic matrix and radial distortion of the binocular camera to improve the accuracy of the binocular camera to be calibrated. When the basic matrix is ​​calibrated using a nonlinear optimization method, the calculation results are not easy to converge, resulting in difficulty in calibration. Moreover, when the calibration of the basic matrix and the calibration of the radial distortion are two independent steps, the calibration efficiency is low. Summary of the invention

[0005] The present application provides a binocular camera calibration method, device and electronic device to solve the problem of low calibration efficiency when the basic matrix and radial distortion calibration of the binocular camera are performed step by step.

[0006] In a first aspect, the present application provides a binocular camera calibration method, comprising:

[0007] Determine the distortion basic matrix equation according to the lens distortion information and basic matrix of the binocular camera to be calibrated;

[0008] Using the binocular camera to be calibrated to capture a first binocular image, and extracting multiple pairs of coordinates of points of the same name from the first binocular image;

[0009] Constructing a first equation based on the basic matrix equation and a plurality of pairs of coordinates of points of the same name to calculate an estimated value of the basic matrix;

[0010] Constructing an objective function for calculating an optimal distortion fundamental matrix according to the first equation and the estimated value of the distortion fundamental matrix;

[0011] The objective function is optimized using a least squares optimization algorithm to obtain an optimal distortion basis matrix.

[0012] In some feasible embodiments, constructing a first equation based on the basic equation and multiple pairs of coordinates of points with the same name includes:

[0013] Constructing a first matrix according to the plurality of pairs of coordinates of the same-name points, and constructing a first vector according to undetermined parameters in the basic matrix equation for characterizing lens distortion information and the basic matrix;

[0014] A first equation is obtained based on the first matrix and the first vector.

[0015] In some feasible embodiments, constructing a first equation based on the distortion basic matrix equation and multiple pairs of coordinates of points of the same name to calculate an estimated value of the distortion basic matrix further includes:

[0016] Performing singular value decomposition on the first matrix, and extracting a first singular value from candidate singular values ​​obtained by the singular value decomposition;

[0017] A first singular vector corresponding to the first singular value is calculated.

[0018] In some feasible embodiments, the method further includes:

[0019] constructing a second matrix based on the first singular vectors;

[0020] Singular value decomposition is performed on the second matrix, and second singular values ​​are extracted from candidate singular values ​​obtained by the singular value decomposition.

[0021] In some feasible embodiments, the method further includes:

[0022] constructing a third matrix based on the second singular values;

[0023] According to a matrix null space algorithm, a left null space solution set and a right null space solution set of the third matrix are calculated.

[0024] In some feasible embodiments, the method further includes:

[0025] An imaging surface coordinate system is established on the imaging plane of the binocular camera to be calibrated; the imaging surface coordinate system includes a first imaging surface coordinate system corresponding to the left null space solution set and a second imaging surface coordinate system corresponding to the right null space solution set;

[0026] A first intersection point of the left null space solution set and a first imaging plane coordinate system is calculated, and a second intersection point of the right null space solution set and a second imaging plane coordinate system is calculated.

[0027] In some feasible embodiments, the method further includes:

[0028] Determine a first optical axis on the first imaging plane coordinate system, and determine a second optical axis on the second imaging plane coordinate system;

[0029] Calculate a first shortest distance point on the first optical axis to the left null space solution set, and calculate a second shortest distance point on the second optical axis to the right null space solution set;

[0030] A first radial distortion parameter is calculated according to the first shortest distance point and the radial distortion vector, and a second radial distortion parameter is calculated according to the second shortest distance point and the radial distortion vector.

[0031] In some feasible embodiments, the method further includes:

[0032] Select the first undetermined coefficient and the second undetermined coefficient to construct a fourth matrix;

[0033] Calculating a specific value of the first undetermined parameter according to the coordinates of the first intersection point and the coordinates of the second intersection point;

[0034] Based on the first radial distortion parameter, the second radial distortion parameter and the first undetermined parameter, a linear equation group for constructing a second equation is obtained;

[0035] According to the linear equation group, a second equation consisting of a fifth matrix and a second vector is constructed; the fifth matrix includes known quantities; the second vector includes unknown quantities, and the second vector includes a second unknown parameter.

[0036] In some feasible embodiments, the method further includes:

[0037] performing a singular value decomposition on the fifth matrix to extract a fifth minimum singular value;

[0038] Based on the fifth matrix and the fifth singular value, calculating a singular vector corresponding to the minimum singular value of the fifth matrix to determine a specific value of a second undetermined parameter;

[0039] An estimated value of the distortion basic matrix is ​​determined based on the first radial distortion parameter, the second radial distortion parameter, a specific value of the first to-be-determined parameter, and a specific value of the second to-be-determined parameter.

[0040] In a second aspect, the present application provides a binocular camera calibration device, including: a preprocessing module, a collection module, a calculation module and an optimization module;

[0041] The preprocessing module is used to construct a distortion basic matrix equation according to the lens distortion information of the binocular camera to be calibrated and the basic matrix;

[0042] The acquisition module is used to use the binocular camera to be calibrated to acquire a first binocular image, and to extract multiple pairs of coordinates of points of the same name from the first binocular image;

[0043] The operation module constructs a first equation based on the basic matrix equation and a plurality of pairs of coordinates of points of the same name to calculate an estimated value of the distortion basic matrix;

[0044] The operation module is further used to construct an objective function for calculating an optimal distortion basic matrix according to the first equation and the estimated value of the distortion basic matrix;

[0045] The optimization module is used to optimize the objective function using a least squares optimization algorithm to obtain an optimal distortion basic matrix.

[0046] In a third aspect, the present application provides an electronic device comprising a memory, a processor, and a computer program stored in the memory and runnable on the processor, wherein the processor executes the computer program to perform the method for calibrating the basic distortion matrix of a binocular camera as described in the above technical solution.

[0047] It can be seen from the above technical content that the present application provides a binocular camera calibration method, device and electronic device. The method constructs a distortion basic matrix as a calibration basis through the lens distortion information and basic matrix of the binocular camera to be calibrated. The first binocular image is collected using the binocular camera to be calibrated, and the coordinates of the same-name points are extracted to construct a first equation with the distortion basic matrix equation. The estimated value of the distortion basic matrix used to calibrate the binocular camera to be calibrated is calculated by the first equation to construct an objective function for calculating the optimal distortion basic matrix. The objective function is optimized using the least squares optimization algorithm to obtain the optimal distortion basic matrix. The method is divided into two stages of preliminary estimation and optimization in the process of obtaining the optimal distortion basic matrix, which alleviates the problem that it is difficult to obtain results for nonlinear optimization methods. At the same time, the description of radial distortion parameters is added in the process of obtaining the optimal distortion basic matrix to improve the calibration efficiency and accuracy of the binocular camera to be calibrated. BRIEF DESCRIPTION OF THE DRAWINGS

[0048] In order to more clearly illustrate the technical solution of the present application, the drawings required for use in the embodiments are briefly introduced below. Obviously, for ordinary technicians in this field, other drawings can be obtained based on these drawings without any creative work.

[0049] Figure 1 A schematic diagram of a process for calibrating a binocular camera distortion basic matrix provided in an embodiment of the present application;

[0050] Figure 2 A schematic diagram of decomposition of the first equation provided in an embodiment of the present application;

[0051] Figure 3 A schematic diagram of a second matrix decomposition provided in an embodiment of the present application;

[0052] Figure 4 A schematic diagram of the relationship between the imaging plane, optical axis and zero space position provided in an embodiment of the present application;

[0053] Figure 5 A fourth matrix decomposition schematic diagram provided in an embodiment of the present application;

[0054] Figure 6 A fifth matrix decomposition diagram provided in an embodiment of the present application. DETAILED DESCRIPTION

[0055] The following embodiments are described in detail, and examples thereof are shown in the accompanying drawings. When the following description refers to the drawings, unless otherwise indicated, the same numbers in different drawings represent the same or similar elements. The implementations described in the following embodiments do not represent all implementations consistent with the present application. They are only examples of systems and methods consistent with some aspects of the present application as detailed in the claims.

[0056] Before a binocular camera is put into use, the model parameters of the imaging model need to be calibrated to more accurately establish the mapping relationship between the camera image and the physical world and improve the imaging accuracy of the binocular camera. Some binocular cameras use short focal length lenses, which will produce radial distortion during operation, resulting in reduced accuracy in acquiring images. Therefore, the calibration of this type of binocular camera includes the calibration of the basic matrix and the calibration of the radial distortion parameters to improve its accuracy.

[0057] During the calibration process, the calibration of the basic matrix and the calibration of the radial distortion parameters are two independent parts. Even if they are performed simultaneously, the calibration efficiency is low. In addition, directly using the nonlinear optimization method to obtain the basic matrix is ​​prone to fail to obtain the parameters used to describe the basic matrix due to the characteristics of the algorithm itself and the calibration data.

[0058] In view of the above problems, if Figure 1 As shown, some embodiments of the present application provide a binocular camera calibration method, including:

[0059] S100: determining a distortion basic matrix equation according to the lens distortion information of the binocular camera to be calibrated and the basic matrix;

[0060] In some embodiments, the basic matrix equation of the binocular camera simultaneously describes the influence of lens distortion and binocular camera position on image acquisition accuracy. Therefore, the distortion basic matrix equation can be solved according to the lens distortion information and configuration information of the binocular camera as the basis for subsequent calibration.

[0061] S200: using the binocular camera to be calibrated to acquire a first binocular image, and extracting a plurality of pairs of coordinates of points of the same name from the first binocular image;

[0062] The number of first binocular images is at least 1, and the number of same-name point coordinates extracted from the first binocular image is at least 15 pairs for providing calibration data. It is understandable that when the amount of calibration data required is large, it can be achieved by increasing the number of first binocular images collected and the number of same-name point coordinates extracted. Conversely, the number of first binocular images collected and the number of same-name point coordinates extracted can be reduced. It should be noted that in some embodiments, the calibration of the binocular camera can be achieved by collecting one binocular image. Among them, 15 pairs of same-name point coordinates can be used to form a 4×4 matrix.

[0063] It should be noted that when constructing a matrix based on the coordinates of the same-name points, it is necessary to detect whether the binocular camera to be calibrated has radial distortion. In some embodiments, if the lens of the binocular camera does not have radial distortion, the binocular camera conforms to the pinhole imaging model. u ,y u For example, F u For x u ,y u The corresponding basic matrix can be expressed as n u represents a straight line on the imaging plane of the binocular camera, The straight line is the projection of a straight line in three-dimensional space on the camera imaging plane. Based on the characteristics of the binocular camera, the plane where the projection is located can be respectively Distinguish between the left and right imaging planes.

[0064] In other embodiments, if the binocular camera to be calibrated has lens distortion, taking a set of coordinates x and y of the same name as an example, F is the distortion basic matrix corresponding to x and y. Because of the lens distortion, the distortion basic matrix contains the parameter λ used to describe the radial distortion. L ,λ R . Among them, x, y, x u ,y u Both are used to represent the coordinate vectors of a pair of points of the same name in the left and right imaging planes, where x u ,y u The subscript u is used to distinguish the coordinates of the same-name points without radial distortion from the coordinates of the same-name points with mirror distortion.

[0065] It can be understood that the embodiment of the present application aims to establish an association relationship between a basic matrix without radial distortion and a distortion basic matrix with radial distortion, so as to obtain an estimated value of the distortion basic matrix of the binocular camera to be calibrated with radial distortion, and optimize the distortion basic matrix according to the estimated value to obtain the optimal distortion basic matrix to complete the calibration of the binocular camera to be calibrated.

[0066] In order to introduce distortion parameters into the basic matrix, a mathematical model describing radial distortion of the lens is proposed in the embodiment of the present application:

[0067] x u =δ -1 (x)=(wx,wy,w 2 +λ(x 2 +y 2 )) T

[0068] Among them, λ is the distortion parameter introduced by the model, λ L ,λ R They are used to represent the distortion parameters of the two lenses of the binocular camera; w is the scaling factor, x and y are the coordinates of the vector on the plane; T is used to represent the transpose operation of the matrix; δ(x) is the distortion function, δ -1 (x) is the inverse function operation of the distortion function.

[0069] According to the position relationship equation of points and lines in space, when x u On the straight line n u When , it can be expressed as:

[0070] (n u ) T ·x u =0

[0071] At this time, substituting the lens radial distortion model into the above formula, we can get:

[0072]

[0073] The formula is consistent with the general expression of a circle on a plane, so it can be further organized as:

[0074] a(x 2 +y 2 )+d·wx+e·wy+f·w 2 =0

[0075] Therefore, when there is a straight line in the three-dimensional space, after it is projected onto the imaging surface of the binocular camera by a distorted lens, the resulting image is a circle, which can be expressed as:

[0076]

[0077] in, are all four-dimensional vectors. The vector representation of can be defined as:

[0078]

[0079] Right now Will Substituting into the above formula, we can get It is known that without considering lens distortion, the same-name point x u ,y u With the camera basic matrix F u The relationship between them conforms to:

[0080] (y u ) T F u x u =0

[0081] in, is a 3x3 matrix. Substitute (y u ) T F u x u = 0, that is, the expression of the distortion basic matrix containing the distortion in four-dimensional space can be obtained:

[0082]

[0083] Further adjusted to:

[0084]

[0085] in, This is the estimated value of the distortion matrix of the binocular camera to be calibrated taking into account the lens distortion. The 16 elements contained in it need to meet certain constraints before they can be used as the estimated value of the distortion matrix for subsequent optimization. Let the estimated value of the matrix be The specific constraints are:

[0086]

[0087] Based on the constraints, a particular solution of the basic matrix can be obtained, It should be noted that the special solution vector D can be described by a radial distortion vector in the embodiment of the present application, and can be used to obtain radial distortion parameters. The radial distortion vector is located at the optical axis of the imaging plane. superior,

[0088] Based on the above basic matrix equation and the coordinates of the points of the same name to be calibrated, we can solve The values ​​of the 16 elements in , and then determine the estimated value of the distortion basic matrix. In some embodiments, a pair of images captured by the binocular camera to be calibrated can be used (I L ,I R ) indicates that at least 15 pairs of points with the same name (x) are extracted from the image. k ,y k ), where k = 1, 2, ..., K, K ≥ 15. At least 15 pairs of coordinates of points with the same name can be constructed into four-dimensional homogeneous coordinate point pairs. This is done to align the dimension of the estimated value expression with the distortion basic matrix, making it easier to obtain the estimated value later.

[0089] S300: constructing a first equation based on the distortion basic matrix equation and a plurality of pairs of coordinates of points with the same name to calculate an estimated value of the distortion basic matrix;

[0090] The first equation can be organized into the form of the product of the first matrix and the first vector, and the first matrix is ​​gradually decomposed into singular values, and the unknown parameters contained in the first vector are obtained, and finally an estimated value of the distortion basic matrix is ​​obtained. In some embodiments, the first equation is constructed based on the basic equation and multiple pairs of coordinates of the same name points, including:

[0091] Constructing a first matrix according to the plurality of pairs of coordinates of the same-name points, and constructing a first vector according to undetermined parameters in the distortion basic matrix equation for characterizing lens distortion information and the basic matrix;

[0092] A first equation is obtained based on the first matrix and the first vector.

[0093] Four-dimensional homogeneous point pair Substitute the value of It can be organized into the form of A·f1=0, where A is an m×n square matrix, m≥15, n=15, and the elements of the square matrix are based on It turns out that f1 is a 15×1 column vector containing the matrix There are 15 undetermined parameters in the equation, namely f1=(c 11, ...,c 43 ), here it is agreed It has been normalized, that is, c 44 =1.

[0094] The first equation can form a corresponding relationship with the basic matrix containing distortion information, so each element in the distortion basic matrix can be indirectly obtained by obtaining the first matrix and the first vector in the first equation. Figure 2As shown, in some embodiments, a first equation is constructed based on the distortion basic matrix equation and multiple pairs of coordinates of points of the same name to calculate an estimated value of the distortion basic matrix, and further includes:

[0095] Performing singular value decomposition on the first matrix, and extracting a first minimum singular value from candidate singular values ​​obtained by the singular value decomposition;

[0096] A first singular vector corresponding to the first minimum singular value is calculated.

[0097] It can be understood that in order to describe the correspondence between singular values ​​and matrices, in an embodiment of the present application, the minimum singular value obtained by decomposing the first matrix is ​​called the first singular value, the singular value obtained by decomposing the second matrix is ​​called the second singular value, and the minimum singular value obtained by decomposing the fifth matrix is ​​called the fifth singular value.

[0098] Performing singular value decomposition on the first matrix can obtain a 15×15 square matrix W1 and a 15×15 diagonal matrix Σ1. Among them, Σ1 contains 15 singular values ​​corresponding to matrix A, and the first singular value is the smallest singular value among the 15 singular values. Based on the smallest singular value, the first singular vector f1 corresponding to it can also be determined. The first singular vector contains 15 elements and can be used to construct the second matrix. It should be noted that in the process of singular value decomposition, each singular value obtained is equivalent to the weight of the matrix corresponding to the singular value. Therefore, the matrix corresponding to the smaller singular value has a smaller weight for the image data, so it can be regarded as noise data. Therefore, taking the minimum singular value is equivalent to denoising the image data, which can improve the calibration accuracy.

[0099] like Figure 3 As shown, in some embodiments, after obtaining the first singular vector f1, the method further includes:

[0100] constructing a second matrix based on the first singular vectors;

[0101] Singular value decomposition is performed on the second matrix, and second singular values ​​are extracted from candidate singular values ​​obtained by the singular value decomposition.

[0102] Based on the 15 elements contained in the first singular vector, a constant 1 is added as the 16th element, and then a new 4×4 matrix, namely the second matrix B, can be constructed. Performing singular value decomposition on the second matrix B can obtain a 4x4 square matrix W2 and a 4x4 diagonal matrix Σ2, where Σ2 contains 4 singular values ​​corresponding to matrix B. The two largest singular values ​​are selected as the second singular values ​​from the four singular values. It can be understood that the two largest singular values ​​can be two equal values ​​or the two larger ones of the four singular values. Performing singular value decomposition on the second matrix is ​​a dimensionality reduction operation, which can make the subsequent calculation of the estimated value of the basic matrix meet the constraints of the estimated value calculation.

[0103] After extracting the second singular value, a matrix may be further established according to the second singular value to solve the estimated value of the basic matrix. In some embodiments, the method further includes:

[0104] constructing a third matrix based on the second singular values ​​and the singular matrix;

[0105] According to a matrix null space algorithm, a left null space solution set and a right null space solution set of the third matrix are calculated.

[0106] Select the two largest singular values ​​from the four singular values ​​as the second singular values, and construct a 4×4 diagonal matrix Σ′2 with the two second singular values ​​and two zero elements. Based on the diagonal matrix Σ′2, the third matrix can be obtained It can be understood that the rank of the third matrix is ​​2, so the vectors it contains also have two free dimensions.

[0107] In view of the fact that the binocular camera has two types of lenses, the lenses correspond to two different null spaces, which are represented by left null space and right null space in the embodiment of the present application. Satisfy the equation The set of all vectors v, right null space Satisfy the equation The set of all vectors v. It can be understood that the vector v has two free dimensions, correspondingly, the left null space The endpoint trajectories of all vectors in form a straight line. Similarly, the right null space It also corresponds to a straight line in space.

[0108] It is understandable that the left null space Right Null Space It can be collectively called the null space Null Space It can be used to determine a pair of four-dimensional vectors (q,u), and the four-dimensional vector (q,u) can be used to represent the null space Where q = (q0, q1, q2, 1) represents a vector in the null space (can be any vector), and u = (u0, u1, u2, 1) represents the direction vector of the line. Based on the vector representation of the null space, the left null space, the right null space and the imaging surface can be calculated. In some embodiments, the method further comprises:

[0109] An imaging surface coordinate system is established on the imaging plane of the binocular camera to be calibrated; the imaging surface coordinate system includes a first imaging surface coordinate system corresponding to the left null space solution set and a second imaging surface coordinate system corresponding to the right null space solution set;

[0110] A first intersection point of the left null space solution set and a first imaging plane coordinate system is calculated, and a second intersection point of the right null space solution set and a second imaging plane coordinate system is calculated.

[0111] Left Null Space The corresponding imaging surface can be expressed as Its intersection is Right Null Space The corresponding imaging surface can be expressed as Its intersection is According to the definition of pole, is the right pole of the fundamental matrix, is the left pole of the basic matrix. The left pole and the right pole can be used to obtain the radial distortion parameters in the basic matrix.

[0112] In some embodiments, after calculating the left pole and the right pole, the method further includes:

[0113] Determine a first optical axis on the first imaging plane coordinate system, and determine a second optical axis on the second imaging plane coordinate system;

[0114] Calculate a first shortest distance point on the first optical axis to the left null space solution set, and calculate a second shortest distance point on the second optical axis to the right null space solution set;

[0115] A first radial distortion parameter is calculated according to the first shortest distance point and the radial distortion vector, and a second radial distortion parameter is calculated according to the second shortest distance point and the radial distortion vector.

[0116] The imaging planes corresponding to the left null space and the right null space can respectively determine the optical axis and the straight line where the optical axis is located. Combined with the radial distortion vector obtained in the above content, the radial distortion parameters of the binocular camera to be calibrated can be obtained.

[0117] like Figure 4As shown, taking the left null space as an example, the calculation method of the right pole of the basic matrix is ​​explained: According to the position relationship equation between straight lines in space, the optical axis of the imaging surface can be calculated On the straight line, Nearest coordinates:

[0118]

[0119] Combination and the radial distortion vector The radial distortion parameters can be obtained:

[0120]

[0121] Similarly, another radial distortion parameter can be calculated, which will not be described in detail here. It should be noted that the two radial distortion parameters mentioned in this embodiment are hereinafter referred to as the first radial distortion parameter λ L and the second radial distortion parameter λ R describe.

[0122] After the first radial distortion parameter and the second radial distortion parameter are obtained by calculation, the radial distortion parameter needs to be substituted into the distortion basic matrix. Therefore, it is also necessary to combine the basic matrix F under ideal conditions (where the lens does not have radial distortion) u ,like Figure 5 As shown, in some embodiments, the step of calculating the estimated value of the distortion base matrix further includes:

[0123] Select the first undetermined coefficient and the second undetermined coefficient to construct a fourth matrix, wherein the fourth matrix includes the first undetermined parameter and the second undetermined parameter;

[0124] Calculating a specific value of the first undetermined parameter according to the coordinates of the first intersection point and the coordinates of the second intersection point;

[0125] Based on the first radial distortion parameter, the second radial distortion parameter and the first undetermined parameter, a linear equation group for constructing a second equation is obtained;

[0126] According to the linear equation group, a second equation consisting of a fifth matrix and a second vector is constructed; the fifth matrix includes known quantities; the second vector includes unknown quantities, and the second vector includes a second unknown parameter.

[0127] Fundamental matrix F u Contains 9 elements, and it is known that So we can parameterize the fundamental matrix F u The estimated value of the parameterized distortion matrix is Then we can get the specific estimated value. R To RSubstituting into the defined matrix H, we get the matrix, To L Substitute the matrix obtained by substituting it into the defined matrix H.

[0128] For example, the matrix F u One form of expression is a fourth matrix, which is constructed by introducing the first undetermined coefficients (α, β, α′, β′) and the second undetermined coefficients (a, b, c, d). The fourth matrix can be expressed as:

[0129]

[0130] According to the expression of the fourth matrix, we can get equation F u A special solution for v = 0 is ξ1 = (α, β, -1) T According to the definition of the basic matrix, the vector ξ1 is the matrix F u The right pole of The values ​​of α and β can be known. Similarly, equation v T ·F u = 0 is ξ2 = (α′, β′, -1) T , the vector ξ2 is the matrix F u The left pole of Through the above calculation method, the specific value of the first undetermined coefficient in the fourth matrix can be obtained.

[0131] On the basis of obtaining the specific values ​​of the first radial distortion parameter, the second radial distortion parameter, and the first undetermined coefficient, the fourth matrix and the radial distortion parameter can be substituted into the matrix expression corresponding to the estimated value of the distortion base matrix to obtain the expression of the estimated value of the distortion base matrix including the undetermined coefficient and the radial distortion parameter. Among them, the first undetermined coefficient is a known coefficient, and the second undetermined coefficient is an unknown coefficient. Furthermore, the estimated value expression of the distortion base matrix including the undetermined coefficient and the coordinate values ​​of the same name points can be substituted into In the equation, a set of linear equations is obtained. The linear equations can be arranged in the form of C·f2=0, that is, the second equation. Wherein, C is the fifth matrix, and f2 is the second vector, which is composed of the unknown second undetermined coefficients.

[0132] like Figure 6 As shown, the singular value decomposition is continued to be performed on the fifth matrix, and the specific value for obtaining the second undetermined coefficient in the second vector can be obtained through the decomposition result, and the steps further include:

[0133] performing singular value decomposition on the fifth matrix to extract fifth singular values;

[0134] Based on the fifth matrix and the fifth singular value, calculating the eigenvector of the fifth matrix to determine the specific value of the second undetermined parameter;

[0135] An estimated value of the distortion basic matrix is ​​determined based on the first radial distortion parameter, the second radial distortion parameter, a specific value of the first to-be-determined parameter, and a specific value of the second to-be-determined parameter.

[0136] Performing singular value decomposition on the fifth matrix can obtain a 4x4 square matrix W3 and a 4x4 diagonal matrix Σ3. Among them, Σ3 contains 4 singular values ​​corresponding to the matrix C, and the smallest singular value is selected as the fifth singular value. Based on the fifth singular value, its corresponding singular vector f3 can be calculated. The elements contained in the singular vector f3 are compared with the elements contained in the second vector f2 to obtain the specific value of the second unknown coefficient (a, b, c, d).

[0137] After obtaining the specific value of the second unknown coefficient, it is used to parameterize the estimated value of the distortion basic matrix Parameters: First radial distortion parameter λ L , the second radial distortion parameter λ R , the first unknown coefficients (α, β, α′, β′), and the second unknown coefficients (a, b, c, d) are all known, so substitute the known parameters into F u The expression of F u The parameterized specific matrix is ​​then substituted into That is, the estimated value of the distortion basic matrix can be obtained

[0138] S400: constructing an objective function for calculating an optimal distortion fundamental matrix according to the first equation and the estimated value of the distortion fundamental matrix;

[0139] According to the first equation and the estimated value of the distortion basic matrix, constructing an objective function for calculating an optimal distortion basic matrix, including:

[0140] Using the binocular camera to be calibrated to collect a second binocular image, and extracting multiple pairs of coordinates of points of the same name from the second binocular image;

[0141] Based on multiple pairs of coordinates of the same-name points extracted from the second binocular image, a first matrix in the objective function is constructed to obtain the objective function; an initial value of the distortion basic matrix in the objective function is an estimated value of the distortion basic matrix.

[0142] According to the first equation and the basic matrix, the reprojection error function can be defined as: h(F) = F T A1 T ·A1·F.

[0143] It should be noted that the second binocular image and the first binocular image are different images collected twice, and the multiple pairs of same-name point coordinates extracted therefrom are also different. The first binocular image is used to construct the first equation for calculating the distortion basic matrix estimate, and the second binocular image is used to construct the matrix in the reprojection error function. The number of the second binocular images is at least one pair, and the number of same-name point coordinates extracted from the second binocular image is at least 15 pairs.

[0144] After extracting enough coordinates of the same-name points from the second binocular image, the matrix A1 can be constructed, and the reprojection error function can be formed with the estimated value of the basic matrix. It can be understood that the initial value of the reprojection error function in the embodiment of the present application is the estimated value of the basic matrix, so the basic matrix can be optimized based on the estimated value as the initial value to obtain the optimal basic matrix.

[0145] S500: Optimizing the objective function using a least squares optimization algorithm to obtain an optimal distortion basic matrix.

[0146] In some embodiments, the least squares method is used to optimize the objective function, i.e., the reprojection error function. During the optimization process, the undetermined parameters contained in the distortion basic matrix are iterated to achieve the optimization of the distortion basic matrix. So that the reprojection error value is minimized in the sense of the A1 matrix. The part of using the least squares method to optimize the reprojection error function is not the focus of the description of this application, so it will not be repeated. The embodiment of the present application aims to complete the calibration of the radial distortion parameters of the lens of the binocular camera to be calibrated and the calibration of the basic matrix in one calibration process. The basic matrix is ​​parameterized, and the radial distortion parameters are used as one of the parameters of the distortion basic matrix. On the basis of taking the estimated value of the distortion basic matrix as the optimization initial value, the least squares optimization algorithm is used for iterative optimization to calibrate the radial distortion parameters and the basic matrix at the same time, thereby improving the calibration efficiency.

[0147] Some embodiments of the present application also provide a binocular camera calibration device, including: a preprocessing module, a collection module, a calculation module and an optimization module;

[0148] The preprocessing module is used to construct a distortion basic matrix equation according to the lens distortion information of the binocular camera to be calibrated and the basic matrix;

[0149] The acquisition module is used to use the binocular camera to be calibrated to acquire a first binocular image, and to extract multiple pairs of coordinates of points of the same name from the first binocular image;

[0150] The operation module constructs a first equation based on the distortion basic matrix equation and a plurality of pairs of coordinates of points of the same name to calculate an estimated value of the distortion basic matrix;

[0151] The operation module is further used to construct an objective function for calculating an optimal distortion basic matrix according to the first equation and the estimated value of the distortion basic matrix;

[0152] The optimization module is used to optimize the objective function using a least squares optimization algorithm to obtain an optimal basic matrix.

[0153] An embodiment of the present application also provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to perform the binocular camera calibration method as described in the above technical content.

[0154] It can be pointed out from the above technical content that the present application provides a binocular camera calibration method, device and electronic device. The method constructs a distortion basic matrix equation as a calibration basis through the lens distortion information and basic matrix of the binocular camera to be calibrated. The first binocular image is collected using the binocular camera to be calibrated, and the coordinates of the same-name points are extracted to construct a first equation with the distortion basic matrix equation. The estimated value of the distortion basic matrix used to calibrate the binocular camera to be calibrated is calculated by the first equation to construct an objective function for calculating the optimal distortion basic matrix. The objective function is optimized using the least squares optimization algorithm to obtain the optimal distortion basic matrix. The method is divided into two stages of preliminary estimation and optimization in the process of obtaining the optimal distortion basic matrix, which alleviates the problem that it is difficult to obtain results for nonlinear optimization methods. At the same time, the description of radial distortion parameters is added in the process of obtaining the optimal distortion basic matrix, which improves the calibration efficiency and accuracy of the binocular camera to be calibrated.

[0155] Similar parts between the embodiments provided in this application can be referenced to each other. The specific implementation methods provided above are only a few examples under the general concept of this application and do not constitute a limitation on the protection scope of this application. For those skilled in the art, any other implementation methods expanded based on the scheme of this application without creative work belong to the protection scope of this application.

Claims

1. A binocular camera calibration method, characterized in that: include: Construct a distortion basic matrix according to the lens distortion information and basic matrix of the binocular camera to be calibrated; The basic matrix is ​​a matrix used to describe the absence of radial distortion phenomenon, and the distortion basic matrix is ​​a matrix used to describe the presence of radial distortion phenomenon; Using the binocular camera to be calibrated to capture a first binocular image, and extracting multiple pairs of coordinates of points of the same name from the first binocular image; Constructing a first equation based on the distortion basic matrix and a plurality of pairs of coordinates of points of the same name to calculate an estimated value of the distortion basic matrix; Using the binocular camera to be calibrated to collect a second binocular image, and extracting multiple pairs of coordinates of points of the same name from the second binocular image; Constructing a first matrix in the objective function based on multiple pairs of coordinates of points of the same name extracted from the second binocular image; An objective function is constructed according to the distortion basic matrix and a first matrix constructed based on multiple pairs of coordinates of the same-name points extracted from the second binocular image; the objective function is used to calculate the optimal distortion basic matrix; the initial value of the distortion basic matrix in the objective function is an estimated value of the distortion basic matrix; and the objective function is optimized using a least squares optimization algorithm to obtain the optimal distortion basic matrix.

2. The method according to claim 1, characterized in that Constructing a first equation based on the distortion basic matrix and multiple pairs of coordinates of points with the same name includes: Constructing a first matrix according to the multiple pairs of coordinates of the same-name points, and constructing a first vector according to undetermined parameters in the distortion basic matrix used to characterize lens distortion information and the basic matrix; A first equation is obtained based on the first matrix and the first vector.

3. The method according to claim 2, characterized in that Constructing a first equation based on the distortion basic matrix and multiple pairs of coordinates of points of the same name to calculate an estimated value of the distortion basic matrix, further comprising: Performing singular value decomposition on the first matrix, and extracting a first singular value from candidate singular values ​​obtained by the singular value decomposition; A first singular vector corresponding to the first singular value is calculated.

4. The method according to claim 3, characterized in that Also includes: constructing a second matrix based on the first singular vectors; Singular value decomposition is performed on the second matrix, and second singular values ​​are extracted from candidate singular values ​​obtained by the singular value decomposition.

5. The method according to claim 4, characterized in that Also includes: constructing a third matrix based on the second singular values; According to a matrix null space algorithm, a left null space solution set and a right null space solution set of the third matrix are calculated.

6. The method according to claim 5, characterized in that Also includes: An imaging surface coordinate system is established on the imaging plane of the binocular camera to be calibrated; the imaging surface coordinate system includes a first imaging surface coordinate system corresponding to the left null space solution set and a second imaging surface coordinate system corresponding to the right null space solution set; A first intersection point of the left null space solution set and a first imaging plane coordinate system is calculated, and a second intersection point of the right null space solution set and a second imaging plane coordinate system is calculated.

7. The method according to claim 6, characterized in that Also includes: Determine a first optical axis on the first imaging plane coordinate system, and determine a second optical axis on the second imaging plane coordinate system; Calculate a first shortest distance point on the first optical axis to the left null space solution set, and calculate a second shortest distance point on the second optical axis to the right null space solution set; A first radial distortion parameter is calculated according to the first shortest distance point and the radial distortion vector, and a second radial distortion parameter is calculated according to the second shortest distance point and the radial distortion vector.

8. The method according to claim 7, characterized in that Also includes: Select the first undetermined coefficient and the second undetermined coefficient to construct a fourth matrix; Calculating a specific value of the first undetermined coefficient according to the coordinates of the first intersection point and the coordinates of the second intersection point; Based on the specific values ​​of the first radial distortion parameter, the second radial distortion parameter and the first undetermined coefficient, a linear equation group for constructing a second equation is obtained; According to the linear equation group, a second equation consisting of a fifth matrix and a second vector is constructed; the fifth matrix includes known quantities; the second vector includes unknown quantities, and the second vector includes a second unknown coefficient.

9. The method according to claim 8, characterized in that Also includes: performing singular value decomposition on the fifth matrix to extract fifth singular values; Based on the fifth matrix and the fifth singular value, calculating the eigenvector of the fifth matrix to determine the specific value of the second undetermined coefficient; An estimated value of the distortion basic matrix is ​​determined based on the first radial distortion parameter, the second radial distortion parameter, a specific value of the first to-be-determined coefficient, and a specific value of the second to-be-determined coefficient.

10. A binocular camera calibration device, characterized in that: include: Preprocessing module, acquisition module, calculation module and optimization module; The preprocessing module is used to determine the distortion basic matrix according to the lens distortion information of the binocular camera to be calibrated and the basic matrix; the basic matrix is ​​a matrix used to describe the absence of radial distortion phenomenon, and the distortion basic matrix is ​​a matrix used to describe the presence of radial distortion phenomenon; The acquisition module is used to use the binocular camera to be calibrated to acquire a first binocular image, and to extract multiple pairs of coordinates of points of the same name from the first binocular image; The acquisition module is also used to use the binocular camera to be calibrated to acquire a second binocular image, and to extract multiple pairs of coordinates of points of the same name from the second binocular image; The operation module constructs a first equation based on the distortion basic matrix and a plurality of pairs of coordinates of points of the same name to calculate an estimated value of the distortion basic matrix; The operation module is also used to construct a first matrix in the objective function based on multiple pairs of coordinates of points of the same name extracted from the second binocular image; The operation module is also used to construct an objective function based on the distortion basic matrix and a first matrix constructed based on multiple pairs of coordinates of the same-name points extracted from the second binocular image; the objective function is used to calculate the optimal distortion basic matrix; the initial value of the distortion basic matrix in the objective function is an estimated value of the distortion basic matrix; The optimization module is used to optimize the objective function using a least squares optimization algorithm to obtain an optimal distortion basic matrix.

11. An electronic device, characterized in that: The method comprises a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to execute the binocular camera calibration method according to any one of claims 1 to 9.

Images