Method and apparatus for determining camera relative pose parameters based on binocular vision

Abstract

This application relates to a method, apparatus, terminal device, and storage medium for determining camera relative pose parameter values ​​based on binocular vision. The method includes: obtaining a first relation by taking the partial derivative of each matrix element in a first camera extrinsic parameter matrix according to the collinearity equation; obtaining a second relation by taking the partial derivative of each relative pose parameter according to the relative pose parameter matrix and determining its product with a preset second camera extrinsic parameter matrix; determining a partial derivative function relation for the collinearity equation with respect to the relative pose parameters based on the product of the first and second relational expressions; obtaining a pixel deviation relation between the true image coordinates and the collinearity equation based on the product of the increment of the relative pose parameters and the partial derivative function relation; iterating over the pixel deviation relation, and obtaining the relative pose parameter values ​​of the first camera when the iteration is complete. This method can reduce computational complexity and improve camera calibration efficiency.

Description

Technical Field

[0001] This application relates to the field of computer technology, and in particular to a method, apparatus, terminal device, and computer-readable storage medium for determining camera relative pose parameter values ​​based on binocular vision. Background Technology

[0002] The calibration of a binocular system mainly consists of three modules: single-camera calibration, binocular calibration, and epipolar correction. Among them, binocular calibration is mainly based on the calibration model, and obtains a relatively accurate relative pose between the two cameras, including distance and relative attitude, by optimizing the global reprojection residual.

[0003] Generally, single-camera calibration and binocular calibration are performed independently. First, each camera in the binocular system is calibrated. After obtaining the intrinsic and extrinsic parameters of the single-camera calibration, the relative pose of the binocular cameras is then calibrated. However, many papers and researchers optimize both single-camera and binocular systems together when considering adjustment or global residual optimization. This significantly increases computation time, and the final result is not significantly improved compared to separate calibration. Most people perform separate calibrations.

[0004] In dual-target calibration based on adjustment models, most traditional methods involve substituting the relative pose as the required variable into the adjustment model to obtain the final value. Because this involves multiple iterations of the model, the entire calculation process is cumbersome, resulting in low camera calibration efficiency. Summary of the Invention

[0005] Therefore, it is necessary to provide a method, apparatus, terminal device, and computer-readable storage medium for determining the relative pose parameters of a camera based on binocular vision, which can reduce computational complexity and improve camera calibration efficiency, in order to address the aforementioned technical problems.

[0006] A method for determining camera relative pose parameters based on binocular vision, the method comprising:

[0007] The first relation is obtained by taking the partial derivative of each matrix element in the first camera extrinsic matrix with respect to the collinearity equation;

[0008] The second relation is obtained by taking the partial derivative of each relative pose parameter with respect to the relative pose parameter matrix and determining the product with the preset second camera extrinsic parameter matrix.

[0009] Based on the product of the first relation and the second relation, determine the partial derivative function relation of the collinear equation with respect to the relative pose parameters;

[0010] The pixel deviation relationship between the true image coordinates and the collinearity equation is obtained by multiplying the increment of the relative pose parameter with the partial derivative function relationship.

[0011] The pixel deviation relationship is iterated, and when the iteration is completed, the relative pose parameter values ​​of the first camera are obtained.

[0012] A device for determining camera relative pose parameters based on binocular vision, the device comprising:

[0013] The first partial derivative module is used to calculate the partial derivative of each matrix element in the first camera extrinsic matrix based on the collinearity equation to obtain the first relational expression.

[0014] The second partial derivative module is used to calculate the partial derivative of each relative pose parameter based on the relative pose parameter matrix, and determine the product with the preset second camera extrinsic parameter matrix to obtain the second relational expression;

[0015] The third partial derivative module is used to determine the partial derivative function relationship of the collinear equation with respect to the relative pose parameters based on the product of the first relation and the second relation.

[0016] The pixel deviation relationship acquisition module is used to obtain the pixel deviation relationship between the true coordinates of the image and the collinearity equation based on the product of the increment of the relative pose parameter and the partial derivative function relationship.

[0017] The iteration module is used to iterate the pixel deviation relationship and obtain the relative pose parameter values ​​of the first camera when the iteration is completed.

[0018] A terminal device includes a memory and a processor. The memory stores a computer program, and the processor executes the computer program to implement the steps of various embodiments of the method for determining the relative pose parameter values ​​of a camera based on binocular vision.

[0019] A computer-readable storage medium having a computer program stored thereon, wherein the computer program, when executed by a processor, implements the steps of various embodiments of the method for determining camera relative pose parameter values ​​based on binocular vision.

[0020] The aforementioned method, apparatus, terminal device, and computer-readable storage medium for determining camera relative pose parameters based on binocular vision, after analysis, revealed that the coefficients in the pixel deviation relationship can be decomposed into a first relationship and a second relationship. Furthermore, the collinearity equation contains each matrix element of the first camera extrinsic parameter matrix, and the relative pose parameter matrix contains the relative pose parameters. By using matrix form to solve the problem, matrix partial differentials are used to replace the cumbersome calculation of partial derivatives of variables in the collinearity equation. The model is simple in form, very convenient to program, and greatly reduces the computational complexity. It can quickly complete the calculation of relative pose parameter values, and the results of binocular vision calibration are consistent with the calibration results of traditional methods. Attached Figure Description

[0021] Figure 1 This is an application environment diagram of a method for determining camera relative pose parameters based on binocular vision in one embodiment;

[0022] Figure 2 This is a flowchart illustrating a method for determining camera relative pose parameter values ​​based on binocular vision in one embodiment.

[0023] Figure 3 This is a flowchart of a camera relative pose parameter value determination device based on binocular vision in one embodiment;

[0024] Figure 4 This is an internal structure diagram of a terminal device in one embodiment. Detailed Implementation

[0025] It should be understood that the specific embodiments described herein are merely illustrative of this application and are not intended to limit this application.

[0026] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of this application, and not all of the embodiments. Based on the embodiments of this application, all other embodiments obtained by those of ordinary skill in the art without creative effort are within the scope of protection of this application.

[0027] It should be noted that all directional indicators (such as up, down, left, right, front, back, etc.) in the embodiments of this application are only used to explain the relative positional relationship and movement of each component in a certain specific posture (as shown in the figure). If the specific posture changes, the directional indicator will also change accordingly. The connection can be a direct connection or an indirect connection.

[0028] Furthermore, the use of terms such as "first" and "second" in this application is for descriptive purposes only and should not be construed as indicating or implying their relative importance or implicitly specifying the number of technical features indicated. Therefore, a feature defined as "first" or "second" may explicitly or implicitly include at least one of those features. Additionally, the technical solutions of the various embodiments can be combined with each other, but only on the basis of being achievable by those skilled in the art. If the combination of technical solutions is contradictory or impossible to implement, such a combination of technical solutions should be considered non-existent and not within the scope of protection claimed in this application.

[0029] The terms "first," "second," etc., used in this application may be used herein to describe various elements, but these elements are not limited by these terms. These terms are only used to distinguish one element from another. For example, without departing from the scope of this application, a first relation may be referred to as a second relation, and similarly, a second relation may be referred to as a first relation. Both the first relation and the second relation are relations, but they are not the same relation.

[0030] It is understood that the term "connection" in the following embodiments should be understood as "electrical connection," "communication connection," etc., if the connected circuits, modules, units, etc., have electrical signal or data transmission with each other.

[0031] The camera parameter determination method provided in this application can be applied to, for example... Figure 1 In the application environment. Figure 1 This is an application environment diagram of a camera parameter determination method in one embodiment. It includes a first camera 110, a second camera 120, and a terminal device 130. If the first camera 110 is a left camera, then the second camera 120 is a right camera; similarly, if the first camera 110 is a right camera, then the second camera 120 is a left camera. In this application, embodiments are described using the first camera 110 as the right camera and the second camera 120 as the left camera as an example. The terminal device 130 can be, but is not limited to, various personal computers, laptops, smartphones, tablets, and portable wearable devices.

[0032] In one embodiment, the study found that the purpose of the entire bi-target calibration is to serve the subsequent epipolar correction. During epipolar correction, most researchers prefer to use libraries like OpenCV and often choose the Zhang Zhengyou calibration method. However, the Zhang Zhengyou calibration method involves large rotations and translations of both cameras during epipolar correction, thus requiring high precision in the bi-target calibration results. However, in this embodiment, it is noted that the principle of epipolar correction is to transform and adjust the distorted original image to the same epipolar line, forming a distortion-free corrected image where each corresponding point lies on an epipolar line. Therefore, the rotation and transformation of the camera plane can be selected based on the model, requiring only subsequent image movement and cropping. Based on this, if the left camera plane remains almost unchanged while the right camera plane is transformed into the left camera coordinate system, the extrinsic parameter error of the left camera has almost zero impact on epipolar correction. Therefore, to simplify the bi-target calibration model and accelerate computation, in each embodiment of this application, the intrinsic and extrinsic parameters of the left camera are fixed, and the relative pose of the two cameras and the extrinsic parameters of the right camera are processed in the adjustment model. However, it's important to note that the extrinsic parameters of the right camera can be replaced by the extrinsic parameters of the left camera and the relative pose of the cameras. Therefore, the entire model only needs to process the relative poses between the cameras, significantly reducing computational complexity. Furthermore, since epipolar correction is primarily based on the left camera, the final error is very small, comparable to the results of all iterations. Understandably, depending on specific needs, it's also possible to prioritize the right camera and map the left camera plane into the right camera coordinate system.

[0033] like Figure 2 The diagram shown is a flowchart of a camera calibration method in one embodiment, including:

[0034] Step 202: Take the partial derivative of each matrix element in the first camera extrinsic matrix with respect to the collinearity equation to obtain the first relational expression.

[0035] The collinearity equation expresses the mathematical relationship that the object point, image point, and projection center lie on the same straight line. For single-camera calibration, the bundle adjustment model studies the collinearity equation, and the parameters of each camera must satisfy the following collinearity equation:

[0036]

[0037]

[0038] Where Δx and Δy are distortions, (x, y) are the pixel coordinates of the image, and (X, Y, Z) are the true coordinates of the marker point on the calibration plate. x0, y0 are the pixel coordinates of the principal point projected onto the image plane by the camera's optical center, and f is the camera's focal length. The extrinsic parameters a, b, and c are determined by the camera's attitude angles. and camera position X s ,Ys Z s It's a decision. Understandably, distortion can be added to the collinearity equations, or it can be left out.

[0039] a, b, and c mentioned above, as well as the attitude angle and camera position X s ,Y s Z s All are unknowns. The pixel coordinates of the image can be obtained by reading the image, and the actual coordinates of the calibration points on the calibration plate can also be obtained through other calibration methods.

[0040] The extrinsic parameter matrix of the first camera is determined by the extrinsic parameters of the first camera. Specifically, the R corresponding to the extrinsic parameter matrix of the first camera is... r for

[0041]

[0042] Where r represents the first camera. Actually, a in Rr... r1 This is equivalent to a1 in the collinearity equation… and so on, which will not be elaborated here. Similarly, all elements in the extrinsic parameter matrix of the first camera are also unknowns.

[0043] Since the collinear equations contain some matrix elements from the first camera extrinsic matrix, it is relatively easy and fast to find the partial derivatives.

[0044] Step 204: Calculate the partial derivatives of each relative pose parameter based on the relative pose parameter matrix, and determine the product with the preset second camera extrinsic parameter matrix to obtain the second relational expression.

[0045] Specifically, the relative pose parameter matrix is ​​based on the relative pose parameters. The relative pose parameter matrix Rc is determined to be...

[0046]

[0047] The preset second camera extrinsic matrix refers to the known camera extrinsic matrix pre-stored in the terminal device, as follows:

[0048]

[0049] R l The values ​​of parameters a, b, and c in the equation are all known.

[0050] The second relational expression represents the extrinsic parameter matrix R of the first camera. r The expression for the partial derivative of each matrix element with respect to the relative pose parameters, where the first camera extrinsic parameter matrix R... r Unknown. Then there is.

[0051]

[0052] Step 206: Based on the product of the first relation and the second relation, determine the partial derivative function relation of the collinear equation with respect to the relative pose parameters.

[0053] Specifically, the expression of the collinearity equation is changed to

[0054]

[0055]

[0056] The partial derivatives of the collinear equations with respect to the relative pose parameters are expressed as follows:

[0057]

[0058] Among them, the partial derivative function relationship of the collinear equation with respect to the relative pose parameters can be decomposed based on the chain rule into the partial derivative of the collinear equation with respect to the first camera extrinsic matrix, and the partial derivative of the first camera extrinsic matrix with respect to the relative pose parameters.

[0059] Step 208: Based on the product of the increment of the relative pose parameters and the partial derivative function, obtain the pixel deviation relationship between the true image coordinates and the collinearity equation.

[0060] The pixel deviation formula can refer to either the pixel deviation formula in the x-direction or the pixel deviation formula in the y-direction. The calculation method for both is the same.

[0061] Specifically, based on the first-order Taylor expansion, partial derivatives with respect to each extrinsic parameter in the collinear equations can be obtained as follows:

[0062]

[0063]

[0064] in, Δξ represents the extrinsic parameters of the first camera. r This refers to the increment of the first camera's extrinsic parameters. Furthermore, the relative pose parameter matrix has the following relationship with the first camera's extrinsic parameter matrix and the unknown second camera's extrinsic parameter matrix: R... c =R l R' r

[0065] Therefore, the partial derivative of the above relationship with respect to the extrinsic parameters of the first camera can be transformed into taking the partial derivative with respect to the relative pose parameters between the two cameras, and can be rewritten as follows:

[0066]

[0067]

[0068] The left side of the equation represents the pixel deviation between the true x-coordinate of the image and the collinearity equation F, and the pixel deviation between the true y-coordinate of the image and the collinearity equation G.

[0069] Step 210: Iterate the pixel deviation relationship. When the iteration is complete, obtain the relative pose parameter values ​​of the first camera.

[0070] Specifically, the terminal device iterates the relative pose parameters in the pixel deviation relationship. When the iteration reaches a preset number of times or when the error of the pixel deviation relationship is less than or equal to a preset error, the relative pose parameter value of the first camera is obtained.

[0071] In this embodiment, analysis revealed that the coefficients in the pixel deviation relationship can be decomposed into a first relationship and a second relationship. Furthermore, the collinearity equation contains each matrix element of the first camera extrinsic parameter matrix, and the relative pose parameter matrix contains the relative pose parameters. By using matrix form to solve the problem, matrix partial differentials are used to replace the cumbersome calculation of partial derivatives of variables in the collinearity equation. The model is simple in form and very convenient to program, greatly reducing the computational complexity. It can quickly complete the calculation of the relative pose parameter values, and the results of binocular vision calibration are consistent with the calibration results of traditional methods.

[0072] In one embodiment, the pixel offset relationship is iterated, and when the iteration is complete, the relative pose parameter values ​​of the first camera are obtained, including:

[0073] Adjust the values ​​of the relative pose parameters in the pixel deviation formula. When the pixel deviation formula meets the error condition, obtain the values ​​of each relative pose parameter of the first camera.

[0074] Among them, satisfying the error condition can be that the value of the pixel deviation relationship is less than or equal to the preset error, etc.

[0075] Specifically, the terminal device adjusts the values ​​of the relative pose parameters in the pixel deviation relationship. When the value of the pixel deviation relationship is less than or equal to the preset error, or when the square value of the pixel deviation relationship is less than or equal to the preset error, the adjusted values ​​at this time are used as the relative pose parameter values ​​of the first camera.

[0076] In this embodiment, for the pixel deviation relationship, only the following problem needs to be optimized.

[0077]

[0078] The above formula uses the square of the pixel deviation relationship as an example for illustration, and the same applies to the y-coordinate values ​​of the image. It can be understood that the x and y coordinates can also be iterated independently using the pixel deviation relationship.

[0079] In this embodiment, the values ​​of the relative pose parameters in the pixel deviation relationship are adjusted. When the pixel deviation relationship satisfies the error condition, the values ​​of each relative pose parameter of the first camera are obtained. Only one model needs to be iterated and optimized, which can greatly reduce the computational complexity, improve the efficiency of determining the relative pose parameter values ​​of the camera, and thus improve the calibration efficiency.

[0080] In one embodiment, the method further includes: substituting the relative pose parameter values ​​into the relative pose parameter matrix to obtain the target relative pose matrix; and obtaining the target first camera extrinsic matrix based on the product of the transpose of the target relative pose matrix and a preset second camera extrinsic matrix.

[0081] Specifically, the values ​​in the target's first camera extrinsic parameter matrix are all known. The first camera extrinsic parameter matrix and the relative pose matrix can be converted to each other, i.e.:

[0082] R2=R′ c R1

[0083] Then, by substituting the relative pose parameter values ​​into the relative pose parameter matrix, the target relative pose matrix is ​​obtained. Based on the product of the transpose of the target relative pose matrix and the preset second camera extrinsic matrix, the target first camera extrinsic matrix is ​​obtained.

[0084] In this embodiment, the relative pose parameter values ​​are substituted into the relative pose parameter matrix to obtain the target relative pose matrix; the target first camera extrinsic matrix is ​​obtained by multiplying the transpose of the target relative pose matrix with the preset second camera extrinsic matrix; the extrinsic matrices of the first camera and the second camera are obtained; the relative pose matrix between the first camera and the second camera is obtained; and all calibration parameters of the stereo camera are obtained by combining the known intrinsic parameters of the first camera and the second camera.

[0085] In one embodiment, the relative pose parameter matrix contains relative pose parameters; the collinearity equation contains matrix elements of the first camera extrinsic parameter matrix; the partial derivative function relationship is determined based on the product of two partial derivative functions decomposed by the chain rule, one of which is the partial derivative of the collinearity equation with respect to the first camera extrinsic parameter matrix, and the other is the partial derivative of the first camera extrinsic parameter matrix with respect to the relative pose parameters.

[0086] Specifically, directly calculating the partial derivatives of the collinearity equations with respect to the relative pose parameters is computationally complex. However, analysis reveals that decomposing the equations using the chain rule yields two partial derivative functions, which are very simple to calculate, allowing for a rapid acquisition of the relative pose parameter values.

[0087] In one embodiment, in dual-target calibration based on adjustment models, many papers adopt the approach of fixing the intrinsic parameters of the cameras, using the extrinsic parameters of the two cameras and the relative pose between the cameras as required variables, and substituting all of them into the adjustment model for iteration. Finally, a set of variables is calculated as the final value by optimizing the minimum residual. Because it involves three iterative models that are optimized together, the whole process is quite cumbersome.

[0088] In this application, the study found that the purpose of the entire bi-target calibration is to serve the subsequent epipolar correction. During epipolar correction, most researchers prefer to use libraries like OpenCV and often choose the Zhang Zhengyou calibration method. However, the Zhang Zhengyou calibration method involves large rotations and translations of both cameras during epipolar correction, thus requiring high precision in the bi-target calibration results. This paper notes that the principle of epipolar correction is to transform and adjust the distorted original image to the same epipolar line, forming a distortion-free corrected image where each corresponding point lies on an epipolar line. Therefore, the rotation and transformation of the camera plane can be chosen based on the model, requiring only subsequent image movement and cropping. Based on this, if the left camera plane remains almost unchanged while the right camera plane is transformed into the left camera coordinate system, the extrinsic parameter error of the left camera has almost zero impact on epipolar correction. Therefore, to simplify the bi-target calibration model and accelerate computation, this paper fixes the intrinsic and extrinsic parameters of the left camera and processes the relative pose of the two cameras and the extrinsic parameters of the right camera in the adjustment model. However, it's important to note that the extrinsic parameters of the right camera can be replaced by the extrinsic parameters of the left camera and the relative pose of the cameras. Therefore, the entire model only needs to process the relative pose between the cameras, significantly reducing computational complexity. Furthermore, since epipolar correction primarily focuses on the left camera, the final error is very small, comparable to the results of all iterations.

[0089] The following detailed discussion begins with the collinearity equations of the cameras and is further elaborated using the two-dimensional DLT algorithm.

[0090] 1. Global minimum reprojection residual optimization with all parameters

[0091] For single-camera calibration, the bundle adjustment model studies the collinearity equations, and the parameters of each camera must satisfy the following model:

[0092]

[0093]

[0094] Where Δx and Δy are distortions, (x,y) are the pixel coordinates of the image, and (X,Y,Z) are the true coordinates of the marker point on the calibration board. The selected distortion model is as follows:

[0095] Δx=(x-x0)(K1r2 +K2r 4 )+P1[r 2 +2(x-x0) 2 ]+2P2(x-x0)(y-y0)+P3(x-x0)+P4(y-y0)

[0096] Δy=(y-y0)(K1r 2 +K2r 4 )+P2[r 2 +2(y-y0) 2 ]+2P1(x-x0)(y-y0) (2)

[0097] K1, K2, P1, P2, P3, P4 are the camera distortion coefficients, x0, y0 are the pixel coordinates of the principal point of the camera's optical center projected onto the image plane, and f is the camera's focal length. These nine parameters are the intrinsic parameters of the camera model used in this paper. According to model (1), the extrinsic parameters correspond to a, b, c, which are determined by the camera's attitude angles. With Camera X s ,Y s Z s It was decided.

[0098] For a fully parameter-optimized bi-objective positioning model, the goal is to ensure that both cameras not only satisfy model (1), but also meet the following conditions.

[0099] R c =R l R' r ,T c =Rl(T r -T l (3)

[0100] Let the two equations in model (1) be F(x) and G(y), respectively. At this point, the camera's intrinsic parameters do not need to be considered, therefore the variables are... and These are the extrinsic parameters of the left camera and the right camera, respectively. In this embodiment, l represents the left camera, corresponding to the second camera; r represents the right camera, corresponding to the second camera. During the optimization process, model (1) obtains the iterative equation (first-order Taylor expansion) for each point by taking the partial derivatives of the extrinsic parameters:

[0101]

[0102]

[0103]

[0104]

[0105] Combining equations (3) and (4), the Lagrange multiplier optimization model for the global parameters is as follows:

[0106]

[0107] The traditional model (5) has high computational complexity, requires multiple iterations, and generates a large amount of computation. This paper adopts an alternative approach to solve the same problem.

[0108] 2. The binocular residual optimization in this application embodiment is mainly based on monocular vision.

[0109] Based on the above description, this application proposes a binocular residual optimization model primarily based on monocular vision, and transforms the extrinsic parameters of a single camera to form a model that is only related to the relative pose of the two cameras. However, it should be noted that this operation is related to the subsequent epipolar correction step. That is, if the reference plane selected for epipolar correction is the pixel plane of the left camera, then the model in this paper is mathematically almost identical to step 1, but it is faster and less complex. If another plane is selected for epipolar correction, then the extrinsic parameters of both cameras are involved in the optimization, which is an aspect that needs to be considered in pursuit of accuracy.

[0110] Based on model (1), the partial derivatives of each extrinsic parameter in the adjustment model of the right camera can be obtained.

[0111]

[0112]

[0113]

[0114] in This is a simple transformation of model (1).

[0115] Note that in equation (3), the partial derivatives of equation (6-1) with respect to the extrinsic parameters of the right camera can be transformed into the partial derivatives of model (1) with respect to the relative pose between the two cameras, given that the relative pose is...

[0116] Therefore, equation (6-1) can be rewritten as follows:

[0117]

[0118]

[0119] For equation (6-2), it is not necessary to deal with the complex problems in model (5), but only to optimize the following problem.

[0120]

[0121] Solving this problem is quite simple. In this embodiment, we only need to iterate over equation (7) to find the relative pose that satisfies the condition of very small error.

[0122] 3. Traditional methods for calculating binocular models

[0123] As explained above, optimizing model (7) is equivalent to iteratively obtaining the true value of equation (6-2). For equation (6-2), it is necessary to adjust the relative pose for model (1). Calculate the partial derivative for each component. Note that solving this problem requires knowing R in equation (3). c and T c ,in ω c and κ c The corresponding R c Translation vector T c They are respectively

[0124]

[0125] T c =(X c ,Y c Z c (8)

[0126] Given the extrinsic parameters of the right camera The corresponding R r and T r for

[0127]

[0128] T r =(X r ,Y r Z r (9-1)

[0129] Left camera external reference The corresponding R l and T l for

[0130]

[0131] T l =(X l ,Y l Z l (9-2)

[0132] The values ​​mentioned above are fixed constant values.

[0133] By combining formulas (3), (8), and (9-2) in the traditional way, we can obtain that each element of the right camera extrinsic parameter matrix and vector satisfies the following system of equations.

[0134]

[0135] a r2 =cos(ω c sin(κ) c )a l1 +cos(ω c cos(κ) c )a l2 -sin(ω c )a l3

[0136]

[0137]

[0138] b r2 =cos(ω c sin(κ) c )b l1 +cos(ω c )cos(κ c )b l2 -sin(ω c )b l3

[0139]

[0140]

[0141] c r2 =cos(ω c sin(κ) c )c l1 +cos(ω c )cos(κ c )c l2 -sin(ω c )c l3

[0142]

[0143] X r =X c a l1 +Y c a l2 +Z c a l3 +X l

[0144] Yr =X c b l1 +Y c b l2 +Z c b l3 +Y l

[0145] Z r =X c c l1 +Y c c l2 +Z c a l3 +Z l (10)

[0146] The traditional method involves substituting formula (10) into model (1), which can convert all a... r ,b r ,c r ,X r ,Y r Z r use The nonlinear combination of the equations can be used to obtain the partial derivative equations of the collinear model (1) with respect to all relative pose variables. The numerical values ​​close to the true values ​​can be calculated by iteration.

[0147] In this process, calculating the partial derivatives of each parameter of the collinear equations for the relative pose relationship variables is extremely complex. It can be noted that in equation (10), there are a large number of variables, and all of them are nonlinear combination equations.

[0148] 4. Optimization methods used in the embodiments of this application

[0149] Therefore, in actual programming, it is necessary to find methods to reduce computational complexity. Through observation of the model, the core of formula (6-2) is to obtain the coefficient of each variable before its increment during iteration. That is, to obtain the coefficient of each... and Based on the chain rule of differentials, the derivation can be performed.

[0150]

[0151] Note that R on the right side of equation (11) rij R is the extrinsic parameter matrix of the right camera. r For each element, therefore, we need to obtain the right side of equation (11). To find the value of , we only need to obtain the entire matrix R. r right The matrix partial derivatives of each variable can be obtained. According to the transformation relation equation (3), we can obtain...

[0152] R r =R' c R l ,T r =R' l T c +T l

[0153]

[0154] Therefore, in equation (11) It becomes

[0155]

[0156] That is, in R' c For each Find the partial derivative matrix of the variables, and then multiply the right side of the matrix by the left camera extrinsic parameter matrix R. l Taking the element in the i-th row and j-th column is... The value.

[0157] Then solve equation (11) After the value, The value is obtained by taking the partial derivative of the first equation in the collinear equation (1) with respect to the element in the i-th row and j-th column of the right camera extrinsic matrix. Thus, the value of each coefficient in equation (11) is obtained. It is not necessary to solve the complex nonlinear equation system that combines the collinear equation (1) and equation (10). It is only necessary to find the elements in the matrix according to equation (11) and equation (13). It should be noted that the partial derivative of each of the above matrices is just the derivative of a very simple trigonometric function. Therefore, the complexity is very low and the model is very intuitive. Thus, the model is completed. The matrix model is based on the collinear equation (1) and is constructed according to equation (11), equation (13) and equation (6-2). In this embodiment, by understanding the model and combining the chain rule and simple partial differential calculation, the computational complexity of dual-target calibration is greatly reduced, the calibration efficiency is improved, and fast dual-target calibration is achieved. Furthermore, through the above analysis, the implementation method is improved, and the obtained calibration results are no different from those of the traditional method.

[0158] It should be understood that, although the above Figure 2 The steps in the flowchart are shown sequentially as indicated by the arrows, but these steps are not necessarily executed in the exact order indicated by the arrows or numbers. Unless otherwise specified in this document, there is no strict order in which these steps are performed; they can be executed in other orders. Figure 2At least some of the steps in the process may include multiple steps or multiple stages. These steps or stages are not necessarily completed at the same time, but may be executed at different times. The execution order of these steps or stages is not necessarily sequential, but may be executed in turn or alternately with other steps or at least some of the steps or stages in other steps.

[0159] In one embodiment, such as Figure 3 The diagram shown is a structural block diagram of a camera relative pose parameter value determination device based on binocular vision in one embodiment. Figure 3 A device for determining camera relative pose parameters based on binocular vision is provided. This device can be a software module, a hardware module, or a combination of both as part of a terminal device. Specifically, the device includes: a first partial derivative module 302, a second partial derivative module 304, a third partial derivative module 306, a pixel deviation relationship acquisition module 308, and an iteration module 310, wherein:

[0160] The first partial derivative module 302 is used to calculate the partial derivative of each matrix element in the first camera extrinsic matrix based on the collinearity equation to obtain the first relational expression.

[0161] The second partial derivative module 304 is used to calculate the partial derivative of each relative pose parameter based on the relative pose parameter matrix, and determine the product with the preset second camera extrinsic parameter matrix to obtain the second relational expression.

[0162] The third partial derivative module 306 is used to determine the partial derivative function relationship of the collinear equation with respect to the relative pose parameters based on the product of the first relation and the second relation.

[0163] The pixel deviation relationship acquisition module 308 is used to obtain the pixel deviation relationship between the true coordinates of the image and the collinearity equation based on the product of the increment of the relative pose parameter and the partial derivative function relationship.

[0164] The iteration module 310 is used to iterate the pixel deviation relationship. When the iteration is completed, the relative pose parameter values ​​of the first camera are obtained.

[0165] In this embodiment, analysis revealed that the coefficients in the pixel deviation relationship can be decomposed into a first relationship and a second relationship. Furthermore, the collinearity equation contains each matrix element of the first camera extrinsic parameter matrix, and the relative pose parameter matrix contains the relative pose parameters. By using matrix form to solve the problem, matrix partial differentials are used to replace the cumbersome calculation of partial derivatives of variables in the collinearity equation. The model is simple in form and very convenient to program, greatly reducing the computational complexity. It can quickly complete the calculation of the relative pose parameter values, and the results of binocular vision calibration are consistent with the calibration results of traditional methods.

[0166] In one embodiment, the iteration module 310 is used to: adjust the values ​​of the relative pose parameters in the pixel deviation relationship, and obtain the values ​​of each relative pose parameter of the first camera when the pixel deviation relationship satisfies the error condition.

[0167] In this embodiment, the values ​​of the relative pose parameters in the pixel deviation relationship are adjusted. When the pixel deviation relationship satisfies the error condition, the values ​​of each relative pose parameter of the first camera are obtained. Only one model needs to be iterated and optimized, which can greatly reduce the computational complexity, improve the efficiency of determining the relative pose parameter values ​​of the camera, and thus improve the calibration efficiency.

[0168] In one embodiment, the device further includes a target first camera extrinsic parameter matrix acquisition module, configured to: substitute the relative pose parameter values ​​into the relative pose parameter matrix to obtain the target relative pose matrix; and obtain the target first camera extrinsic parameter matrix based on the product of the transpose of the target relative pose matrix and a preset second camera extrinsic parameter matrix.

[0169] In this embodiment, the relative pose parameter values ​​are substituted into the relative pose parameter matrix to obtain the target relative pose matrix; the target first camera extrinsic matrix is ​​obtained by multiplying the transpose of the target relative pose matrix with the preset second camera extrinsic matrix; the extrinsic matrices of the first camera and the second camera are obtained; the relative pose matrix between the first camera and the second camera is obtained; and all calibration parameters of the stereo camera are obtained by combining the known intrinsic parameters of the first camera and the second camera.

[0170] In one embodiment, the relative pose parameter matrix contains relative pose parameters; the collinearity equation contains matrix elements of the first camera extrinsic parameter matrix; the partial derivative function relationship is determined based on the product of two partial derivative functions decomposed by the chain rule, one of which is the partial derivative of the collinearity equation with respect to the first camera extrinsic parameter matrix, and the other is the partial derivative of the first camera extrinsic parameter matrix with respect to the relative pose parameters.

[0171] In this embodiment, directly calculating the partial derivatives of the collinearity equations with respect to the relative pose parameters is computationally complex. However, analysis reveals that decomposing the equations using the chain rule yields two partial derivative functions, which are very simple to calculate and can quickly provide the values ​​of the relative pose parameters.

[0172] Specific limitations regarding the device for determining camera relative pose parameters based on binocular vision can be found in the limitations of the method for determining camera relative pose parameters based on binocular vision described above, and will not be repeated here. Each module in the aforementioned device for determining camera relative pose parameters based on binocular vision can be implemented entirely or partially through software, hardware, or a combination thereof. These modules can be embedded in or independent of the processor in the terminal device in hardware form, or stored in the memory of the terminal device in software form, so that the processor can call and execute the operations corresponding to each module.

[0173] In one embodiment, a terminal device is provided, the internal structure of which can be shown as follows: Figure 4 As shown, the terminal device includes a processor, memory, communication interface, display screen, and input device connected via a system bus. The processor provides computing and control capabilities. The memory includes non-volatile storage media and internal memory. The non-volatile storage media stores the operating system and computer programs. The internal memory provides an environment for the operation of the operating system and computer programs stored in the non-volatile storage media. The communication interface is used for wired or wireless communication with external terminals; wireless communication can be achieved through Wi-Fi, carrier networks, NFC (Near Field Communication), or other technologies. When executed by the processor, the computer program implements a method for determining camera relative pose parameters based on binocular vision. The display screen can be an LCD screen or an e-ink screen. The input device can be a touch layer covering the display screen, buttons, a trackball, or a touchpad on the terminal device's casing, or an external keyboard, touchpad, or mouse.

[0174] Those skilled in the art will understand that Figure 4 The structure shown is merely a block diagram of a portion of the structure related to the present application and does not constitute a limitation on the terminal device to which the present application is applied. A specific terminal device may include more or fewer components than those shown in the figure, or combine certain components, or have different component arrangements.

[0175] In one embodiment, a terminal device is provided, including a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to implement the steps of the above-described method embodiments.

[0176] In one embodiment, a computer-readable storage medium is provided having a computer program stored thereon, which, when executed by a processor, implements the steps of the above-described method embodiments.

[0177] In one embodiment, a computer program product or computer program is provided, comprising computer instructions stored in a computer-readable storage medium. A processor of a terminal device reads the computer instructions from the computer-readable storage medium and executes the computer instructions, causing the terminal device to perform the steps described in the above method embodiments.

[0178] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. This computer program can be stored in a non-volatile computer-readable storage medium, and when executed, it can include the processes described in the embodiments of the above methods. Any references to memory, storage, databases, or other media used in the embodiments provided in this application can include at least one of non-volatile and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, or optical storage, etc. Volatile memory can include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM can be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM), etc.

[0179] The above description is only a preferred embodiment of this application and does not limit the patent scope of this application. Any equivalent structural or procedural changes made based on the content of this application's specification and drawings, or direct or indirect applications in other related technical fields, are similarly included within the patent protection scope of this application.

Claims

1. A method for determining relative pose parameter values of cameras based on binocular vision, characterized in that, The method comprises: According to the partial derivative of the collinear equation to each matrix element in the first camera extrinsic parameter matrix, a first relationship is obtained; According to the partial derivative of the relative pose parameter matrix to each relative pose parameter, and the product with the preset second camera extrinsic parameter matrix, a second relationship is obtained; the preset second camera extrinsic parameter matrix refers to a known camera extrinsic parameter matrix pre-stored in the terminal device; According to the product of the first relationship and the second relationship, a partial derivative function relationship of the collinear equation to the relative pose parameter is determined; the partial derivative function relationship is determined based on the product of two partial derivative functions decomposed through the chain rule, one of which is the partial derivative of the collinear equation to the first camera extrinsic parameter matrix, and the other is the partial derivative of the first camera extrinsic parameter matrix to the relative pose parameter; According to the product of the increment of the relative pose parameter and the partial derivative function relationship, a pixel deviation relationship between the image real coordinates and the collinear equation is obtained; The pixel deviation relationship is iterated, and when the iteration is completed, the values of each relative pose parameter of the first camera are obtained.

2. The method of claim 1, wherein, The pixel deviation relationship is iterated, and when the iteration is completed, the values of each relative pose parameter of the first camera are obtained. The method further comprises:

3. The method of claim 1, wherein, The values of the relative pose parameters are substituted into the relative pose parameter matrix to obtain a target relative pose matrix; According to the product of the transpose matrix of the target relative pose matrix and the preset second camera extrinsic parameter matrix, a target first camera extrinsic parameter matrix is obtained. The relative pose parameter matrix contains the relative pose parameters; the collinear equation contains the matrix elements of the first camera extrinsic parameter matrix.

4. The method according to any one of claims 1 to 3, characterized in that, The apparatus comprises:

5. A device for determining relative pose parameter values of cameras based on binocular vision, characterized in that A first partial derivative module is configured to obtain a first relationship by performing partial derivation of a collinear equation to each matrix element in a first camera extrinsic parameter matrix; A second partial derivative module is configured to obtain a second relationship by performing partial derivation of a relative pose parameter matrix to each relative pose parameter, and determining the product with a preset second camera extrinsic parameter matrix; the preset second camera extrinsic parameter matrix refers to a known camera extrinsic parameter matrix pre-stored in the terminal device; A third partial derivative module is configured to determine a partial derivative function relationship of the collinear equation to the relative pose parameter according to the product of the first relationship and the second relationship; A pixel deviation relationship obtaining module is configured to obtain a pixel deviation relationship between image real coordinates and the collinear equation according to the product of the increment of the relative pose parameter and the partial derivative function relationship; the partial derivative function relationship is determined based on the product of two partial derivative functions decomposed through the chain rule, one of which is the partial derivative of the collinear equation to the first camera extrinsic parameter matrix, and the other is the partial derivative of the first camera extrinsic parameter matrix to the relative pose parameter. ​ An iteration module is configured to iterate the pixel deviation relation, and obtain each relative pose parameter value of the first camera when the iteration is completed.

6. The apparatus of claim 5, wherein, The iteration module is configured to: adjust the value of the relative pose parameter in the pixel deviation relation, and obtain each relative pose parameter value of the first camera when the pixel deviation relation satisfies an error condition.

7. The apparatus of claim 5, wherein, Further comprising a target first camera extrinsic parameter matrix obtaining module configured to: substitute the relative pose parameter value into the relative pose parameter matrix to obtain a target relative pose matrix; obtain a target first camera extrinsic parameter matrix according to the product of the transpose matrix of the target relative pose matrix and the preset second camera extrinsic parameter matrix.

8. The device of any one of claims 5 to 7, wherein, The relative pose parameter matrix contains the relative pose parameter; and the collinear equation contains the matrix element of the first camera extrinsic parameter matrix. 9.A terminal device, comprising a memory and a processor, wherein the memory stores a computer program, and the terminal device is characterized in that, The processor executes the computer program to implement the steps of the method in any one of claims 1 to 4.

10. A computer-readable storage medium having stored thereon a computer program, characterized in that, The computer program is executed by the processor to implement the steps of the method in any one of claims 1 to 4.

Images