Relative pose calibration method, device and equipment of binocular camera and medium

Abstract

The embodiment of the invention discloses a relative pose calibration method and device of a binocular camera, equipment and a medium, and relates to the technical field of vision measurement, and the method comprises the steps: determining an affine corresponding relation between two lenses in the binocular camera, and respectively determining Pluecker straight lines at different positions; according to the original rotation matrix and the original translation vector of the binocular camera, combining an affine corresponding relation and a Pluecker straight line to construct a generalized camera epipolar constraint model; respectively obtaining attitude information at different positions through an inertial measurement unit, aligning a Y axis of a binocular camera coordinate system to obtain an alignment rotation matrix and an alignment translation vector, and adjusting a generalized camera polar constraint model; and performing polynomial eigenvalue solution to obtain a relative pose. Through geometric model optimization, efficient and robust calibration of the relative pose of the binocular camera under the unknown focal length condition is realized, a plug-and-play calibration solution is provided for a visual system of a dynamic platform, and the technical blank in the field is filled.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of visual measurement, in particular to a relative pose calibration method, device and equipment of a binocular camera and a medium. BACKGROUND

[0002] In the field of computer vision and robotics, the relative pose estimation of a binocular camera on a moving platform is widely used in visual odometry, motion structure from motion, simultaneous localization and mapping, and other fields. With the development of technology, more and more application scenarios require the use of binocular camera systems. Compared with monocular cameras, binocular cameras can provide a larger field of view and can estimate the scale information of translation. Common methods for relative pose estimation of binocular cameras include the 17-point method and the 6-point method.

[0003] However, the above methods are developed under the condition that the focal length is known. For a binocular camera on a moving platform, the focal length of the binocular camera may be missing. If the parameters of different lenses are adjusted, the focal length of the binocular camera may change. Related technologies need to reacquire the focal length of the binocular camera through calibration, and then recalculate the relative pose of the binocular camera according to the calibrated focal length.

[0004] However, compared with the pinhole model of a monocular camera, all light rays in the generalized model of a binocular camera cannot be focused on a light center, which makes the generalized limit constraint model of a binocular camera more complex than the limit constraint model of a monocular camera. In the generalized limit constraint model of a binocular camera, the focal length and the relative pose of the binocular camera are coupled together, resulting in high computational complexity. It is therefore difficult to separately calculate the focal length and the relative pose of the binocular camera when the focal length is unknown.

[0005] Therefore, there is currently a lack of a method for calibrating the relative pose of a binocular camera when the focal length is unknown. SUMMARY

[0006] The embodiments of the present application provide a relative pose calibration method, device, equipment and medium of a binocular camera to solve the defects of the related art. The technical solution is as follows: In a first aspect, the embodiments of the present application provide a relative pose calibration method of a binocular camera, comprising: moving the binocular camera from a first position to a second position, and determining an affine correspondence relationship between two lenses in the binocular camera according to images collected at the first position and images collected at the second position; determining a Plücker line at the first position and a Plücker line at the second position based on the affine correspondence relationship; According to the original rotation matrix and the original translation vector of the binocular camera moving from the first position to the second position, the affine correspondence, the Plücker line at the first position and the Plücker line at the second position are combined to construct a generalized camera epipolar constraint model; The attitude information at the first position and the attitude information at the second position are obtained respectively by an inertial measurement unit fixedly connected with the binocular camera; The Y axes of the first binocular camera coordinate system at the first position and the second binocular camera coordinate system at the second position are aligned based on the two attitude information, an alignment rotation matrix and an alignment translation vector are obtained, and the generalized camera epipolar constraint model is adjusted; The adjusted generalized camera epipolar constraint model is solved by polynomial eigenvalue, and a relative pose obtained by the solving is output, the relative pose including the original rotation matrix and the original translation vector.

[0007] In an optional solution of the first aspect, the affine correspondence between the two lenses in the binocular camera is determined according to images collected at the first position and images collected at the second position when the binocular camera moves from the first position to the second position, and the affine correspondence includes: The binocular camera is located at the first position at a first time and moves to the second position at a second time; Image coordinates of each target point in the images collected at the first position are obtained, and image coordinates of each target point in the images collected at the second position are obtained; A first rotation matrix and a first translation vector of a first camera in the binocular camera moving from the first position to the second position are determined, and a second rotation matrix and a second translation vector of a second camera in the binocular camera moving from the second position to the second position are determined; The affine correspondence is obtained based on the image coordinates, the first rotation matrix, the first translation vector, the second rotation matrix and the second translation vector.

[0008] In an optional solution of the first aspect, the Plücker line at the first position and the Plücker line at the second position are determined based on the affine correspondence, and the Plücker line at the first position includes: A Plücker line corresponding to the first position is determined based on the image coordinates of each target point in the images collected at the first position, the first rotation matrix and an intrinsic parameter matrix of the binocular camera, and a formula is applied: ; ; ; determining a Plucker line corresponding to the second position based on image coordinates of each of the target points in the image collected at the second position, the second rotation matrix, and an internal parameter matrix of the binocular camera, and applying a formula: ; ; ; ; wherein i represents a parameter corresponding to the first time, subscript j represents a parameter corresponding to the second time, subscript represents a parameter corresponding to the first camera, and subscript represents a parameter corresponding to the second camera, is a Plucker line corresponding to the first position, is the first rotation matrix, represents image coordinates of each target point in the image collected at the first position, is a Plucker line corresponding to the second position, is the second rotation matrix, represents image coordinates of each target point in the image collected at the first position, , , , are all parameters, is the internal parameter matrix, and f is the reciprocal of the focal length.

[0009] In an optional solution of the first aspect, an essential matrix is constructed according to an original rotation matrix and an original translation vector of the binocular camera moving from the first position to the second position, and a formula is applied: ; The generalized camera epipolar constraint model is constructed by combining the affine correspondence relationship, the Plucker line at the first position, and the Plucker line at the second position, and a formula is applied: ; wherein is the essential matrix, is the original rotation matrix, is the original translation vector, is the skew-symmetric matrix of the original translation vector t.

[0010] In an optional solution of the first aspect, the attitude information at the first position and the attitude information at the second position are respectively obtained by an inertial measurement unit fixedly connected to the binocular camera, and the inertial measurement unit includes: The pitch angle and the roll angle corresponding to the first moment and the pitch angle and the roll angle corresponding to the second moment are obtained respectively under the premise that the position and the connection relationship between the inertial measurement unit and the binocular camera are calibrated through the inertial measurement unit fixedly connected with the binocular camera. The Y axes of the first binocular camera coordinate system at the first position and the second binocular camera coordinate system at the second position are aligned based on the two pieces of attitude information, to obtain an alignment rotation matrix and an alignment translation vector, including: The first inertial system rotation matrix corresponding to the first moment is estimated based on the pitch angle and the roll angle of the first moment, and the second inertial system rotation matrix corresponding to the second moment is estimated based on the pitch angle and the roll angle of the second moment. The first binocular camera coordinate system and the second binocular camera coordinate system are aligned to a preset vertical direction based on the first inertial system rotation matrix and the second inertial system rotation matrix, to determine the relationship between the alignment rotation matrix and the rotation angle between the first binocular camera coordinate system and the second binocular camera coordinate system after alignment, and the formula is applied: ; ; ; The relationship between the alignment translation vector and the original translation vector between the first binocular camera coordinate system and the second binocular camera coordinate system after alignment is determined, and the formula is applied: ; An adjusted generalized camera epipolar constraint model is obtained, and the formula is applied: ; ; wherein, is the first inertial system rotation matrix, is the second inertial system rotation matrix, y is an intermediate parameter, is a parameter, is the alignment translation vector, is a rotation angle around the Y axis when the Y axes of the first binocular camera coordinate system at the first position and the second binocular camera coordinate system at the second position are aligned, is the alignment rotation matrix between the first binocular camera coordinate system and the second binocular camera coordinate system after alignment.

[0011] In a second aspect, the embodiments of the present application further provide a relative pose calibration device of a binocular camera, including: The affine relationship determination unit is configured to move the binocular camera from the first position to the second position, and determine an affine correspondence relationship between two lenses in the binocular camera according to images collected at the first position and images collected at the second position. The constraint model construction unit is configured to determine a Plücker straight line at the first position and a Plücker straight line at the second position based on the affine correspondence relationship. The constraint model construction unit is further configured to construct a generalized camera epipolar constraint model by combining the affine correspondence relationship, the Plücker straight line at the first position, and the Plücker straight line at the second position according to an original rotation matrix and an original translation vector of the binocular camera when moving from the first position to the second position. The constraint model construction unit is further configured to obtain attitude information at the first position and attitude information at the second position respectively by using an inertial measurement unit fixedly connected to the binocular camera. The constraint model construction unit is further configured to align a Y-axis of a first binocular camera coordinate system at the first position with a Y-axis of a second binocular camera coordinate system at the second position based on the two pieces of attitude information, and adjust the generalized camera epipolar constraint model by using an alignment rotation matrix and an alignment translation vector. The calculation unit is configured to perform polynomial eigenvalue solving on the adjusted generalized camera epipolar constraint model, and output a relative pose obtained by solving, wherein the relative pose includes the original rotation matrix and the original translation vector.

[0012] In an optional implementation of the second aspect, the affine relationship determination unit is configured to move the binocular camera from the first position to the second position, and determine an affine correspondence relationship between two lenses in the binocular camera according to images collected at the first position and images collected at the second position, and the affine relationship determination unit includes: The binocular camera is located at the first position at a first time, and is moved to the second position at a second time. The affine relationship determination unit is configured to obtain image coordinates of each target point in the images collected at the first position, and obtain image coordinates of each target point in the images collected at the second position. The affine relationship determination unit is further configured to determine a first rotation matrix and a first translation vector of a first camera in the binocular camera when moving from the first position to the second position, and determine a second rotation matrix and a second translation vector of a second camera in the binocular camera when moving from the second position to the second position. The affine relationship determination unit is further configured to obtain the affine correspondence relationship based on the image coordinates, the first rotation matrix, the first translation vector, the second rotation matrix, and the second translation vector.

[0013] In an optional implementation of the second aspect, the constraint model construction unit is configured to determine the Plücker line at the first position and the Plücker line at the second position based on the affine correspondence, including: The constraint model construction unit determines the Plücker line corresponding to the first position based on the image coordinates of each target point in the image collected at the first position, the first rotation matrix, and an intrinsic parameter matrix of the binocular camera, and applies the formula: ; ; ; The constraint model construction unit determines the Plücker line corresponding to the second position based on the image coordinates of each target point in the image collected at the second position, the second rotation matrix, and the intrinsic parameter matrix of the binocular camera, and applies the formula: ; ; ; ; wherein i represents a parameter corresponding to the first time, subscript j represents a parameter corresponding to the second time, subscript i represents a parameter corresponding to the first camera, and subscript j represents a parameter corresponding to the second camera. is a parameter corresponding to the first camera, and subscript is a parameter corresponding to the second camera, is the Plücker line corresponding to the first position, is the first rotation matrix, is the image coordinates of each target point in the image collected at the first position, is the Plücker line corresponding to the second position, is the second rotation matrix, is the image coordinates of each target point in the image collected at the first position, , , , are all parameters, is the intrinsic parameter matrix, and f is the reciprocal of the focal length.

[0014] In a third aspect, the embodiments of the present application further provide an electronic device, including a memory, a processor, and a computer program stored in the memory and capable of running on the processor, and the processor executes the program to implement the method provided by the first aspect or any one of the implementation manners of the first aspect.

[0015] In a fourth aspect, the present application also provides a non-transitory computer-readable storage medium having stored thereon a computer program which, when executed by a processor, implements the method provided in the first aspect of the embodiments of the present application or any of the implementation manners of the first aspect.

[0016] The technical scheme provided by some embodiments of the present application has at least the following beneficial effects: The embodiments of the present application decouple the focal length parameter and the pose by establishing the Plucker line through the affine correspondence, so that the model can directly solve the relative pose without pre-calibrating the focal length, solve the focal length re-calibration problem caused by lens parameter adjustment or data loss, and reduce the system maintenance complexity. On the other hand, the embodiments of the present application also obtain the attitude information of the double positions through the fixedly connected IMU, and unify the coordinate system by using the Y-axis alignment strategy, which significantly reduces the influence of the cumulative error in the motion process. Compared with the pure visual method, the inertial data provided by the IMU can compensate for the pose estimation deviation in the fast motion or texture missing scene. The aligned coordinate system further simplifies the solving process of the generalized epipolar constraint model, so that the polynomial eigenvalue solving is more stable, and is especially suitable for the high-frequency calibration demand of dynamic platforms such as unmanned aerial vehicles and mobile robots. In addition, the embodiments of the present application realize the efficient and robust calibration of the relative pose of the binocular camera under the condition of unknown focal length through geometric model optimization, provide a plug-and-play calibration solution for the visual system of the dynamic platform, and fill the technical gap in this field. BRIEF DESCRIPTION OF DRAWINGS

[0017] In order to more clearly illustrate the technical solutions in the present application or the related art, the following will briefly introduce the drawings needed to be used in the embodiments or the related art description. Obviously, the drawings in the following description are some embodiments of the present application, and those skilled in the art can also obtain other drawings according to these drawings without any creative effort.

[0018] Figure 1 is a flow diagram of a relative pose calibration method of a binocular camera provided by an embodiment of the present application; Figure 2 is an affine correspondence diagram of a relative pose calibration method of a binocular camera provided by an embodiment of the present application; Figure 3 is an affine correspondence diagram of a relative pose calibration method of a binocular camera provided by an embodiment of the present application; Figure 4 is a structural diagram of a relative pose calibration device of a binocular camera provided by an embodiment of the present application; Figure 5 is a structural diagram of an electronic device provided by an embodiment of the present application. DETAILED DESCRIPTION

[0019] In order to make the objects, technical solutions and advantages of the present application clearer, the following will clearly and completely describe the technical solutions in the present application with reference to the drawings in the present application. Obviously, the described embodiments are only a part of the embodiments of the present application, rather than all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by a person of ordinary skill in the art without creative work fall within the protection scope of the present application.

[0020] The terms "comprising" and "having" and any variations thereof in the specification and claims of the present application and the above drawings are intended to cover the non-exclusive inclusion. For example, a process, method, system, product or device including a series of steps or modules is not limited to the listed steps or modules, but can optionally further include steps or modules not listed, or can optionally further include other steps or modules inherent to the process, method, product or device.

[0021] It should be noted that the terms "first" and "second" in the present application are only to distinguish similar objects, and do not represent a specific order of the objects. Understandably, "first" and "second" can be interchanged in a specific order or sequence as permitted. It should be understood that the objects distinguished by "first" and "second" can be interchanged under appropriate circumstances, so that the embodiments of the present application described herein can be implemented in an order other than those described or illustrated herein.

[0022] Next, in conjunction with Figure 1 , a relative pose calibration method of a binocular camera provided by an embodiment of the present application is introduced. For details, please refer to Figure 1 , Figure 1 Fig. 1 shows a flowchart of a relative pose calibration method of a binocular camera provided by an embodiment of the present application. As shown in Figure 1 , the method comprises the following steps: S101, moving the binocular camera from a first position to a second position, and determining an affine correspondence relationship between two lenses in the binocular camera according to an image collected at the first position and an image collected at the second position; S102, determining a Plücker line at the first position and a Plücker line at the second position based on the affine correspondence relationship; S103, constructing a generalized camera epipolar constraint model according to an original rotation matrix and an original translation vector of the binocular camera moving from the first position to the second position, in combination with the affine correspondence relationship, the Plücker line at the first position and the Plücker line at the second position; S104, obtaining attitude information at the first position and attitude information at the second position respectively through an inertial measurement unit fixedly connected with the binocular camera; S105: aligning the Y axis of the first binocular camera coordinate system at the first position with the Y axis of the second binocular camera coordinate system at the second position based on the two posture information, obtaining an alignment rotation matrix and an alignment translation vector, and adjusting the generalized camera epipolar constraint model; S106: Solve polynomial eigenvalues ​​of the adjusted generalized camera epipolar constraint model, and output a relative pose obtained by the solution, where the relative pose includes the original rotation matrix and the original translation vector.

[0023] In some embodiments, in S101, combining Figures 2-3 As shown, Figure 2 As shown, taking the target point P as an example, one of the lenses of the binocular camera is the first camera, denoted as , the other lens is the second camera, denoted as , get the first camera at the first moment i The image collected contains point P, and the corresponding projection point of point P in the image is point Pi. The coordinates of the projection point Pi are , get the second camera at the second moment j The image collected contains point P, and the corresponding projection point of point P in the image is point Pj. The coordinates of the projection point Pj are .

[0024] like Figure 3 As shown, the binocular camera is mounted on a moving platform. After the binocular camera moves, it is located at the first position at the first moment i and moves to the second position at the second moment j. The relative posture can be understood as the relative posture change of each lens in the binocular camera from the first position to the second position. The binocular camera coordinate system at the first moment i at the first position is recorded as the first binocular camera coordinate system, and the binocular camera coordinate system at the second position at the second moment j is recorded as the second binocular camera coordinate system; The rotation matrix relative to the second binocular camera coordinate system is recorded as , the translation vector is recorded as , and cameras The rotation matrix relative to the second binocular camera coordinate system is recorded as , the translation vector is recorded as .

[0025] Thus, the affine correspondence relationship of the binocular camera can be obtained based on the image coordinates, the first rotation matrix, the first translation vector, the second rotation matrix and the second translation vector. .

[0026] Furthermore, in S102, the Plücker line at the first position and the Plücker line at the second position may be determined based on the affine correspondence, including the following steps: determining a Plucker line corresponding to the first position based on the image coordinates of each target point in the image collected at the first position, the first rotation matrix, and an internal parameter matrix of the binocular camera, by applying a formula: ; ; ; determining a Plucker line corresponding to the second position based on the image coordinates of each target point in the image collected at the second position, the second rotation matrix, and the internal parameter matrix of the binocular camera, by applying a formula: ; ; ; ; wherein i represents a parameter corresponding to the first time, subscript j represents a parameter corresponding to the second time, subscript represents a parameter corresponding to the first camera, and subscript represents a parameter corresponding to the second camera, is the Plucker line corresponding to the first position, is the first rotation matrix, represents the image coordinates of each target point in the image collected at the first position, is the Plucker line corresponding to the second position, is the second rotation matrix, represents the image coordinates of each target point in the image collected at the first position, , , , are all parameters, is the internal parameter matrix, the internal parameter matrix includes all internal parameters of the binocular camera except the focal length, and f is the reciprocal of the focal length. In the embodiments of the present application, the focal length is an unknown quantity.

[0027] Further, before S103, the essential matrix E can be constructed according to the original rotation matrix and the original translation vector of the binocular camera when moving from the first position to the second position, by applying a formula: ; wherein R is an original rotation matrix of the first binocular camera coordinate system relative to the second binocular camera coordinate system before Y axes of the first binocular camera coordinate system and the second binocular camera coordinate system are aligned to a preset vertical direction (for example, the preset vertical direction is determined according to a world coordinate system); t is an original translation vector of the first binocular camera coordinate system relative to the second binocular camera coordinate system before vertical directions of the first binocular camera coordinate system and the second binocular camera coordinate system are aligned, is a skew-symmetric matrix of the original translation vector t.

[0028] Further, S103 further includes: Combined with the affine correspondence, the Plücker line at the first position and the Plücker line at the second position, a generalized camera epipolar constraint is constructed: .

[0029] In some embodiments, S104 obtains the attitude information at the first position and the attitude information at the second position through an inertial measurement unit fixedly connected with the binocular camera, including: Through the inertial measurement unit fixedly connected with the binocular camera, the pitch angle and the roll angle corresponding to the first time and the pitch angle and the roll angle corresponding to the second time are obtained respectively on the premise that the position and the connection relationship between the inertial measurement unit and the binocular camera have been calibrated; The Y axes of the first binocular camera coordinate system at the first position and the second binocular camera coordinate system at the second position are aligned based on the two attitude information, to obtain an alignment rotation matrix and an alignment translation vector , including: The first inertial system rotation matrix corresponding to the first time i is estimated based on the pitch angle and the roll angle; the second inertial system rotation matrix corresponding to the second time j is estimated based on the pitch angle and the roll angle; Further, the step of S105 is executed, the first binocular camera coordinate system and the second binocular camera coordinate system are aligned to the preset vertical direction based on the first inertial system rotation matrix and the second inertial system rotation matrix: The relationship between the alignment rotation matrix and the rotation angle between the first binocular camera coordinate system and the second binocular camera coordinate system after alignment is determined, specifically the relationship between the Y axis component of the alignment rotation matrix and the rotation angle , the formula is applied: ; ; ; determining an alignment translation vector between the aligned first binocular camera coordinate system and the second binocular camera coordinate system a relationship with the original translation vector t, applying the formula: ; Further, adjusting the generalized camera epipolar constraint model based on the alignment rotation matrix and the alignment translation vector, comprising: When the vertical direction of the Y axis of the binocular camera system coordinate system is known, the generalized camera epipolar constraint can be expressed as: =0; Based on the Plücker line at time i and the Plücker line at time j, the processing obtains: =0; After decoupling the rotation matrix and the translation vector, the camera epipolar constraint model to be solved can be obtained: ; wherein, , is the alignment translation vector of the Y axis of the binocular camera system coordinate system at two adjacent moments aligned to the vertical direction. is the first inertial system rotation matrix, is the second inertial system rotation matrix, , y is an intermediate parameter, is a parameter, is the rotation angle around the Y axis when aligning the Y axis of the first binocular camera coordinate system at the first position with the Y axis of the second binocular camera coordinate system at the second position, is the alignment rotation matrix between the aligned first binocular camera coordinate system and the second binocular camera coordinate system.

[0030] Further, the step of performing S106, the camera epipolar constraint model is solved by polynomial eigenvalue, comprising: It can be understood that when the camera and the inertial measurement unit are fixedly connected, and the direct relationship between the camera and the inertial measurement unit has been calibrated. The inertial measurement unit provides the pitch angle and the roll angle to align the Y axis of the binocular camera system coordinate system at two adjacent moments to the vertical direction. In this case, the degree of freedom of the relative rotation matrix is reduced from 3 to 1, the degree of freedom of the translation vector is 3, and the degree of freedom of the focal length is 1. In this case, the number of unknowns to be estimated is 5.

[0031] Therefore, five points are randomly selected from the images imaged by the binocular camera, affine correspondence information of the five points is extracted, and the following polynomial eigenvalue solving model is constructed based on the camera epipolar constraint model: ; Wherein, M is a matrix, the dimension of the matrix is 5*4, the matrix is composed of unknown number f and y.

[0032] Considering the non-zero solution in the polynomial characteristic solving model, therefore, the determinant of any 4 rows in the matrix M is zero, therefore, 5 equations can be obtained, the 5 equations only contain unknown number and , the highest order coefficient is , apply formula: ; ; The matrix contains 7 monomials, and the number of equations is 5. In order to make the number of equations equal to the number of monomials, the first three equations in are multiplied by , which obtains: ; ; ; The characteristic value matrix G is extracted: ; Based on the characteristic value matrix G, the solving includes the following steps: Firstly, the characteristic value y of the characteristic value matrix G is obtained by Schur method; Based on the characteristic value y, the characteristic vector L is solved, and the reciprocal of focal length f can be extracted from the characteristic vector L.

[0033] Further, the characteristic value y obtained by solving is solved to obtain the Y axis component rotation matrix ; Further, based on the calculated reciprocal of focal length f and the characteristic value y, substitute into the expression of matrix M, take any four rows to form a 4*4 determinant, solve to obtain the characteristic vector , based on the characteristic vector , the translation vector of the Y axis aligned to the vertical direction can be calculated, and then the original translation vector t is calculated based on the second alignment rotation matrix , apply formula: ; Based on the first alignment rotation matrix , the second alignment rotation matrix , and the Y axis component rotation matrix , the original rotation matrix R is calculated, apply formula: ; Finally, output the relative pose of the binocular camera, including the original rotation matrix R , the original original translation vector t and the focal length 1 / f.

[0034] The following is an apparatus embodiment of the present application, which can be used to execute the method embodiment of the present application. For details not disclosed in the apparatus embodiment of the present application, please refer to the method embodiment of the present application.

[0035] Next, please refer to Figure 4 , a structural schematic diagram of a relative pose calibration device of a binocular camera provided for an exemplary embodiment of the present application. The device can be realized as all or part of a terminal by software, hardware or a combination of the two, and can also be integrated as an independent module on a server. The relative pose calibration device of a binocular camera in an embodiment of the present application can be applied to a terminal or cloud, and the device comprises an affine relationship determination unit, a constraint model construction unit and a calculation unit, wherein: The affine relationship determination unit is configured to move the binocular camera from a first position to a second position, and determine the affine correspondence relationship between two lenses in the binocular camera according to images collected at the first position and images collected at the second position; The constraint model construction unit is configured to determine the Plucker line at the first position and the Plucker line at the second position based on the affine correspondence relationship; The constraint model construction unit is further configured to construct a generalized camera epipolar constraint model according to the original rotation matrix and the original translation vector of the binocular camera moving from the first position to the second position, in combination with the affine correspondence relationship, the Plucker line at the first position and the Plucker line at the second position; The constraint model construction unit is further configured to obtain the attitude information at the first position and the attitude information at the second position respectively through an inertial measurement unit fixedly connected with the binocular camera; The constraint model construction unit is further configured to align the Y-axis of the first binocular camera coordinate system at the first position with the Y-axis of the second binocular camera coordinate system at the second position based on the two attitude information, and adjust the generalized camera epipolar constraint model by using the alignment rotation matrix and the alignment translation vector; The calculation unit is configured to perform polynomial eigenvalue solving on the adjusted generalized camera epipolar constraint model, and output the solved relative pose, wherein the relative pose comprises the original rotation matrix and the original translation vector.

[0036] It should be noted that the apparatus provided in the above embodiments is only used for illustrating the division of the above functional modules when the relative pose calibration method of the binocular camera is performed, and in actual application, the above functions can be completed by different functional modules according to needs, that is, the internal structure of the device is divided into different functional modules to complete all or part of the above described functions. In addition, the apparatus provided in the above embodiments and the relative pose calibration method of the binocular camera belong to the same concept, and the implementation process is embodied in the method embodiments, which will not be described here.

[0037] The embodiments of the present application further provide an electronic device, including a memory, a processor and a computer program stored in the memory and executable on the processor, and the processor implements the steps of the method of any one of the above embodiments when executing the program.

[0038] Please refer to Figure 5 The structure block diagram of an electronic device provided in the embodiments of the present application is shown.

[0039] As Figure 5 shown, the electronic device 500 includes a processor 501 and a memory 502.

[0040] In the embodiments of the present application, the processor 501 is the control center of the computer system, which can be the processor of a physical machine or the processor of a virtual machine. The processor 501 can include one or more processing cores, such as a 4-core processor, an 8-core processor, etc. The processor 501 can be implemented in at least one of the hardware forms of a DSP (Digital Signal Processing), a FPGA (Field-Programmable Gate Array) and a PLA (Programmable Logic Array).

[0041] The processor 501 can also include a main processor and a coprocessor. The main processor is a processor for processing data in a wake-up state, also known as a CPU (Central Processing Unit). The coprocessor is a low-power processor for processing data in a standby state.

[0042] The memory 502 can include one or more computer-readable storage media. The computer-readable storage media can be non-transitory. The memory 502 can also include high-speed random access memory and can include non-volatile memory, such as one or more magnetic disk storage devices, flash memory devices, or other non-volatile solid-state storage devices. In some embodiments of the present application, the non-transitory computer-readable storage medium in the memory 502 is used to store at least one instruction for being executed by the processor 501 to implement the method in the embodiments of the present application.

[0043] In some embodiments, the electronic device 500 further includes a peripheral device interface 503 and at least one peripheral device 504. The processor 501, the memory 502, and the peripheral device interface 503 can be connected through a bus or a signal line. Each peripheral device 504 can be connected to the peripheral device interface 503 through a bus, a signal line, or a circuit board. Specifically, the peripheral device interface 503 can be used to connect at least one peripheral device related to I / O (Input / Output) to the processor 501 and the memory 502.

[0044] In some embodiments of the present application, the processor 501, the memory 502, and the peripheral device interface 503 are integrated on the same chip or circuit board; in some other embodiments of the present application, any one or two of the processor 501, the memory 502, and the peripheral device interface 503 can be implemented on a separate chip or circuit board. The embodiments of the present application do not make specific limitations in this regard.

[0045] The structure block diagram of the electronic device shown in the embodiments of the present application does not constitute a limitation on the electronic device 500, which can include more or fewer components than shown, or combine certain components, or use different arrangements of components.

[0046] The embodiments of the present application also provide a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of the method of any of the preceding embodiments. The computer-readable storage medium can include, but is not limited to, any type of disk including a floppy disk, an optical disk, a DVD, a CD-ROM, a micro-drive, and a magneto-optical disk, a ROM, a RAM, an EPROM, an EEPROM, a DRAM, a VRAM, a flash memory device, a magnetic or optical card, a nanosystem (including molecular memory ICs), or any type of media or device suitable for storing instructions and / or data.

[0047] Those skilled in the art can clearly understand the implementation of the various embodiments by means of software and the necessary general hardware platform from the above description of the embodiments, and of course, the embodiments can also be implemented by hardware. Based on such understanding, the above technical solutions or the part that contributes to the related art can be embodied in the form of a software product, which can be stored in a computer readable storage medium, such as a ROM / RAM, a magnetic disk, an optical disk, and the like, and includes a plurality of instructions to make a computer device (which can be a personal computer, a server, or a network device, etc.) execute the methods described in the various embodiments or some parts of the embodiments.

[0048] Finally, it should be noted that: the above examples are only used to illustrate the technical solutions of the present application, and not to limit them; although the present application has been described in detail with reference to the foregoing examples, those skilled in the art should understand that: it can still modify the technical solutions recorded in the foregoing examples, or make equivalent replacement for some technical features; and these modifications or replacements do not make the essence of the corresponding technical solutions deviate from the spirit and scope of the technical solutions of the embodiments of the present application.

Claims

1. A relative pose calibration method for a binocular camera, characterized in that: include: Moving the binocular camera from a first position to a second position, and determining an affine correspondence between two lenses in the binocular camera based on an image captured at the first position and an image captured at the second position; Determine a Plücker line at a first position and a Plücker line at a second position based on the affine correspondence; A generalized camera epipolar constraint model is constructed based on the original rotation matrix and the original translation vector of the binocular camera moving from the first position to the second position, in combination with the affine correspondence, the Plücker line at the first position, and the Plücker line at the second position; Obtaining posture information at the first position and posture information at the second position respectively through an inertial measurement unit fixedly connected to the binocular camera; Aligning the Y-axis of the first binocular camera coordinate system at the first position with the Y-axis of the second binocular camera coordinate system at the second position based on the two posture information, obtaining an alignment rotation matrix and an alignment translation vector, and adjusting the generalized camera epipolar constraint model; A polynomial eigenvalue solution is performed on the adjusted generalized camera epipolar constraint model, and a relative pose obtained by the solution is output, where the relative pose includes the original rotation matrix and the original translation vector.

2. The relative pose calibration method of a binocular camera according to claim 1, characterized in that: The step of moving the binocular camera from a first position to a second position and determining an affine correspondence between two lenses in the binocular camera according to an image captured at the first position and an image captured at the second position comprises: The binocular camera is located at a first position at a first moment and moves to a second position at a second moment; Obtaining the image coordinates of each target point in the image collected at the first position, and obtaining the image coordinates of each target point in the image collected at the second position; Determine a first rotation matrix and a first translation vector for a first camera in the binocular camera to move from a first position to a second position, and determine a second rotation matrix and a second translation vector for a second camera in the binocular camera to move from a second position to a second position; The affine correspondence is obtained based on the image coordinates, the first rotation matrix, the first translation vector, the second rotation matrix, and the second translation vector.

3. The relative pose calibration method of a binocular camera according to claim 2, characterized in that: The determining of the Plücker line at the first position and the Plücker line at the second position based on the affine correspondence includes: Based on the image coordinates of each target point in the image collected at the first position, the first rotation matrix, and the intrinsic parameter matrix of the binocular camera, the Plücker line corresponding to the first position is determined, and the formula is applied: ； ； ； Based on the image coordinates of each of the target points in the image collected at the second position, the second rotation matrix, and the intrinsic parameter matrix of the binocular camera, the Plücker line corresponding to the second position is determined, and the formula is applied: ； ； ； ； Wherein, i indicates that the corresponding parameter is the parameter at the first moment, subscript j indicates that the corresponding parameter is the parameter at the second moment, and subscript Indicates that the corresponding parameters are the parameters of the first camera, the subscript Indicates that the corresponding parameters are the parameters of the second camera, is the Plücker line corresponding to the first position, is the first rotation matrix, represents the image coordinates of each target point in the image collected at the first position, is the Plücker line corresponding to the second position, is the second rotation matrix, represents the image coordinates of each target point in the image collected at the first position, 、 、 、 All are parameters, is the intrinsic parameter matrix, and f is the reciprocal of the focal length.

4. The relative pose calibration method of a binocular camera according to claim 3, characterized in that: The essential matrix is ​​constructed according to the original rotation matrix and the original translation vector of the binocular camera moving from the first position to the second position, and the formula is applied: ； The generalized camera epipolar constraint model is constructed by combining the affine correspondence, the Plücker line at the first position, and the Plücker line at the second position, and the formula is applied: ； in, is the essential matrix, is the original rotation matrix, is the original translation vector, is the antisymmetric matrix of the original translation vector t.

5. The relative pose calibration method of a binocular camera according to claim 4, characterized in that: The inertial measurement unit fixedly connected to the binocular camera is used to obtain the posture information at the first position and the posture information at the second position, respectively, including: Using an inertial measurement unit fixedly connected to the binocular camera, and assuming that the positions and connection relationship between the inertial measurement unit and the binocular camera have been calibrated, the pitch angle and roll angle corresponding to the first moment and the pitch angle and roll angle corresponding to the second moment are obtained respectively; The method aligns the Y axis of the first binocular camera coordinate system at the first position with the Y axis of the second binocular camera coordinate system at the second position based on the two posture information to obtain an alignment rotation matrix and an alignment translation vector, including: Estimating the first inertial system rotation matrix at the corresponding moment based on the pitch angle and roll angle at the first moment; estimating the second inertial system rotation matrix at the corresponding moment based on the pitch angle and roll angle at the second moment; Based on the first inertial system rotation matrix and the second inertial system rotation matrix, the first binocular camera coordinate system and the second binocular camera coordinate system are aligned to a preset vertical direction. The relationship between the alignment rotation matrix and the rotation angle between the first binocular camera coordinate system and the second binocular camera coordinate system after alignment is determined, and the formula is applied: ； ； ； Determine the relationship between the alignment translation vector and the original translation vector between the first and second binocular camera coordinate systems after alignment, and apply the formula: ； Get the adjusted generalized camera epipolar constraint model and apply the formula: ； ； in, is the first inertial system rotation matrix, is the second inertial system rotation matrix, , y is the intermediate parameter, As parameters, is the alignment translation vector, is the rotation angle around the Y axis when aligning the Y axis of the first binocular camera coordinate system at the first position with the Y axis of the second binocular camera coordinate system at the second position, is the alignment rotation matrix between the first binocular camera coordinate system and the second binocular camera coordinate system after alignment.

6. A relative pose calibration device for a binocular camera, characterized in that: include: an affine relationship determining unit, configured to move the binocular camera from a first position to a second position, and determine an affine correspondence between two lenses in the binocular camera based on an image captured at the first position and an image captured at the second position; a constraint model building unit, configured to determine a Plücker line at a first position and a Plücker line at a second position based on the affine correspondence; The constraint model construction unit is further configured to construct a generalized camera epipolar constraint model based on an original rotation matrix and an original translation vector of the binocular camera moving from a first position to a second position, in combination with the affine correspondence, the Plücker line at the first position, and the Plücker line at the second position; The constraint model building unit is further configured to obtain the posture information at the first position and the posture information at the second position respectively through an inertial measurement unit fixedly connected to the binocular camera; The constraint model construction unit is further configured to align the Y axis of the first binocular camera coordinate system at the first position with the Y axis of the second binocular camera coordinate system at the second position based on the two posture information, and adjust the generalized camera epipolar constraint model by aligning the rotation matrix and the translation vector; A computing unit is configured to solve polynomial eigenvalues ​​of the adjusted generalized camera epipolar constraint model and output a relative pose obtained by the solution, wherein the relative pose includes the original rotation matrix and the original translation vector.

7. The relative position calibration device of a binocular camera according to claim 6, characterized in that: The affine relationship determining unit is used to move the binocular camera from a first position to a second position, and determine the affine correspondence between two lenses in the binocular camera according to the image collected at the first position and the image collected at the second position, including: The binocular camera is located at a first position at a first moment and moves to a second position at a second moment; The affine relationship determination unit is used to obtain the image coordinates of each target point in the image collected at the first position, and obtain the image coordinates of each target point in the image collected at the second position; The affine relationship determination unit is further configured to determine a first rotation matrix and a first translation vector for a first camera in the binocular camera to move from a first position to a second position, and to determine a second rotation matrix and a second translation vector for a second camera in the binocular camera to move from a second position to a second position; The affine relationship determining unit is further configured to obtain the affine correspondence based on the image coordinates, the first rotation matrix, the first translation vector, the second rotation matrix, and the second translation vector.

8. The relative position calibration device of a binocular camera according to claim 7, characterized in that: The constraint model construction unit is used to determine the Plücker line at the first position and the Plücker line at the second position based on the affine correspondence, including: The constraint model construction unit determines the Plücker line corresponding to the first position based on the image coordinates of each target point in the image collected at the first position, the first rotation matrix, and the intrinsic parameter matrix of the binocular camera, and applies the formula: ； ； ； The constraint model construction unit determines the Plücker line corresponding to the second position based on the image coordinates of each target point in the image collected at the second position, the second rotation matrix, and the intrinsic parameter matrix of the binocular camera, and applies the formula: ； ； ； ； Wherein, i indicates that the corresponding parameter is the parameter at the first moment, subscript j indicates that the corresponding parameter is the parameter at the second moment, and subscript Indicates that the corresponding parameters are the parameters of the first camera, the subscript Indicates that the corresponding parameters are the parameters of the second camera, is the Plücker line corresponding to the first position, is the first rotation matrix, represents the image coordinates of each target point in the image collected at the first position, is the Plücker line corresponding to the second position, is the second rotation matrix, represents the image coordinates of each target point in the image collected at the first position, 、 、 、 All are parameters, is the intrinsic parameter matrix, and f is the reciprocal of the focal length.

9. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein: When the processor executes the program, the steps of the method according to any one of claims 1 to 5 are implemented.

10. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 5 are implemented.

Images