Binocular camera stereo calibration method, system and device

Abstract

The present invention provides a binocular camera stereo calibration method, system, and device. The binocular camera stereo calibration method includes the following steps: S1. Calibrate the left and right cameras of the binocular camera separately and output the intrinsic and extrinsic parameters of the left and right cameras; S2. Use the left and right cameras to synchronously capture left and right images; S3. Perform virtual parallel view stereo calibration on the left and right images based on the intrinsic and extrinsic parameters of the left and right cameras to align the left and right images in rows; S4. Calculate the image coordinate mapping matrix using the results of the virtual parallel view stereo calibration to generate the left and right calibrated images. The binocular camera stereo calibration method, system, and device provided by the present invention can achieve row alignment of the left and right images while retaining more of the original pixel information of the left and right images.

Description

Technical Field

[0001] The present invention relates to the field of camera calibration technology, and in particular to a binocular camera stereo calibration method, system and device. Background Art

[0002] With the development of computer vision technology, stereo vision sensors have become widely used in fields such as robotic navigation, industrial measurement, and virtual reality. Binocular cameras, as a common stereo vision sensor, can perform depth calculation and 3D reconstruction through stereo matching of left and right images. The epipolar constraints of the left and right images of a binocular camera can convert the stereo matching algorithm from 2D to 1D, accelerating 3D reconstruction of the left and right images. Stereo correction achieves row alignment by transforming the left and right images. Stereo matching only requires performing stereo matching within the same row, accelerating the matching algorithm while avoiding errors caused by epipolar constraint calculations. Therefore, stereo correction is often required before stereo matching of binocular images to achieve row alignment.

[0003] The currently mainstream stereo calibration method is the Bouguet algorithm. The Bouguet algorithm achieves row alignment by transforming the binocular camera's coordinate system into a parallel binocular model through two rotations. However, during the Bouguet algorithm's stereo calibration process, the image plane and the optical axis maintain a rigid structure, and their relative relationship remains unchanged, while the absolute position of the optical axis and the image changes. When the rotation angle is large, the field of view of the left and right cameras changes significantly, resulting in significant pixel loss. Summary of the Invention

[0004] The object of the present invention is to provide a binocular camera stereo correction method, system and device, which can retain more original pixel information of the left and right images while achieving left and right image line alignment.

[0005] In order to achieve the above object, as a first aspect of the present invention, a binocular camera stereo calibration method is provided, comprising the following steps:

[0006] S1. Calibrate the left camera and the right camera of the binocular camera respectively, and output the internal and external parameters of the left camera and the right camera;

[0007] S2. synchronously capture a left image and a right image using the left camera and the right camera;

[0008] S3. Performing virtual parallel view stereoscopic correction on the left image and the right image according to the internal and external parameters of the left camera and the right camera, so as to align the left image and the right image in rows, as follows:

[0009] S31, rotating the left image around the image center of its own left image coordinate system and rotating the right image around the image center of its own right image coordinate system, so that the left image coordinate system is parallel to the right image coordinate system;

[0010] S32, rotating the left image around the image center of the left image coordinate system and the right image around the image center of the right image coordinate system, so that the X axis of the left image coordinate system and the X axis of the right image coordinate system are parallel to the baseline of the binocular camera;

[0011] S33, translating the left image along the Y axis of the left image coordinate system and the right image along the Y axis of the right image coordinate system, so that the XZ plane of the left image coordinate system is coplanar with the XZ plane of the right image coordinate system;

[0012] S34, translating the left image along a left optical axis projection direction and translating the right image along a right optical axis projection direction, so that the left image coordinate system and the XY plane of the right image coordinate system are coplanar, where the left optical axis projection direction is the projection of the optical axis of the left camera on the XZ plane of the left image coordinate system, and the right optical axis projection direction is the projection of the optical axis of the right camera on the XZ plane of the right image coordinate system;

[0013] S4. Calculate the image coordinate mapping matrix using the result of the virtual parallel view plane stereoscopic correction to generate a left corrected image and a right corrected image, as follows:

[0014] S41, calculating an image coordinate mapping matrix according to the result of the virtual parallel view plane stereo correction;

[0015] S42, using the image coordinate mapping matrix to transform the coordinates of the left image and the coordinates of the right image into corresponding left corrected image coordinates and right corrected image coordinates;

[0016] S43 . Perform interpolation based on the left corrected image coordinates and the right corrected image coordinates to obtain a left corrected image and a right corrected image with rows aligned.

[0017] Optionally, after executing S2 and before executing S31, the binocular camera stereo calibration method further includes:

[0018] Distortion correction is performed on the left image according to the intrinsic parameters of the left camera and on the right image according to the intrinsic parameters of the right camera.

[0019] Optionally, in S31 and S32, the optical axes of the left image and the right image remain unchanged during the rotation process.

[0020] Optionally, in S33, the optical axes of the left image and the right image change as the images are translated, and in S34, the optical axes of the left image and the right image remain unchanged during the translation.

[0021] Optionally, in S33, the translation amount of the left image and the right image along the Y-axis can be freely controlled to ensure that the XZ planes of the left image coordinate system and the right image coordinate system are coplanar; in S34, the translation amount of the left image and the right image along the optical axis projection direction can be freely controlled to ensure that the XY planes of the left image coordinate system and the right image coordinate system are coplanar.

[0022] As a second aspect of the present invention, a binocular camera stereo correction system is provided, comprising:

[0023] An image acquisition module, comprising a left camera and a right camera, wherein the left camera and the right camera are used to synchronously acquire a left image and a right image;

[0024] A parameter calibration module, configured to calibrate the left camera and the right camera respectively, and output internal and external parameters of the left camera and the right camera;

[0025] a stereo correction module, configured to perform virtual parallel view stereo correction on the left image and the right image according to the intrinsic and extrinsic parameters of the left camera and the right camera, so as to correct the left image and the right image into row alignment;

[0026] An image mapping module is used to calculate an image coordinate mapping matrix using the result of the virtual parallel view plane stereo correction to generate a left corrected image and a right corrected image;

[0027] Wherein, the stereo correction module includes:

[0028] a first rotation unit, configured to rotate the left image around an image center of its own left image coordinate system and the right image around an image center of its own right image coordinate system, so that the left image coordinate system is parallel to the right image coordinate system;

[0029] a second rotation unit, configured to rotate the left image around the image center of the left image coordinate system and the right image around the image center of the right image coordinate system, so that the X axis of the left image coordinate system and the X axis of the right image coordinate system are parallel to the baseline of the binocular camera;

[0030] a first translation unit, configured to translate the left image along the Y axis of the left image coordinate system and the right image along the Y axis of the right image coordinate system, so that the XZ plane of the left image coordinate system is coplanar with the XZ plane of the right image coordinate system;

[0031] A second translation unit is used to translate the left image along the left optical axis projection direction and translate the right image along the right optical axis projection direction, so that the left image coordinate system is coplanar with the XY plane of the right image coordinate system, the left optical axis projection direction is the projection of the optical axis of the left camera on the XZ plane of the left image coordinate system, and the right optical axis projection direction is the projection of the optical axis of the right camera on the XZ plane of the right image coordinate system.

[0032] Optionally, the binocular camera stereo correction system further includes a distortion correction module for performing distortion correction on the left image according to the intrinsic parameters of the left camera and on the right image according to the intrinsic parameters of the right camera before stereo correction.

[0033] Optionally, the binocular camera stereo correction system further includes a depth calculation module for performing depth calculation on the left corrected image and the right corrected image using the binocular camera stereo correction method.

[0034] Optionally, the image acquisition module adopts a fixed binocular camera or a movable binocular camera.

[0035] Optionally, the parameter calibration module uses Zhang calibration or an improved algorithm thereof to output the internal and external parameters of the left camera and the right camera.

[0036] Optionally, when the image acquisition module adopts a zoom lens, the parameter calibration module can perform focal length calibration through the image and recalculate the intrinsic reference focal length.

[0037] Optionally, the parameter calibration module can recalculate the extrinsic parameters of the left camera and the right camera through image feature matching.

[0038] Optionally, the parameter calibration module can calculate the external parameters of the left camera and the right camera through an encoder or an inertial measurement device.

[0039] Optionally, the parameter calibration module can calculate the external parameters of the left camera and the right camera through a laser rangefinder or a TOF sensor.

[0040] Optionally, when the left camera and the right camera are in a standard convergence state, the rotation amounts of the extrinsic parameters of the left camera and the right camera are directly represented by left and right convergence angles.

[0041] Optionally, when the left camera and the right camera are in a standard convergence state, the parameter calibration module can obtain the left and right convergence angles of the left camera and the right camera through a convergence angle measurement element to calculate the external parameters of the left camera and the right camera.

[0042] As a third aspect of the present invention, a binocular camera stereo calibration device is provided, comprising:

[0043] Binocular camera, including left camera and right camera;

[0044] a calibration tool, configured to calibrate the left camera and the right camera respectively, and output internal and external parameters of the left camera and the right camera;

[0045] A motion control component, used to control the relative motion of the left camera and the right camera;

[0046] A motion measurement component, configured to calculate changes in extrinsic parameters of the left camera and the right camera;

[0047] A processor is configured to execute the binocular camera stereo method as described above.

[0048] Optionally, the image capturing time of the left camera and the right camera is synchronized or the capturing time of each camera has a timestamp.

[0049] Optionally, the motion measurement component is an encoder, an inertial element, a laser rangefinder, a TOF sensor and a convergence angle measurement element.

[0050] In the binocular camera stereo correction method, system and device provided by the present invention, more original pixel information of the left and right images can be retained while achieving left and right image row alignment. BRIEF DESCRIPTION OF THE DRAWINGS

[0051] Those skilled in the art will appreciate that the accompanying drawings are provided for a better understanding of the present invention and do not constitute any limitation on the scope of the present invention.

[0052] Figure 1 This is the overall flow chart of the binocular camera stereo calibration method provided in this embodiment;

[0053] Figure 2 A specific flow chart of stereo calibration for a binocular camera provided by this embodiment;

[0054] Figure 3 A schematic diagram of the coordinate system of the left and right cameras provided in this embodiment;

[0055] Figure 4 A schematic diagram of the coordinate system after rotation in S31 provided in this embodiment;

[0056] Figure 5 A schematic diagram of the coordinate system after rotation in S32 provided in this embodiment;

[0057] Figure 6A schematic diagram of the coordinate system after translation in S33 provided in this embodiment;

[0058] Figure 7 This is a diagram showing the changes in the XY viewing angle of the left and right images provided in this embodiment;

[0059] Figure 8 A schematic diagram of the coordinate system after translation in S34 provided in this embodiment;

[0060] Figure 9 This is a diagram showing the changes in the XZ viewing angles of the left and right images provided in this embodiment;

[0061] Figure 10 Schematic diagram of the depth calculation model for the left and right rectified images provided in this embodiment. DETAILED DESCRIPTION

[0062] In order to make the objects, advantages and features of the present invention more clear, the present invention is further described in detail below with reference to the accompanying drawings and specific embodiments. It should be noted that the drawings are in a very simplified form and are not in precise proportions. They are only used to conveniently and clearly assist in explaining the purpose of the embodiments of the present invention, and are not used to limit the conditions for the implementation of the present invention. Therefore, they have no technical significance. Any structural modification, change in proportional relationship or adjustment of size should still fall within the scope of the technical content disclosed by the present invention without affecting the efficacy and purpose of the present invention.

[0063] It should also be understood that, unless otherwise specified or indicated, the terms "first", "second", "third", etc. in the description are merely used to distinguish between the various components, elements, steps, etc. in the description, and are not used to indicate the logical relationship or sequential relationship between the various components, elements, steps, etc. Moreover, the terms "comprise", "include", or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, article, or device that includes a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article, or device. In the absence of further limitations, an element defined by the phrase "comprises a ..." does not exclude the presence of other identical elements in the process, method, article, or device that includes the element.

[0064] Figure 1 This is the overall flow chart of the binocular camera stereo calibration method provided in this embodiment; Figure 2 A specific flow chart of stereo calibration for a binocular camera provided by this embodiment; Figure 3 A schematic diagram of the coordinate system of the left and right cameras provided in this embodiment; Figure 4 A schematic diagram of the coordinate system after rotation in S31 provided in this embodiment; Figure 5 A schematic diagram of the coordinate system after rotation in S32 provided in this embodiment; Figure 6 A schematic diagram of the coordinate system after translation in S33 provided in this embodiment; Figure 7 This is a diagram showing the changes in the XY viewing angle of the left and right images provided in this embodiment; Figure 8 A schematic diagram of the coordinate system after translation in S34 provided in this embodiment; Figure 9 This is a diagram showing the changes in the XZ viewing angles of the left and right images provided in this embodiment; Figure 10 Schematic diagram of the depth calculation model for the left and right rectified images provided in this embodiment.

[0065] Please refer to Figure 1-Figure 2 , this embodiment provides a binocular camera stereo calibration method, comprising the following steps:

[0066] S1. Calibrate the left camera and the right camera of the binocular camera respectively, and output the internal and external parameters of the left camera and the right camera;

[0067] S2. synchronously capture a left image and a right image using the left camera and the right camera;

[0068] S3. Performing virtual parallel view stereoscopic correction on the left image and the right image according to the internal and external parameters of the left camera and the right camera, so as to align the left image and the right image in rows, as follows:

[0069] S31, rotating the left image around the image center of its own left image coordinate system and rotating the right image around the image center of its own right image coordinate system, so that the left image coordinate system is parallel to the right image coordinate system;

[0070] S32, rotating the left image around the image center of the left image coordinate system and the right image around the image center of the right image coordinate system, so that the X axis of the left image coordinate system and the X axis of the right image coordinate system are parallel to the baseline of the binocular camera;

[0071] S33, translating the left image along the Y axis of the left image coordinate system and the right image along the Y axis of the right image coordinate system, so that the XZ plane of the left image coordinate system is coplanar with the XZ plane of the right image coordinate system;

[0072] S34, translating the left image along a left optical axis projection direction and translating the right image along a right optical axis projection direction, so that the left image coordinate system and the XY plane of the right image coordinate system are coplanar, where the left optical axis projection direction is the projection of the optical axis of the left camera on the XZ plane of the left image coordinate system, and the right optical axis projection direction is the projection of the optical axis of the right camera on the XZ plane of the right image coordinate system;

[0073] S4. Calculate the image coordinate mapping matrix using the result of the virtual parallel view plane stereoscopic correction to generate a left corrected image and a right corrected image, as follows:

[0074] S41, calculating an image coordinate mapping matrix according to the result of the virtual parallel view plane stereo correction;

[0075] S42, using the image coordinate mapping matrix to transform the coordinates of the left image and the coordinates of the right image into corresponding left corrected image coordinates and right corrected image coordinates;

[0076] S43 . Perform interpolation based on the left corrected image coordinates and the right corrected image coordinates to obtain a left corrected image and a right corrected image with rows aligned.

[0077] First, execute steps S1-S2 to calibrate the left camera and the right camera of the binocular camera respectively, output the internal and external parameters of the left camera and the right camera, and use the left camera and the right camera to synchronously capture the left image and the right image. In this embodiment, the left camera and the right camera can capture continuous images, and the binocular camera includes a time synchronization module to ensure that the capture time of the two images is synchronized, or the shooting time of each camera has a timestamp. The binocular camera includes but is not limited to a fixed binocular camera and a movable binocular camera. The movable binocular camera may include a motion control component to control the movement of the left and right cameras, and may include an encoder, an inertial element, a convergence angle measurement element, a laser rangefinder, and a TOF sensor motion measurement component for measuring the external parameters of the left and right cameras.

[0078] The camera internal parameters include focal length f, principal point c x and c y And the distortion coefficient D, the camera extrinsic parameters include the rotation vector θ0 (or rotation matrix R0) and translation vector t0 between the left and right cameras. When the left and right cameras are in standard convergence state, the rotation vector of the extrinsic parameter can be used as the left and right convergence angle and Alternative.

[0079] In this embodiment, the initial values of the intrinsic parameters and extrinsic parameters of the left and right cameras can be calibrated and output using Zhang's calibration method or an improved algorithm thereof.

[0080] Preferably, when the image acquisition module adopts a zoom lens, the parameter calibration module can perform focal length calibration through the image and recalculate the intrinsic reference focal length f.

[0081] When the binocular camera is a fixed binocular camera, if the external parameters of the fixed binocular camera change due to vibration, the external parameter rotation vector and translation vector can be recalculated through image feature matching.

[0082] If the fixed binocular camera is in a standard convergence state, the extrinsic parameter rotation vector can be represented by the left and right camera convergence angles.

[0083] When the binocular camera is a movable binocular camera, during the movement of the movable binocular camera, the camera extrinsic parameter rotation vector and translation vector can be recalculated by matching feature points of the left and right images.

[0084] During the movement of the movable binocular camera, the rotation amount of each movement axis of the left and right cameras can be measured by an encoder or an inertial measurement device to obtain the extrinsic parameter rotation vectors of the left and right cameras.

[0085] During the movement of the movable binocular camera, the translation amount of the left and right cameras can be calculated by a laser rangefinder or a TOF sensor to obtain the extrinsic translation vectors of the left and right cameras.

[0086] During the movement of the movable binocular camera, the motion control component can maintain a standard convergence state, and the convergence angle of the left and right cameras can be obtained by the convergence angle measurement element as the extrinsic parameters of the left and right cameras. The left and right convergence angles can be converted into rotation vectors as the extrinsic parameter rotation vectors.

[0087] Then, step S3 is executed to perform virtual parallel view stereo correction on the left image and the right image according to the internal and external parameters of the left camera and the right camera, so as to align the left image and the right image in rows. S3 specifically includes the following steps:

[0088] S31, rotating the left image around the image center of its own left image coordinate system and rotating the right image around the image center of its own right image coordinate system, so that the left image coordinate system is parallel to the right image coordinate system;

[0089] S32, rotating the left image around the image center of the left image coordinate system and the right image around the image center of the right image coordinate system, so that the X axis of the left image coordinate system and the X axis of the right image coordinate system are parallel to the baseline of the binocular camera;

[0090] S33, translating the left image along the Y axis of the left image coordinate system and the right image along the Y axis of the right image coordinate system, so that the XZ plane of the left image coordinate system is coplanar with the XZ plane of the right image coordinate system;

[0091] S34. Translate the left image along the left optical axis projection direction and translate the right image along the right optical axis projection direction so that the left image coordinate system is coplanar with the XY plane of the right image coordinate system, the left optical axis projection direction is the projection of the optical axis of the left camera on the XZ plane of the left image coordinate system, and the right optical axis projection direction is the projection of the optical axis of the right camera on the XZ plane of the right image coordinate system.

[0092] Specific, combined Figure 3 , the camera coordinate systems of the left and right images are defined as O cl -X cl Y cl Z cl and O cr -X cr Y cr Z cr , the image coordinate systems of the left and right images are O il -X il Y il Z il and O ir -X ir Y ir Z ir , the baseline is O cl O cr , the left and right optical axes are O cl O il and O cr O ir .

[0093] First, execute S31, the image center of the left image coordinate system is the origin O of the left image coordinate system. il The image center of the right image coordinate system is the origin O of the right image coordinate system. ir , in the left and right camera coordinate system O cl -X cl Y cl Z cl and O cr -X cr Y cr Z cr The coordinates below are all [0 0 f] T , denoted as the center coordinates of the left and right images. The rotation vectors of the left and right images are θ0 / 2 and -θ0 / 2 respectively, and the corresponding rotation matrices are the first left rotation matrix and the first right rotation matrix

[0094] The coordinate system after S31 is as follows Figure 4 As shown, at this time, the left and right image coordinate systems O il -X il Y il Zil and O ir -X ir Y ir Z ir Parallel. The center coordinates of the left and right images are and They are recorded as the first coordinates of the left and right image centers respectively.

[0095] Preferably, after executing S2 and before executing S31, the binocular camera stereo calibration method further includes:

[0096] Distortion correction is performed on the left image according to the intrinsic parameters of the left camera and on the right image according to the intrinsic parameters of the right camera to obtain undistorted left and right images.

[0097] In the step S31, the left image and the right image are rotated along the optical axis O cl O il and O cr O ir Remain unchanged.

[0098] Then, the step S32 is executed to rotate the left and right images of S31 around the image center respectively so that the x-axes of the left and right image coordinate systems are parallel to the baseline.

[0099] The image center is the origin of the image coordinate system O il and O ir The rotation amount of the left and right images is the same, which is recorded as the second rotation matrix R2. in e2=[0 0 1] T ×e1, e3 = e1×e2, where t0 is the translation vector of the camera extrinsic parameter.

[0100] After step S32, the coordinate system is as follows Figure 5 As shown, at this time, the left and right image coordinate systems O il -X il Y il Z il and O ir -X ir Y ir Z ir X-axis O il -X il and O ir -X ir With baseline O cl O cr Parallel. The first center coordinates of the left and right images change to and Recorded as the second coordinates of the left and right image centers respectively.

[0101] In the step S32, the left image and the right image are rotated along the optical axis O cl O il and O cr O ir Remain unchanged.

[0102] Then, step S33 is executed to translate the left and right images of step S32 along the y-axis of the image coordinate system so that the XZ planes of the left and right image coordinate systems are coplanar. The y-axis of the image coordinate system is the X-axis of the left and right image coordinate systems. il Y il and O ir Y ir .

[0103] The translation vectors of the left and right images are and They are respectively recorded as the first left translation vector and the first right translation vector. The coordinates of the left and right image centers after translation are and Recorded as the third coordinate of the center of the left and right images.

[0104] After step S33, the coordinate system and XY viewing angle change as follows Figure 6-7 As shown, at this time, the left and right image coordinate systems O il -X il Y il Z il and O ir -X ir Y ir Z ir The XZ planes are coplanar.

[0105] Preferably, the translation amounts of the left image and the right image along the Y axis can be freely controlled, as long as the XZ planes of the left image coordinate system and the right image coordinate system are ensured to be coplanar.

[0106] In the step S33, the left and right optical axes O are shifted during the left and right image translation. cl O il and O cr O ir Changes as the image is translated.

[0107] Then, S34 is executed to translate the left and right images of S33 along the optical axis projection direction respectively so that the XY planes of the left and right image coordinate systems are coplanar.

[0108] The optical axis projection directions are respectively the left and right optical axes O cl O il and O cr O ir The projection vectors on the XZ plane of the left and right image coordinate systems after step S33 are and

[0109] Preferably, the translation of the left and right images along the optical axis projection direction can be freely controlled so that the z coordinates of the left and right image centers remain the same, thereby ensuring that the XY planes of the left and right image coordinate systems are coplanar. The z coordinate can be the average z value of the third coordinates of the left and right image centers, or the original focal length f, controlled by a parameter.

[0110] Set the z coordinate to f', and the left and right image centers are translated to the coordinates and Recorded as the fourth coordinate of the center of the left and right images.

[0111] The coordinate system and XZ viewing angle after step S34 are as follows Figure 8-9 As shown, at this time, the left and right image coordinate systems O il -X il Y il Z il and O ir -X ir Y ir Z ir The XY planes are coplanar.

[0112] In the step S34, the left and right optical axes O are shifted during the left and right image translation. cl O il and O cr O ir Remain unchanged.

[0113] After the above steps S31-S34, the row alignment of the left and right images is achieved. It should be understood that the order of the above steps S31-S34 is not limited and can be arbitrarily reversed, for example, the order of steps S31-S32 is changed, the order of steps S33-S34 is changed, steps S33-S34 are before steps S31-S32, etc. This application does not impose any restrictions on this, as long as the final state of the left and right images is as follows Figure 9 Just as shown.

[0114] Preferably, steps S31-S32 of the virtual parallel view stereo correction module can be combined into one step, and the left and right images are rotated once around the image center respectively. and Denoted as the left and right merge rotation matrix.

[0115] In particular, when the left and right cameras are in standard convergence state, the combined rotation matrix can be directly calculated by the convergence angle and in Defined by formula (1).

[0116]

[0117] In particular, when the left and right cameras are in a standard convergence state, step S33 can be omitted, and the XZ planes of the left and right images are coplanar.

[0118] In particular, when the left and right cameras are in the standard convergence state, the z coordinate of step S34 can also be directly set by the left and right convergence angles, for example or

[0119] Finally, step S4 is performed to calculate the image coordinate mapping matrix using the result of the virtual parallel view plane stereo correction to generate the left corrected image and the right corrected image, specifically as follows:

[0120] S41, calculating an image coordinate mapping matrix according to the result of the virtual parallel view plane stereo correction;

[0121] S42, using the image coordinate mapping matrix to transform the coordinates of the left image and the coordinates of the right image into corresponding left corrected image coordinates and right corrected image coordinates;

[0122] S43 . Perform interpolation based on the left corrected image coordinates and the right corrected image coordinates to obtain a left corrected image and a right corrected image with rows aligned.

[0123] Specifically, let the original intrinsic parameter matrix of the left image and the right image be K l and K r After the left image and the right image are corrected to be aligned in rows, the intrinsic parameter matrix of the left corrected image and the right corrected image is set to K′ l and K′ r , then the definition of the internal parameter matrix is shown in Formula 2-4.

[0124]

[0125]

[0126]

[0127] In particular, when the left and right cameras are in a standard convergence state, the intrinsic parameter matrix K′ of the left corrected image and the right corrected image is l and K′ r It can be directly expressed by the convergence angle as and form, in which It is expressed as formula 5.

[0128]

[0129] The image coordinate mapping matrices of the left image and the right image are respectively and

[0130] In particular, when the left and right cameras are in a standard convergence state, the image coordinate mapping matrix of the left image and the right image can be expressed by the convergence angle as and

[0131] Assume that the image coordinates of the left image and the right image are p l (u l , v l ) and p r (u r , v r ), the corresponding image coordinates of the left corrected image and the right corrected image are p′ l (u′ l , v′ l ) and p′ r (u′ r , v′ r ), Formula (6) obtains the left corrected image and the right corrected image by forward transforming the image coordinate mapping matrices of the left image and the right image.

[0132]

[0133] The image coordinates of the left corrected image and the right corrected image may be inversely mapped, as shown in equation (7).

[0134]

[0135] Then, interpolation is performed based on the image coordinates of the left corrected image and the right corrected image to obtain row-aligned left corrected image and right corrected image, and the interpolation method is not limited to nearest neighbor interpolation, linear interpolation, bilinear interpolation, etc.

[0136] After obtaining the left corrected image and the right corrected image, disparity calculation and depth calculation may be performed on the left corrected image and the right corrected image through row matching.

[0137] For details, please refer to Figure 10 , we can see that the optical axes of the left and right cameras are not parallel at this time. Perform row matching on the left and right corrected images to calculate the parallax d = u′ l -u′ r , the depth calculation formula is

[0138]

[0139] In particular, when the left and right cameras are in standard convergence state, the depth calculation formula is expressed by the convergence angle as follows:

[0140]

[0141] Wherein, B is the baseline length of the two cameras, and f′ is the focal length of the left rectified image and the right rectified image.

[0142] The present invention also provides a binocular camera stereo correction system, comprising:

[0143] An image acquisition module, comprising a left camera and a right camera, wherein the left camera and the right camera are used to synchronously acquire a left image and a right image;

[0144] A parameter calibration module, configured to calibrate the left camera and the right camera respectively, and output internal and external parameters of the left camera and the right camera;

[0145] A stereo correction module is configured to perform virtual parallel view stereo correction on the left image and the right image according to the internal and external parameters of the left camera and the right camera, so as to correct the left image and the right image into row alignment;

[0146] An image mapping module is used to calculate an image coordinate mapping matrix using the result of the virtual parallel view plane stereo correction to generate a left corrected image and a right corrected image;

[0147] Wherein, the stereo correction module includes:

[0148] a first rotation unit, configured to rotate the left image around an image center of its own left image coordinate system and the right image around an image center of its own right image coordinate system, so that the left image coordinate system is parallel to the right image coordinate system;

[0149] a second rotation unit, configured to rotate the left image around the image center of the left image coordinate system and the right image around the image center of the right image coordinate system, so that the X axis of the left image coordinate system and the X axis of the right image coordinate system are parallel to the baseline of the binocular camera;

[0150] a first translation unit, configured to translate the left image along the Y axis of the left image coordinate system and the right image along the Y axis of the right image coordinate system, so that the XZ plane of the left image coordinate system is coplanar with the XZ plane of the right image coordinate system;

[0151] A second translation unit is used to translate the left image along the left optical axis projection direction and translate the right image along the right optical axis projection direction, so that the left image coordinate system is coplanar with the XY plane of the right image coordinate system, the left optical axis projection direction is the projection of the optical axis of the left camera on the XZ plane of the left image coordinate system, and the right optical axis projection direction is the projection of the optical axis of the right camera on the XZ plane of the right image coordinate system.

[0152] In this embodiment, the image mapping module calculates the image coordinate mapping matrix using the result of the virtual parallel view plane stereo correction to generate the left corrected image and the right corrected image, as follows:

[0153] Calculating an image coordinate mapping matrix according to the result of the virtual parallel view plane stereo correction;

[0154] transforming the coordinates of the left image and the coordinates of the right image into corresponding left corrected image coordinates and right corrected image coordinates using the image coordinate mapping matrix;

[0155] Interpolation is performed based on the left corrected image coordinates and the right corrected image coordinates to obtain a left corrected image and a right corrected image that are line-aligned.

[0156] Preferably, the binocular camera stereo correction system further includes a distortion correction module for performing distortion correction on the left image according to the intrinsic parameters of the left camera and on the right image according to the intrinsic parameters of the right camera before stereo correction.

[0157] Preferably, the binocular camera stereo correction system further includes a depth calculation module for performing depth calculation on the left corrected image and the right corrected image using the binocular camera stereo correction method.

[0158] Preferably, the image acquisition module adopts a fixed binocular camera or a movable binocular camera.

[0159] Preferably, the parameter calibration module uses Zhang calibration or an improved algorithm thereof to output the internal and external parameters of the left camera and the right camera.

[0160] Preferably, when the image acquisition module adopts a zoom lens, the parameter calibration module can perform focal length calibration through the image and recalculate the intrinsic reference focal length.

[0161] Preferably, the parameter calibration module can recalculate the extrinsic parameters of the left camera and the right camera through image feature matching.

[0162] Preferably, the parameter calibration module can calculate the external parameters of the left camera and the right camera through an encoder or an inertial measurement device.

[0163] Preferably, the parameter calibration module can calculate the external parameters of the left camera and the right camera through a laser rangefinder or a TOF sensor.

[0164] Preferably, when the left camera and the right camera are in a standard convergence state, the rotation amounts of the extrinsic parameters of the left camera and the right camera are directly represented by left and right convergence angles.

[0165] Preferably, when the left camera and the right camera are in a standard convergence state, the parameter calibration module can obtain the left and right convergence angles of the left camera and the right camera through a convergence angle measurement element to calculate the external parameters of the left camera and the right camera.

[0166] Based on this, the present invention also provides a binocular camera stereo calibration device, comprising:

[0167] Binocular camera, including left camera and right camera;

[0168] a calibration tool, configured to calibrate the left camera and the right camera respectively, and output internal and external parameters of the left camera and the right camera;

[0169] A motion control component, used to control the relative motion of the left camera and the right camera;

[0170] A motion measurement component, configured to calculate changes in extrinsic parameters of the left camera and the right camera;

[0171] A processor is configured to execute the binocular camera stereo method as described above.

[0172] In this embodiment, the image capturing time of the left camera and the right camera is synchronized or the capturing time of each camera has a timestamp.

[0173] In this embodiment, the motion measurement component is an encoder, an inertial element, a laser rangefinder, a TOF sensor or a convergence angle measurement element.

[0174] In summary, the present invention provides a binocular camera stereo calibration method, system, and device. The binocular camera stereo calibration method includes the following steps: S1. Calibrate the left and right cameras of the binocular camera separately and output the intrinsic and extrinsic parameters of the left and right cameras; S2. Use the left and right cameras to synchronously capture left and right images; S3. Perform virtual parallel view stereo calibration on the left and right images based on the intrinsic and extrinsic parameters of the left and right cameras to align the left and right images in rows; S4. Calculate the image coordinate mapping matrix using the results of the virtual parallel view stereo calibration to generate the left and right calibrated images. The binocular camera stereo calibration method provided by the present invention can retain more original pixel information of the left and right images while achieving row alignment of the left and right images.

[0175] Furthermore, it should be recognized that although the present invention has been disclosed above with reference to preferred embodiments, the above embodiments are not intended to limit the present invention. Any person skilled in the art can utilize the above disclosed technical content to make many possible changes and modifications to the technical solution of the present invention, or modify it into equivalent embodiments with equivalent variations, without departing from the scope of the technical solution of the present invention. Therefore, any simple modifications, equivalent variations, and modifications made to the above embodiments based on the technical essence of the present invention, without departing from the content of the technical solution of the present invention, shall still fall within the scope of protection of the technical solution of the present invention.

Claims

1. A binocular camera stereo calibration method, characterized in that: The following steps are involved: S1. Calibrate the left camera and the right camera of the binocular camera respectively, and output the internal and external parameters of the left camera and the right camera; S2. synchronously capture a left image and a right image using the left camera and the right camera; S3. Performing virtual parallel view stereoscopic correction on the left image and the right image according to the internal and external parameters of the left camera and the right camera, so as to align the left image and the right image in rows, as follows: S31, rotating the left image around the image center of its own left image coordinate system and rotating the right image around the image center of its own right image coordinate system, so that the left image coordinate system is parallel to the right image coordinate system; S32, rotating the left image around the image center of the left image coordinate system and the right image around the image center of the right image coordinate system, so that the X axis of the left image coordinate system and the X axis of the right image coordinate system are parallel to the baseline of the binocular camera; S33, translating the left image along the Y axis of the left image coordinate system and the right image along the Y axis of the right image coordinate system, so that the XZ plane of the left image coordinate system is coplanar with the XZ plane of the right image coordinate system; S34, translating the left image along a left optical axis projection direction and translating the right image along a right optical axis projection direction, so that the left image coordinate system and the XY plane of the right image coordinate system are coplanar, where the left optical axis projection direction is the projection of the optical axis of the left camera on the XZ plane of the left image coordinate system, and the right optical axis projection direction is the projection of the optical axis of the right camera on the XZ plane of the right image coordinate system; S4. Calculate the image coordinate mapping matrix using the result of the virtual parallel view plane stereoscopic correction to generate a left corrected image and a right corrected image, as follows: S41, calculating an image coordinate mapping matrix according to the result of the virtual parallel view plane stereo correction; S42, using the image coordinate mapping matrix to transform the coordinates of the left image and the coordinates of the right image into corresponding left corrected image coordinates and right corrected image coordinates; S43 . Perform interpolation based on the left corrected image coordinates and the right corrected image coordinates to obtain a left corrected image and a right corrected image with rows aligned.

2. The binocular camera stereo calibration method according to claim 1, wherein: After executing S2 and before executing S31, the binocular camera stereo calibration method further includes: Distortion correction is performed on the left image according to the intrinsic parameters of the left camera and on the right image according to the intrinsic parameters of the right camera.

3. The binocular camera stereo calibration method according to claim 1, wherein: In S31 and S32, the optical axes of the left image and the right image remain unchanged during the rotation process.

4. The binocular camera stereo calibration method according to claim 1, wherein: In S33 , the optical axes of the left image and the right image change as the images are translated. In S34 , the optical axes of the left image and the right image remain unchanged during the translation.

5. The binocular camera stereo calibration method according to claim 1, wherein: In S33, the translation amount of the left image and the right image along the Y axis can be freely controlled to ensure that the XZ planes of the left image coordinate system and the right image coordinate system are coplanar; in S34, the translation amount of the left image and the right image along the optical axis projection direction can be freely controlled to ensure that the XY planes of the left image coordinate system and the right image coordinate system are coplanar.

6. A binocular camera stereo calibration system, characterized in that: include: An image acquisition module, comprising a left camera and a right camera, wherein the left camera and the right camera are used to synchronously acquire a left image and a right image; A parameter calibration module, configured to calibrate the left camera and the right camera respectively, and output internal and external parameters of the left camera and the right camera; a stereo correction module, configured to perform virtual parallel view stereo correction on the left image and the right image according to the intrinsic and extrinsic parameters of the left camera and the right camera, so as to correct the left image and the right image into row alignment; An image mapping module is used to calculate an image coordinate mapping matrix using the result of the virtual parallel view plane stereo correction to generate a left corrected image and a right corrected image; Wherein, the stereo correction module includes: a first rotation unit, configured to rotate the left image around an image center of its own left image coordinate system and the right image around an image center of its own right image coordinate system, so that the left image coordinate system is parallel to the right image coordinate system; a second rotation unit, configured to rotate the left image around the image center of the left image coordinate system and the right image around the image center of the right image coordinate system, so that the X axis of the left image coordinate system and the X axis of the right image coordinate system are parallel to the baseline of the binocular camera; a first translation unit, configured to translate the left image along the Y axis of the left image coordinate system and the right image along the Y axis of the right image coordinate system, so that the XZ plane of the left image coordinate system is coplanar with the XZ plane of the right image coordinate system; A second translation unit is used to translate the left image along the left optical axis projection direction and translate the right image along the right optical axis projection direction, so that the left image coordinate system is coplanar with the XY plane of the right image coordinate system, the left optical axis projection direction is the projection of the optical axis of the left camera on the XZ plane of the left image coordinate system, and the right optical axis projection direction is the projection of the optical axis of the right camera on the XZ plane of the right image coordinate system.

7. The binocular camera stereo calibration system according to claim 6, wherein: The binocular camera stereo correction system also includes a distortion correction module for performing distortion correction on the left image according to the intrinsic parameters of the left camera and on the right image according to the intrinsic parameters of the right camera before stereo correction.

8. The binocular camera stereo calibration system according to claim 6, wherein: The binocular camera stereo correction system also includes a depth calculation module, which uses the binocular camera stereo correction method to perform depth calculation on the left corrected image and the right corrected image.

9. The binocular camera stereo correction system according to claim 6, wherein: The image acquisition module adopts a fixed binocular camera or a movable binocular camera.

10. The binocular camera stereo calibration system according to claim 6, wherein: The parameter calibration module uses Zhang's calibration or an improved algorithm thereof to output the internal and external parameters of the left camera and the right camera.

11. The binocular camera stereo calibration system according to claim 6, wherein: When the image acquisition module adopts a zoom lens, the parameter calibration module can perform focal length calibration through the image and recalculate the intrinsic reference focal length.

12. The binocular camera stereo calibration system according to claim 6, wherein: The parameter calibration module can recalculate the extrinsic parameters of the left camera and the right camera through image feature matching.

13. The binocular camera stereo calibration system according to claim 6, wherein: The parameter calibration module can calculate the external parameters of the left camera and the right camera through an encoder or an inertial measurement device.

14. The binocular camera stereo calibration system according to claim 6, wherein: The parameter calibration module can calculate the external parameters of the left camera and the right camera through a laser rangefinder or a TOF sensor.

15. The binocular camera stereo calibration system according to claim 6, wherein: When the left camera and the right camera are in a standard convergence state, the rotation amounts of the extrinsic parameters of the left camera and the right camera are directly represented by left and right convergence angles.

16. The binocular camera stereo calibration system according to claim 6, wherein: When the left camera and the right camera are in a standard convergence state, the parameter calibration module can obtain the left and right convergence angles of the left camera and the right camera through a convergence angle measurement element to calculate the extrinsic parameters of the left camera and the right camera.

17. A binocular camera stereo calibration device, characterized in that: include: Binocular camera, including left camera and right camera; A calibration tool, configured to calibrate the left camera and the right camera respectively, and output internal and external parameters of the left camera and the right camera; A motion control component, used to control the relative motion of the left camera and the right camera; A motion measurement component, configured to calculate changes in extrinsic parameters of the left camera and the right camera; A processor configured to execute the binocular camera stereo method according to any one of claims 1 to 5.

18. The binocular camera stereo calibration device according to claim 17, wherein: The image capturing times of the left camera and the right camera are synchronized or the capturing times of the cameras have timestamps.

19. The binocular camera stereo calibration device according to claim 17, wherein: The motion measurement component is an encoder, an inertial element, a laser rangefinder, a TOF sensor or a convergence angle measurement element.

Images