Binocular Camera Calibration Method, Device, Equipment and Medium

Abstract

An embodiment of the present invention discloses a binocular camera calibration method, device, equipment and medium. The method obtains calibration wall image pairs corresponding to a to-be-calibrated binocular camera at a first distance, a second distance and a third distance respectively, so as to obtain image pairs containing a plurality of calibration plates at different distances, and then determines the corner image coordinates and corner three-dimensional coordinates of the calibration plate corners in each calibration wall image pair. Based on the corner image coordinates and corner three-dimensional coordinates of the calibration plate corners, the external camera parameters and internal camera parameters corresponding to the to-be-calibrated binocular camera are determined. By using the calibration wall image pairs collected once at each distance, image pairs containing calibration plates in various different poses are obtained at one time, and then the internal and external parameters of the binocular camera are determined through the image pairs collected once at each distance. This method does not need to perform image acquisition separately and multiple times for calibration plates in various different poses, improving the calibration efficiency and calibration accuracy of the binocular camera.

Description

Technical Field

[0001] Embodiments of the present invention relate to the technical field of camera calibration, and in particular, to a binocular camera calibration method, device, equipment and medium. Background Art

[0002] For the calibration of the binocular camera of a navigator, the commonly used method at present is the Zhang Zhengyou calibration method (see "A flexible new technique for camera calibration"), which uses a checkerboard plane calibration board, places the checkerboard in different poses in sequence, and acquires images through the binocular camera at different poses respectively, calculates the homography matrix between the corner coordinates in the image and the corner coordinates in the three-dimensional space at each pose, and then calculates the camera parameters by using multiple groups of homography matrices.

[0003] However, the Zhang Zhengyou calibration method needs to collect images at different poses of the calibration board multiple times, which makes the actual operation time-consuming, and there are differences in each calibration board image each time, resulting in low final calibration efficiency and low calibration result repeatability. Summary of the Invention

[0004] Embodiments of the present invention provide a binocular camera calibration method, device, equipment and medium to improve the calibration efficiency and calibration accuracy of the binocular camera.

[0005] In a first aspect, embodiments of the present invention provide a binocular camera calibration method, and the method includes:

[0006] Determine calibration wall image pairs corresponding to a first distance, a second distance, and a third distance of a binocular camera to be calibrated respectively, where the calibration wall includes at least three calibration boards;

[0007] Determine the corner image coordinates of the calibration board corners of each calibration board in each of the calibration wall image pairs;

[0008] Determine the corner three-dimensional coordinates of each of the calibration board corners, and determine the external camera parameters and internal camera parameters corresponding to the binocular camera to be calibrated based on the corner image coordinates and the corner three-dimensional coordinates.

[0009] Optionally, the calibration wall includes a reference calibration board and at least two reference calibration boards, and the method further includes:

[0010] Convert the calibration board corners other than the calibration board corners on the reference calibration board in the calibration wall image pair corresponding to the first distance to the three-dimensional coordinate system of the reference calibration board in the calibration wall image pair corresponding to the first distance, to obtain the corner three-dimensional reference coordinates of each of the calibration board corners;

[0011] Determine the corner reprojection image coordinates corresponding to the three-dimensional reference coordinates of the corner points based on the external camera parameters and the internal camera parameters;

[0012] Calculate the reprojection residuals based on the corner image coordinates and the corner reprojection image coordinates, and optimize the external camera parameters and / or the internal camera parameters according to the calculation results of the reprojection residuals.

[0013] Optionally, the converting other calibration plate corner points except the calibration plate corner points on the reference calibration plate in the calibration wall image pair corresponding to the first distance to the three-dimensional coordinate system of the reference calibration plate in the calibration wall image pair corresponding to the first distance to obtain the three-dimensional reference coordinates of each calibration plate corner point includes:

[0014] Determine the position conversion relationship between the reference calibration plate and each reference calibration plate;

[0015] Based on the position conversion relationship, determine the three-dimensional reference coordinates of the corner points of other calibration plate corner points except the calibration plate corner points on the reference calibration plate in the calibration wall image pair corresponding to the first distance, which are converted to the three-dimensional coordinate system of the reference calibration plate in the calibration wall image pair corresponding to the first distance.

[0016] Optionally, the determining the three-dimensional reference coordinates of the corner points of other calibration plate corner points except the calibration plate corner points on the reference calibration plate in the calibration wall image pair corresponding to the first distance, which are converted to the three-dimensional coordinate system of the reference calibration plate in the calibration wall image pair corresponding to the first distance includes:

[0017] Determine the measured working distance spacing and each deviation angle corresponding to the binocular camera to be calibrated;

[0018] Based on the measured working distance spacing, each deviation angle and the position conversion relationship, determine the three-dimensional reference coordinates of the corner points of other calibration plate corner points except the calibration plate corner points on the reference calibration plate in the calibration wall image pair corresponding to the first distance, which are converted to the three-dimensional coordinate system of the reference calibration plate in the calibration wall image pair corresponding to the first distance;

[0019] Correspondingly, the optimizing the external camera parameters and / or the internal camera parameters according to the calculation results of the reprojection residuals includes:

[0020] Optimize at least one of the external camera parameters, the internal camera parameters, the measured working distance spacing, each deviation angle and the position conversion relationship according to the calculation results of the reprojection residuals.

[0021] Optionally, the determining the external camera parameters and the internal camera parameters corresponding to the binocular camera to be calibrated based on the corner image coordinates and the three-dimensional corner coordinates includes:

[0022] Determine the projection matrix of the binocular camera to be calibrated based on the corner image coordinates and the three-dimensional coordinates of the corners;

[0023] Calculate the external camera parameters and internal camera parameters corresponding to the binocular camera to be calibrated based on the projection matrix.

[0024] Optionally, the determining the three-dimensional coordinates of the corner points of each calibration plate includes:

[0025] Obtain the relative three-dimensional coordinate relationship between the corner points of each calibration plate in each calibration plate, and determine the three-dimensional coordinates of the corner points of each calibration plate in the calibration wall image pair corresponding to the first distance based on the relative three-dimensional coordinate relationship;

[0026] Based on the three-dimensional coordinates of the corner points of each calibration plate in the calibration wall image pair corresponding to the first distance, respectively determine the three-dimensional coordinates of the corner points of each calibration plate in the calibration wall image pair corresponding to the second distance and the calibration wall image pair corresponding to the third distance.

[0027] Optionally, the determining the corner image coordinates of the calibration plate corner points of each calibration wall image pair includes:

[0028] Obtain the corner image coordinates of the calibration plate corner points of each calibration plate in each calibration wall image pair;

[0029] Perform clustering processing on each corner image coordinate, and respectively determine the binocular calibration plate pairing results of each calibration wall image pair based on the clustering processing results;

[0030] Update the corner image coordinates of each calibration plate corner point based on the binocular calibration plate pairing results.

[0031] In a second aspect, an embodiment of the present invention further provides a binocular camera calibration device, and the device includes:

[0032] An image determination module, configured to respectively determine calibration wall image pairs corresponding to a first distance, a second distance, and a third distance of the binocular camera to be calibrated, where the calibration wall includes at least three calibration plates;

[0033] An image coordinate determination module, configured to determine the corner image coordinates of the calibration plate corner points of each calibration plate in each calibration wall image pair;

[0034] An internal and external parameter determination module, configured to determine the three-dimensional coordinates of the corner points of each calibration plate, and determine the external camera parameters and internal camera parameters corresponding to the binocular camera to be calibrated based on the corner image coordinates and the three-dimensional coordinates of the corners.

[0035] In a third aspect, an embodiment of the present invention further provides an electronic device, and the electronic device includes:

[0036] One or more processors;

[0037] A storage device for storing one or more programs,

[0038] When the one or more programs are executed by the one or more processors, the one or more processors implement the binocular camera calibration method provided in any embodiment of the present invention.

[0039] In a fourth aspect, an embodiment of the present invention further provides a computer-readable storage medium, on which a computer program is stored, and when the program is executed by a processor, the binocular camera calibration method provided in any embodiment of the present invention is implemented.

[0040] The embodiments in the above-mentioned invention have the following advantages or beneficial effects:

[0041] By obtaining the calibration wall image pairs corresponding to the binocular camera to be calibrated at the first distance, the second distance, and the third distance respectively, to obtain the image pairs containing at least three calibration plates at different distances, and then determining the corner image coordinates of the calibration plate corners in each calibration wall image pair, and determining the three-dimensional coordinates of the calibration plate corners, based on the corner image coordinates and the three-dimensional coordinates of the calibration plate corners to determine the external camera parameters and the internal camera parameters corresponding to the binocular camera to be calibrated, it is realized that through the calibration wall image pairs collected once at each distance, the image pairs containing calibration plates in different poses are obtained at one time, and then the internal and external parameters of the binocular camera are determined through the image pairs collected once at each distance. This method does not need to collect images multiple times for each calibration plate in different poses respectively, improving the calibration efficiency and calibration accuracy of the binocular camera. BRIEF DESCRIPTION OF THE DRAWINGS

[0042] In order to more clearly illustrate the technical solutions of the exemplary embodiments of the present invention, the following briefly introduces the drawings required for describing the embodiments. Obviously, the introduced drawings are only the drawings of a part of the embodiments to be described in the present invention, rather than all the drawings. For those of ordinary skill in the art, other drawings can be obtained according to these drawings without creative efforts.

[0043] Figure 1A It is a schematic flowchart of a binocular camera calibration method provided in Embodiment 1 of the present invention;

[0044] Figure 1B It is a schematic diagram of the acquisition of a calibration wall image pair provided in Embodiment 1 of the present invention;

[0045] Figure 2 It is a schematic flowchart of a binocular camera calibration method provided in Embodiment 2 of the present invention;

[0046] Figure 3 It is a schematic flowchart of a binocular camera calibration method provided in the third embodiment of the present invention;

[0047] Figure 4 It is a schematic structural diagram of a binocular camera calibration device provided in the fourth embodiment of the present invention;

[0048] Figure 5 It is a schematic structural diagram of an electronic device provided in the fifth embodiment of the present invention. Specific Embodiments

[0049] The present invention will be further described in detail below with reference to the accompanying drawings and embodiments. It can be understood that the specific embodiments described herein are only used to explain the present invention, rather than limiting the present invention. Additionally, it should be noted that for the sake of description, only parts related to the present invention are shown in the accompanying drawings rather than all structures.

[0050] Embodiment 1

[0051] Figure 1A It is a schematic flowchart of a binocular camera calibration method provided in the first embodiment of the present invention. This embodiment is applicable to the situation of calibrating a binocular camera, especially applicable to the situation of calibrating a binocular camera according to a pre-built calibration wall. This method can be executed by a binocular camera calibration device, which can be implemented by hardware and / or software. The method specifically includes the following steps:

[0052] S110. Respectively determine calibration wall image pairs corresponding to the first distance, the second distance, and the third distance of the binocular camera to be calibrated, where the calibration wall includes at least three calibration plates.

[0053] In this embodiment, the first distance, the second distance, and the third distance can be the working distances between the binocular camera to be calibrated and the calibration wall set in advance. Optionally, the first distance is less than the second distance, and the second distance is less than the third distance; the specific values of the first distance, the second distance, and the third distance can be set according to actual needs. Exemplarily, the first distance can be the minimum working distance of the binocular camera to be calibrated, the third distance can be the maximum working distance of the binocular camera to be calibrated, and the second distance can be the median of the first distance and the third distance.

[0054] Specifically, in this embodiment, calibration wall image pairs are collected by the binocular camera to be calibrated at the first distance, the second distance, and the third distance respectively. That is, calibration wall image pairs are collected at three distances: near, medium, and far. Among them, the calibration wall image pair can be an image pair taken by the binocular camera to be calibrated for the calibration wall, which includes a left-eye image and a right-eye image. It should be noted that in this embodiment, the distance between the binocular camera and the calibration wall can be changed by controlling the movement of the binocular camera, so that the three-dimensional coordinate systems of the calibration wall image pairs at the first distance, the second distance, and the third distance are changed; that is, the movement of the binocular camera is regarded as the movement of the calibration wall.

[0055] Among them, the calibration wall can be configured according to the field of view requirements of the binocular camera to be calibrated, and it includes at least three calibration plates. It should be noted that a calibration wall containing calibration plates with multiple different angular postures can be built according to the field of view requirements. Optionally, the number of calibration plates can be three, four, or five, and each calibration plate can be evenly distributed within the full field of view of the binocular camera to be calibrated. Exemplarily, as Figure 1B shown, a schematic diagram of collecting calibration wall image pairs is presented. This calibration wall includes five calibration plates, each calibration plate has a different pose, and each calibration plate can be set at a certain interval to cover the field of view of the binocular camera to be calibrated. The middle calibration plate is parallel to the calibration wall plane and can be used as a reference calibration plate. The position and pose of the remaining calibration plates relative to the reference calibration plate always remain fixed, and there is a certain angle between the remaining calibration plates and the calibration wall plane.

[0056] It should be noted that in this embodiment, a linear guide rail can be built in advance within the corresponding interval according to the working distance range of the binocular camera to be calibrated. The direction of the guide rail is basically perpendicular to the calibration wall plane. The linear guide rail is equipped with a laser detection scale for later high-precision translational movement of the binocular camera to be calibrated along the guide rail direction. That is, in this embodiment, the binocular camera to be calibrated can be moved to a position spaced from the calibration wall by the first distance, the second distance, and the third distance respectively through the laser detection scale, so as to collect calibration wall image pairs at the first distance, the second distance, and the second distance respectively; as Figure 1B shown. Of course, the position of the calibration wall remains unchanged during this process. By setting the laser detection scale, the accuracy of the first distance, the second distance, and the third distance can be improved, and then the accuracy of the collected calibration wall image pairs can be improved. Further, the accuracy of camera calibration can be improved.

[0057] In the Zhang-Zhengyou calibration method in the prior art, it is necessary to separately collect calibration board images with different poses multiple times. For example, when setting up a calibration board with 9 poses, the Zhang-Zhengyou calibration method needs to collect 9 calibration board images, resulting in low calibration efficiency. Moreover, the repeatability of the calibration results of the images collected multiple times is low, and problems such as the calibration results being prone to falling into local minima may occur, leading to inaccurate calibration parameters and reducing the calibration accuracy. Compared with the Zhang-Zhengyou calibration method in the prior art, this method can collect a calibration wall image once at three distances: near, medium, and far, and can directly obtain a calibration wall image containing calibration boards with various poses in one collection. Continuing with the above example, when setting up a calibration wall with a calibration board containing 9 poses, this method only needs to collect 3 times (the first distance, the second distance, and the third distance) of calibration wall image pairs to obtain images of calibration boards with various poses, reducing the number of times of collecting calibration board images. Moreover, through the known displacement relationship in the three collected calibration wall images, binocular camera calibration is directly performed, improving the calibration accuracy.

[0058] S120. Determine the corner image coordinates of the calibration board corners of each calibration board in each of the calibration wall image pairs.

[0059] Among them, each image in the calibration wall image pair includes at least three calibration boards. For example, both the left-eye image and the right-eye image in the calibration wall image pair include 5 calibration boards. Specifically, each calibration board includes multiple calibration board corners. In this embodiment, it is necessary to determine the coordinates of each calibration board corner of each calibration board in each calibration wall image pair in the image, that is, the corner image coordinates. Specifically, the pixel coordinates of the calibration board corners in the calibration wall image pair can be extracted to obtain the corner image coordinates of the calibration board corners.

[0060] In an alternative embodiment, the determining the corner image coordinates of the calibration board corners of each calibration board in each of the calibration wall image pairs may be: obtaining the corner image coordinates of the calibration board corners of each calibration board in each of the calibration wall image pairs; performing clustering processing on each of the corner image coordinates, and respectively determining the binocular calibration board pairing results of each of the calibration wall image pairs based on the clustering processing results; updating the corner image coordinates of each of the calibration board corners based on the binocular calibration board pairing results. That is, extract the coordinates of all calibration board corners in each calibration wall image pair, distinguish each calibration board corner through a clustering method, and then identify the calibration boards in each calibration wall image pair according to the relative position relationship between the calibration boards. Determine the calibration board in the left-eye image of the calibration wall image pair and the calibration board at the corresponding position in the right-eye image as a pair, and use it as the binocular calibration board pairing result. Generate the corner image coordinates corresponding one by one between the left-eye image and the right-eye image based on the binocular calibration board pairing result.

[0061] Alternatively, it can also be: Obtain the rough corner coordinates of each calibration board corner in each of the calibration wall image pairs; perform clustering processing on the rough corner coordinates, and respectively determine the binocular calibration board pairing results of each of the calibration wall image pairs based on the clustering processing results; based on the binocular calibration board pairing results, determine the corner image coordinates of each calibration board corner in each of the calibration wall image pairs. That is, roughly extract the coordinates of all calibration board corners in each calibration wall image pair, distinguish each calibration board corner through a clustering method, and then identify the calibration boards in each calibration wall image pair according to the relative position relationship between the calibration boards. Determine the calibration board in the left-eye image of the calibration wall image pair and the calibration board in the right-eye image at the corresponding position as a pair, which is used as the binocular calibration board pairing result. Then, perform sub-pixel extraction on the binocular calibration board pairing result to obtain the sub-pixel coordinates of each calibration board corner, and use this sub-pixel coordinate as the corner image coordinate.

[0062] Taking a calibration wall containing 5 calibration boards as an example, according to the calibration wall image pairs corresponding to the first distance, 5 binocular calibration board pairing results can be paired; according to the calibration wall image pairs corresponding to the second distance, 5 binocular calibration board pairing results can be paired; according to the calibration wall image pairs corresponding to the third distance, 5 binocular calibration board pairing results can be paired; the sets of corner image coordinates of each calibration board corner in the binocular calibration board pairing results can be respectively denoted as {uv l} and {uv r}. Through this optional implementation manner, the accurate determination of the corner image coordinates of the calibration board corners can be realized, and then the calibration accuracy of the binocular camera can be improved.

[0063] S130. Determine the corner three-dimensional coordinates of each of the calibration board corners, and based on the corner image coordinates and the corner three-dimensional coordinates, determine the camera extrinsic parameters and camera intrinsic parameters corresponding to the binocular camera to be calibrated.

[0064] Among them, the corner three-dimensional coordinates can be the actual coordinates of the calibration board corners in the calibration wall in three-dimensional space. In this embodiment, the corner three-dimensional coordinates can be in the world coordinate system or in the three-dimensional coordinate system of the reference calibration board in the calibration wall image pair at the first distance.

[0065] In one implementation manner, the corner three-dimensional coordinates of the calibration board corners in the calibration wall can be obtained by measurement. In another implementation manner, the relative relationship of the three-dimensional coordinates of each calibration board corner can also be measured, and based on this three-dimensional coordinate relative relationship, the corner three-dimensional coordinates of each calibration board corner are determined.

[0066] For example, optionally, determining the three-dimensional coordinates of each calibration board corner point includes: obtaining the relative three-dimensional coordinate relationship between the calibration board corner points in each calibration board, and determining the three-dimensional coordinates of each calibration board corner point in the calibration wall image pair corresponding to the first distance based on the relative three-dimensional coordinate relationship; based on the three-dimensional coordinates of each calibration board corner point in the calibration wall image pair corresponding to the first distance, respectively determining the three-dimensional coordinates of each calibration board corner point in the calibration wall image pair corresponding to the second distance and the calibration wall image pair corresponding to the third distance.

[0067] In the above optional embodiment, the relative three-dimensional coordinate relationship between the calibration board corner points in the calibration board can be determined by a high-precision measuring ruler. Further, the three-dimensional coordinates of each calibration board corner point in the calibration wall image pair corresponding to the first distance can be determined based on the relative three-dimensional coordinate relationship. Since the three-dimensional coordinates of the calibration board corner points at the same position only have a change value in the Z-axis direction at the first distance, the second distance, and the third distance, the calibration board corner points at the second distance can be determined by the calibration board corner points at the first distance and the distance between the first distance and the second distance; and the calibration board corner points at the third distance can be determined by the calibration board corner points at the first distance and the distance between the first distance and the third distance. Exemplarily, if the set of the three-dimensional coordinates of each calibration board corner point in the calibration wall image pair corresponding to the first distance is {P0}, then the set of the three-dimensional coordinates of each calibration board corner point in the calibration wall image pair corresponding to the second distance is {P0 + (0, 0, d) T}, and the set of the three-dimensional coordinates of each calibration board corner point in the calibration wall image pair corresponding to the third distance is {P0 + (0, 0, 2d) T}, where d is the distance between the first distance and the second distance, and 2d is the distance between the first distance and the third distance; the three-dimensional coordinates of all calibration board corner points can be arranged in order and integrated as {P all}, {P all0} = {P0, P0 + (0, 0, d) T , P0(0, 0, 2d) T}. By this method, the rapid determination of the three-dimensional coordinates of the calibration board corner points can be realized, and the calibration efficiency of the binocular camera is improved.

[0068] Further, after determining the three-dimensional coordinates of the corner points of each calibration board in this embodiment, the projection matrix of the binocular camera to be calibrated can be calculated based on the image coordinates and three-dimensional coordinates of the corner points of the calibration board. Exemplarily, determining the external camera parameters and internal camera parameters corresponding to the binocular camera to be calibrated based on the corner image coordinates and the three-dimensional coordinates of the corner points includes: determining the projection matrix of the binocular camera to be calibrated based on the corner image coordinates and the three-dimensional coordinates of the corner points; calculating the external camera parameters and internal camera parameters corresponding to the binocular camera to be calibrated based on the projection matrix.

[0069] Among them, taking the left-eye camera as an example, the corner image coordinates (u, v) of a certain point in the set of corner image coordinates {uv l} can be taken respectively, and the three-dimensional coordinates (X, Y, Z) of the corresponding point in the set of three-dimensional coordinates of the corner points. The two can be calculated through the projection matrix. As shown in the following formula, based on the corner image coordinates and three-dimensional coordinates of all observed calibration board corner points, the projection matrix M can be calculated using the least squares method.

[0070]

[0071] Calculating the external camera parameters and internal camera parameters corresponding to the binocular camera to be calibrated based on the projection matrix can be: performing orthogonal triangular decomposition on the first three columns of the projection matrix to obtain the internal camera parameters and rotation matrix of the binocular camera to be calibrated; calculating the translation vector according to the data of the fourth column of the projection matrix; determining the external camera parameters of the binocular camera to be calibrated based on the rotation matrix and the translation vector. The rotation matrix and the translation vector are the transformation relationship between the binocular camera to be calibrated and the reference calibration board, and can be combined into the external camera parameters. As shown in the following formula:

[0072]

[0073] Among them, K represents the internal camera parameters, R cn is the rotation matrix, t cb is the translation vector, [R cb t cb represents the internal camera parameters, and M is the projection matrix. The initial value of the distortion coefficient of the binocular camera to be calibrated is defaulted to 0. Performing the above operations on the binocular camera to be calibrated can calculate the initial values of all internal and external parameters of the binocular camera to be calibrated.

[0074] It should be noted that the binocular camera to be calibrated in this embodiment includes a left-eye camera and a right-eye camera. Through the above method, the external camera parameters, internal camera parameters of the left-eye camera, and the external camera parameters and internal camera parameters of the right-eye camera can be calculated respectively. Of course, the method provided in this embodiment can also be used for calibrating a monocular camera.

[0075] The technical solution of this embodiment is to obtain the calibration wall image pairs corresponding to the binocular camera to be calibrated at the first distance, the second distance, and the third distance respectively, so as to obtain the image pairs containing at least three calibration plates at different distances. Furthermore, determine the corner image coordinates of the calibration plate corners of each calibration plate in each calibration wall image pair, and determine the three-dimensional coordinates of the calibration plate corners of each calibration plate. Based on the corner image coordinates and the three-dimensional coordinates of the calibration plate corners, determine the external camera parameters and the internal camera parameters corresponding to the binocular camera to be calibrated. It realizes obtaining the image pairs containing calibration plates in different poses through the calibration wall image pairs collected once at each distance, and then determining the internal and external parameters of the binocular camera through the image pairs collected once at each distance. This method does not need to collect images multiple times for calibration plates in different poses respectively, improving the calibration efficiency and calibration accuracy of the binocular camera.

[0076] Embodiment 2

[0077] Figure 2 It is a schematic flowchart of a binocular camera calibration method provided by the second embodiment of the present invention. On the basis of the above embodiment, optionally, the calibration wall includes a reference calibration plate and at least two reference calibration plates. The method further includes: converting the calibration plate corners other than the calibration plate corners on the reference calibration plate in the calibration wall image pair corresponding to the first distance to the three-dimensional coordinate system of the reference calibration plate in the calibration wall image pair corresponding to the first distance to obtain the three-dimensional reference coordinates of the calibration plate corners; determining the corner reprojection image coordinates corresponding to the three-dimensional reference coordinates of the corners based on the external camera parameters and the internal camera parameters; calculating the reprojection residual based on the corner image coordinates and the corner reprojection image coordinates, and optimizing the external camera parameters and / or the internal camera parameters according to the calculation result of the reprojection residual. The explanations of the same or corresponding terms in the above embodiments are not repeated here. See Figure 2 , the binocular camera calibration provided by this embodiment includes the following steps:

[0078] S210. Respectively determine the calibration wall image pairs corresponding to the first distance, the second distance, and the third distance of the binocular camera to be calibrated, where the calibration wall includes a reference calibration plate and at least two reference calibration plates.

[0079] In this embodiment, the reference calibration plate is a calibration plate parallel to the calibration wall plane. Exemplarily, Figure 1B the calibration plate in the middle is the reference calibration plate. The reference calibration plates are the remaining calibration plates in the calibration wall except the reference calibration plate, such as Figure 1B the calibration plates in the upper left corner, lower left corner, upper right corner, and lower right corner of

[0080] S220. Determine the corner image coordinates of the calibration board corners of each calibration board in each calibration wall image pair, and determine the corner three-dimensional coordinates of each calibration board corner.

[0081] S230. Based on the corner image coordinates and the corner three-dimensional coordinates, determine the external camera parameters and internal camera parameters corresponding to the binocular camera to be calibrated.

[0082] S240. Convert the calibration board corners other than the calibration board corners on the reference calibration board in the calibration wall image pair corresponding to the first distance to the three-dimensional coordinate system of the reference calibration board in the calibration wall image pair corresponding to the first distance, to obtain the corner three-dimensional reference coordinates of each calibration board corner.

[0083] In this embodiment, all calibration board corners include the calibration board corners in the calibration wall image pair corresponding to the first distance, the calibration board corners in the calibration wall image pair corresponding to the second distance, and the calibration board corners in the calibration wall image pair corresponding to the third distance; among all calibration board corners, convert the calibration board corners other than the calibration board corners on the reference calibration board in the calibration wall image pair corresponding to the first distance to the three-dimensional coordinate system of the reference calibration board in the calibration wall image pair corresponding to the first distance, to obtain the corner three-dimensional reference coordinates of each other calibration board corner. It should be noted that the three-dimensional coordinate systems of the reference calibration boards in the calibration wall image pairs corresponding to the first distance, the second distance, and the third distance are different. Specifically, since in this embodiment, the binocular camera to be calibrated can be moved to set the first distance, the second distance, and the third distance between the binocular camera to be calibrated and the calibration wall respectively, therefore, there is relative movement between the binocular camera to be calibrated and the calibration wall. In this embodiment, it can also be regarded as the binocular camera to be calibrated being fixed, and the calibration wall is moved to the first distance, the second distance, and the third distance respectively. From this, it can be seen that the three-dimensional coordinate systems of the reference calibration boards at different distances are different.

[0084] Specifically, the conversion of the calibration board corners other than the calibration board corners on the reference calibration board in the calibration wall image pair corresponding to the first distance to the three-dimensional coordinate system of the reference calibration board in the calibration wall image pair corresponding to the first distance, to obtain the corner three-dimensional reference coordinates of each calibration board corner, can be: determine the position conversion relationship between the reference calibration board and each reference calibration board; based on the position conversion relationship, determine the corner three-dimensional reference coordinates of the calibration board corners other than the calibration board corners on the reference calibration board in the calibration wall image pair corresponding to the first distance, converted to the three-dimensional coordinate system of the reference calibration board in the calibration wall image pair corresponding to the first distance.

[0085] Among them, the position conversion relationship is the conversion relationship of the reference calibration board at any distance relative to the reference calibration board at the first distance. The position conversion relationship can include a conversion matrix and a displacement matrix; for example, the position conversion relationship is Among them, T bi is the position conversion relationship of the i-th reference calibration board relative to the reference calibration board, R bi is the transformation matrix of the i-th reference calibration board, and t bi is. The displacement matrix of the exemplary i-th reference calibration board. Exemplarily, based on the position relationship, the three-dimensional reference coordinates of the corner points of each calibration board converted to the corner points of the calibration wall image corresponding to the first distance with respect to the reference calibration board are determined, and the following formula is satisfied: P b0 = R bi P bi + t bi , where P b0 is the three-dimensional corner coordinates of the calibration board corner points on the reference calibration board, and P bi is the three-dimensional reference coordinates of the calibration board corner points on the reference calibration board.

[0086] In this embodiment, considering the relative relationship between the calibration boards collected at different distances, it is used to determine the three-dimensional reference coordinates of the corner points of each calibration board converted to the three-dimensional coordinate system of the reference calibration board in the calibration wall image corresponding to the first distance, and then based on the three-dimensional reference coordinates of the corner points, a mapping from the three-dimensional coordinates of the reference calibration board to the image coordinates is established, that is, the camera reprojection process.

[0087] S250. Determine the corner re-projection image coordinates corresponding to the three-dimensional reference coordinates of the corner points based on the external camera parameters and the internal camera parameters.

[0088] Specifically, through the external camera parameters and internal camera parameters determined in the foregoing steps, the corner re-projection image coordinates corresponding to the three-dimensional reference coordinates of the corner points can be calculated, that is, the coordinates of the calibration board corner points mapped to the image. For example:

[0089]

[0090] Among them, is the homogeneous coordinate form of the three-dimensional reference coordinates of the corner points converted to the reference standard board, R cb , t cb is the transformation relationship between the binocular camera to be calibrated and the reference calibration board, D represents the five distortion coefficients, {D} represents the five distortion coefficient model operation (note: here the {·} operator represents the distortion operation transformation, not matrix multiplication), K is the internal camera parameter, and (u, v) p is the corner re-projection image coordinate.

[0091] S260. Calculate the reprojection residual based on the corner image coordinates and the corner re-projection image coordinates, and optimize the external camera parameters and / or internal camera parameters according to the calculation result of the reprojection residual.

[0092] Specifically, the difference between the corner image coordinates and the corner reprojection image coordinates can be used as the reprojection residual, and the external camera parameters and / or the internal camera parameters can be adjusted backward according to the reprojection residual. It should be noted that the above steps S250 - S260 can be executed repeatedly for multiple times to iteratively and repeatedly perform the operation of optimizing and adjusting the external camera parameters and / or the internal camera parameters until the iteration stop condition is met; among them, the iteration stop condition can be that the number of iterations exceeds a preset number, or the calculation result of the reprojection residual is less than a preset threshold, etc. Of course, in the process of backward optimizing and adjusting the external camera parameters and / or the internal camera parameters in this embodiment, the position conversion relationship between the reference calibration board and each reference calibration board, that is, T bi , the initial value of the position conversion relationship can be 0. Through the above continuous iterative optimization process, the final accurate position conversion relationship can be determined.

[0093] In addition, this embodiment takes into account that due to measurement errors, for example, during the process of the binocular camera to be calibrated collecting the calibration wall image pairs corresponding to the first distance, the second distance, and the third distance, the moving direction of the binocular camera to be calibrated may not be completely perpendicular, etc., which will cause a certain deviation between the three-dimensional coordinates of each calibration board corner and the ideal value. Therefore, during the optimization process of the internal and external camera parameters, this part of the influence can be further considered to further improve the optimization accuracy.

[0094] Specifically, considering the change in the three-dimensional coordinates of the corners caused by the relative translation of the binocular camera to be calibrated, where the camera translation direction cannot be completely perpendicular to the plane of the reference calibration board, therefore, a deviation angle θ is introduced to represent it, and the camera displacement distance d is actually measured. Denote the serial numbers of the first distance, the second distance, and the third distance as 0, 1, 2 respectively. Therefore, the three-dimensional coordinates P bij of the calibration board corners at any distance and the three-dimensional coordinates P bi0 of the calibration board corners at the close distance are related as follows:

[0095]

[0096] In summary, by way of example, determining the three-dimensional reference coordinates of the corner points of the calibration plate other than the calibration plate corner points on the reference calibration plate in the calibration wall image pair corresponding to the first distance and converting them to the three-dimensional coordinate system of the reference calibration plate in the calibration wall image pair corresponding to the first distance based on the position conversion relationship includes the following steps: determining the measured working distance spacing corresponding to the binocular camera to be calibrated and each deviation angle; based on the measured working distance spacing, each deviation angle, and the position conversion relationship, determining the three-dimensional reference coordinates of the corner points of the calibration plate other than the calibration plate corner points on the reference calibration plate in the calibration wall image pair corresponding to the first distance and converting them to the three-dimensional coordinate system of the reference calibration plate in the calibration wall image pair corresponding to the first distance; correspondingly, optimizing the external camera parameters and / or internal camera parameters according to the calculation result of the reprojection residual includes: optimizing at least one of the external camera parameters, the internal camera parameters, the measured working distance spacing, each deviation angle, and the position conversion relationship according to the calculation result of the reprojection residual.

[0097] Among them, the measured working distance spacing is the camera displacement distance measured actually. Optionally, determining the three-dimensional reference coordinates of the corner points of the calibration plate other than the calibration plate corner points on the reference calibration plate in the calibration wall image pair corresponding to the first distance and converting them to the three-dimensional coordinate system of the reference calibration plate in the calibration wall image pair corresponding to the first distance satisfies the following formula:

[0098]

[0099] Among them, R bi is the rotation matrix, t bi is the displacement matrix, P bij is the three-dimensional coordinates of the corner point, P b00 is the three-dimensional reference coordinates of the corner point, j is the preset distance coefficient, d is the measured working distance spacing, θ, is the deviation angle. If it is the calibration plate corner point in the calibration wall image corresponding to the first distance, the preset distance coefficient can be 0; if it is the calibration plate corner point in the calibration wall image corresponding to the second distance, the preset distance coefficient can be 1; if it is the calibration plate corner point in the calibration wall image corresponding to the third distance, the preset distance coefficient can be 2.

[0100] In this exemplary embodiment, the calculation result of the reprojection residual can also be used to reversely optimize and adjust the measured working distance spacing, each deviation angle, or the position conversion relationship, as shown in the following formula:

[0101]

[0102] Among them, e ij is the calculation result of the reprojection residual, (u, v) p$(X, Y, Z)$ is the three-dimensional reference coordinate of the corner point for reprojection, $(u, v)$ is the corner point image coordinate obtained by extracting sub-pixel coordinates, and $P$ b00 is the three-dimensional reference coordinate of the corner point.

[0103] It should be noted that for each calibration board corner point in the camera to be calibrated, the corner point image coordinate can define an optimization objective function of the reprojection residual as shown above. Through non-linear optimization using the above formula, high-precision binocular calibration results (camera internal parameters and camera external parameters) can be obtained.

[0104] In the technical solution of this embodiment, by converting other calibration board corner points except the calibration board corner points on the reference calibration board in the calibration wall image pair corresponding to the first distance to the three-dimensional coordinate system of the reference calibration board in the calibration wall image pair corresponding to the first distance, the three-dimensional reference coordinates of each calibration board corner point are obtained. Then, based on the camera external parameters and the camera internal parameters, the corner point reprojection image coordinates corresponding to the three-dimensional reference coordinates of the corner points are determined. The reprojection residual is calculated based on the corner point image coordinates and the corner point reprojection image coordinates, and the camera external parameters and / or the camera internal parameters are reversely optimized according to the calculation result of the reprojection residual, realizing the optimization of the calibration result of the binocular camera to be calibrated and improving the camera calibration accuracy.

[0105] Embodiment 3

[0106] Figure 3 is a schematic flowchart of a binocular camera calibration method provided by Embodiment 3 of the present invention. As Figure 3 shown, the binocular camera calibration method provided in this embodiment includes the following steps:

[0107] S310. Build a calibration wall and a moving environment for the binocular camera to be calibrated.

[0108] This step only needs to be operated offline once. The goal is to build a calibration wall containing calibration boards with multiple different angular postures according to the field of view requirements of the binocular camera to be calibrated. As shown in the appendix Figure 1B shown, the number of calibration boards is not less than three, and it is recommended to use five, distributed in the full field of view. There is one calibration board parallel to the calibration wall plane, which is used as the reference calibration board, and the relative positions and postures of the remaining calibration boards with respect to the reference calibration board always remain fixed. The relative relationships of the three-dimensional coordinates of the calibration board corner points on each calibration board can be measured using other calibrated binocular imaging devices, and the world coordinate system is converted to the three-dimensional coordinate system of the reference calibration board. The set of three-dimensional reference coordinates of all calibration board corner points is denoted as $\{P_0\}$, which is used for subsequent initial value calculation. In addition, according to the working distance range of the binocular camera to be calibrated, a linear guide rail is built in the corresponding interval range, and the direction is basically perpendicular to the reference calibration board. The linear guide rail is equipped with a laser detection scale, which is used for subsequent high-precision translational movement of the binocular camera to be calibrated along the guide rail direction.

[0109] S320. The binocular camera to be calibrated collects three groups of calibration wall image pairs at three working distances of near, medium, and far respectively.

[0110] Among them, each group of calibration wall image pairs contains one image from each of the left and right cameras. The binocular camera to be calibrated is fixed on a linear guide rail. At the near working distance, a group of calibration wall image pairs is collected, then it moves a fixed distance d along the linear guide rail to the medium working distance, and a group of calibration wall image pairs is collected again. Then it moves a fixed distance d along the linear guide rail to the far working distance, and a group of calibration wall image pairs is collected again.

[0111] S330. Pair the calibration plates in the calibration wall image pairs.

[0112] First, extract the coordinates of all calibration plate corner points in the calibration wall image pairs. The corner points of each calibration plate can be distinguished through a clustering algorithm. According to the relative position relationship between the calibration plates, each calibration plate can be identified, and the pairing of the calibration plates in the calibration wall image pairs is realized. Finally, the set of corner image coordinates of the binocular calibration plate corner points is denoted as {uv l} and {uv r} respectively.

[0113] S340. Determine the three-dimensional coordinates of the calibration plate corner points.

[0114] According to the relative relationship of the three-dimensional coordinates of the corner points on each calibration plate determined in S310, directly assign the set of three-dimensional coordinates of the corner points of the calibration plate in the calibration wall image pair at the near distance as {P0}, then the set of three-dimensional coordinates of the corner points of the calibration plate in the calibration wall image pair at the medium distance is {P0+(0, 0, d) T}, and the set of three-dimensional coordinates of the corner points of the calibration plate in the calibration wall image pair at the far distance is {P0+(0, 0, 2d) T}. Arrange the three-dimensional coordinates of all corner points in order and integrate them as {P all}, as shown in the following formula:

[0115] {P all0} = {P0, P0+(0, 0, d) T , P0+(0, 0, 2d) T}

[0116] S350. Calculate the projection matrix according to the corner image coordinates and the three-dimensional coordinates of the corner points.

[0117] Taking the left-eye camera as an example, take the point (u, v) in the set of corner image coordinates {uv l} and the corresponding point (X, Y, Z) in the set of three-dimensional coordinates of the corner points {P all}. The two can be calculated through the projection matrix M. According to the information of all observed points, the projection matrix M can be calculated using the least squares method.

[0118] S360. Calculate the camera internal parameters and the camera external parameters based on the projection matrix.

[0119] S370. Determine the three-dimensional reference coordinates of all calibration plate corner points.

[0120] Specifically, the position conversion relationship between the reference calibration plate and each reference calibration plate can be determined first. Then, based on the measured working distance spacing, each deviation angle, and the position conversion relationship, other calibration plate corner points except the calibration plate corner points on the reference calibration plate in the close-range calibration wall image pair are determined, and the three-dimensional reference coordinates of the corner points are converted to the three-dimensional coordinate system of the reference calibration plate in the close-range calibration wall image pair.

[0121] The three-dimensional coordinates P of the corner points on any calibration plate at any distance from near to far bij are converted to the three-dimensional coordinates P of the corner points of the reference standard plate at close range b00 , and the conversion relationship is as follows:

[0122]

[0123] S380. Determine the corner re-projection image coordinates corresponding to the three-dimensional reference coordinates of the corner points based on the camera internal parameters and the camera external parameters.

[0124] S390. Construct a re-projection residual optimization objective function based on the corner re-projection image coordinates and the corner image coordinates, and optimize the camera internal parameters, the camera external parameters, the position conversion relationship, the measured working distance spacing, and the deviation angle through the re-projection residual optimization objective function.

[0125] Exemplarily, the re-projection residual optimization objective function can be:

[0126]

[0127] Specifically, a re-projection residual optimization objective function can be defined for the corner image coordinates and the corner re-projection coordinates of each calibration plate corner point.

[0128] In this embodiment, high-precision calibration can be achieved with a small number of image groups, which is easy to operate in a process flow, has a high calibration efficiency. Moreover, by using the determined position relationship between the calibration plates and the precise movement of the binocular camera, the high calibration accuracy of the system and the high repeatability of the calibration results are ensured. The actual process operation is simple, and workers with no basic knowledge can also realize the image acquisition of the calibration process, which is easy to be process-flowed and has a high efficiency; the experimental environment is easy to build, and the calibration wall, linear guide rails, etc. are all equipment that is easy to process and obtain. By using the determined position relationship between the calibration plates and the precise movement of the binocular camera, and by reconstructing the target optimization function, the calibration accuracy and stability of the navigator binocular system can be effectively ensured.

[0129] Embodiment 4

[0130] Figure 4 FIG. 5 is a schematic structural diagram of a binocular camera calibration device provided in Embodiment 4 of the present invention. This embodiment is applicable to the situation of calibrating a binocular camera, especially applicable to the situation of calibrating a binocular camera according to a pre-built calibration wall. The device specifically includes: an image determination module 410, an image coordinate determination module 420, and an internal and external parameter determination module 430.

[0131] The image determination module 410 is configured to respectively determine calibration wall image pairs corresponding to the first distance, the second distance, and the third distance of the binocular camera to be calibrated, where the calibration wall includes at least three calibration plates;

[0132] The image coordinate determination module 420 is configured to determine the corner image coordinates of the calibration plate corner points of each calibration plate in each of the calibration wall image pairs;

[0133] The internal and external parameter determination module 430 is configured to determine the three-dimensional coordinates of the corner points of each calibration plate corner point, and based on the corner image coordinates and the three-dimensional coordinates of the corner points, determine the camera external parameters and camera internal parameters corresponding to the binocular camera to be calibrated.

[0134] Optionally, the calibration wall includes a reference calibration plate and at least two reference calibration plates. The device further includes an internal and external parameter optimization module, configured to convert other calibration plate corner points except the calibration plate corner points on the reference calibration plate in the calibration wall image pair corresponding to the first distance to the three-dimensional coordinate system of the reference calibration plate in the calibration wall image pair corresponding to the first distance, to obtain the three-dimensional reference coordinates of the corner points of each calibration plate corner point; determine the corner reprojection image coordinates corresponding to the three-dimensional reference coordinates of the corner points based on the camera external parameters and the camera internal parameters; calculate the reprojection residual based on the corner image coordinates and the corner reprojection image coordinates, and optimize the camera external parameters and / or the camera internal parameters according to the calculation result of the reprojection residual.

[0135] Optionally, the internal and external parameter optimization module includes a corner point conversion unit, configured to determine the position conversion relationship between the reference calibration plate and each of the reference calibration plates; based on the position conversion relationship, determine other calibration plate corner points except the calibration plate corner points on the reference calibration plate in the calibration wall image pair corresponding to the first distance, and convert them to the three-dimensional reference coordinates of the corner points in the three-dimensional coordinate system of the reference calibration plate in the calibration wall image pair corresponding to the first distance.

[0136] Optionally, the corner conversion unit is specifically configured to determine the measured working distance spacing corresponding to the binocular camera to be calibrated and each deviation angle; based on the measured working distance spacing, each of the deviation angles, and the position conversion relationship, determine other calibration plate corner points except the calibration plate corner points on the reference calibration plate in the calibration wall image pair corresponding to the first distance, and convert them to the corner three-dimensional reference coordinates in the three-dimensional coordinate system of the reference calibration plate in the calibration wall image pair corresponding to the first distance;

[0137] Correspondingly, the internal and external parameter optimization module is used to optimize at least one of the camera external parameters, the camera internal parameters, the measured working distance spacing, each of the deviation angles, and the position conversion relationship according to the calculation result of the reprojection residual.

[0138] Optionally, the internal and external parameter determination module 430 includes a projection matrix determination unit, and the projection matrix determination unit is configured to determine the projection matrix of the binocular camera to be calibrated based on the corner image coordinates and the corner three-dimensional coordinates; calculate the camera external parameters and camera internal parameters corresponding to the binocular camera to be calibrated based on the projection matrix.

[0139] Optionally, the internal and external parameter determination module 430 includes a three-dimensional coordinate determination unit, and the three-dimensional coordinate determination unit is configured to obtain the three-dimensional coordinate relative relationship between the calibration plate corner points in each calibration plate, and determine the corner three-dimensional coordinates of each calibration plate corner point in the calibration wall image pair corresponding to the first distance based on the three-dimensional coordinate relative relationship; based on the corner three-dimensional coordinates of each calibration plate corner point in the calibration wall image pair corresponding to the first distance, respectively determine the corner three-dimensional coordinates of each calibration plate corner point in the calibration wall image pair corresponding to the second distance and the calibration wall image pair corresponding to the third distance.

[0140] Optionally, the image coordinate determination module 420 is specifically configured to obtain the corner image coordinates of the calibration plate corner points in each calibration wall image pair; perform clustering processing on each of the corner image coordinates, and respectively determine the binocular calibration plate pairing results of each calibration wall image pair based on the clustering processing results; update the corner image coordinates of each calibration plate corner point based on the binocular calibration plate pairing results.

[0141] In this embodiment, through the image determination module, calibration wall image pairs corresponding to the binocular camera to be calibrated at the first distance, the second distance, and the third distance are acquired, so as to obtain image pairs containing at least three calibration plates at different distances. Furthermore, through the image coordinate determination module, the corner image coordinates of the calibration plate corners of each calibration plate in each calibration wall image pair are determined, and through the internal and external parameter determination module, the three-dimensional coordinates of the calibration plate corners of each calibration plate are determined. Based on the corner image coordinates and the three-dimensional coordinates of the calibration plate corners, the external parameters and internal parameters of the binocular camera to be calibrated are determined, realizing obtaining image pairs containing calibration plates in different poses through the calibration wall image pairs collected once at each distance. Furthermore, the internal and external parameters of the binocular camera are determined through the image pairs collected once at each distance. This method does not require separately collecting images multiple times for calibration plates in different poses, improving the calibration efficiency and calibration accuracy of the binocular camera.

[0142] The binocular camera calibration device provided by the embodiment of the present invention can execute the binocular camera calibration method provided by any embodiment of the present invention, and has corresponding functional modules and beneficial effects for executing the method.

[0143] It should be noted that the various units and modules included in the above system are only divided according to functional logic, but are not limited to the above division, as long as the corresponding functions can be realized; in addition, the specific names of the functional units are only for the convenience of mutual distinction and do not limit the protection scope of the embodiments of the present invention.

[0144] Embodiment Five

[0145] Figure 5 It is a schematic structural diagram of an electronic device provided by Embodiment Five of the present invention. Figure 5 It shows a block diagram of an exemplary electronic device 12 suitable for implementing the embodiments of the present invention. Figure 5 The shown electronic device 12 is only an example and should not bring any limitation to the functions and usage scope of the embodiments of the present invention. The device 12 is typically an electronic device undertaking the binocular camera calibration function.

[0146] As Figure 5 shown, the electronic device 12 is presented in the form of a general-purpose computing device. The components of the electronic device 12 may include but are not limited to: one or more processors or processing units 16, a memory 28, and a bus 18 connecting different components (including the memory 28 and the processing unit 16).

[0147] Bus 18 represents one or more of several types of bus architectures, including a memory bus or memory controller, a peripheral bus, an Accelerated Graphics Port, a processor, or a local bus using any of the various bus architectures. By way of example, these architectures include, but are not limited to, Industry Standard Architecture (ISA) bus, Micro Channel Architecture (MCA) bus, Enhanced ISA bus, Video Electronics Standards Association (VESA) local bus, and Peripheral Component Interconnect (PCI) bus.

[0148] Electronic device 12 typically includes a variety of computer-readable media. These media can be any available media that can be accessed by electronic device 12, including volatile and nonvolatile media, removable and non-removable media.

[0149] Memory 28 may include computer device-readable media in the form of volatile memory, such as Random Access Memory (RAM) 30 and / or cache memory 32. Electronic device 12 may further include other removable / non-removable, volatile / nonvolatile computer storage media. By way of example only, storage device 34 can be used for reading and writing on non-removable, nonvolatile magnetic media ( Figure 5 not shown, typically referred to as a "hard disk drive"). Although Figure 5 not shown in the figure, a disk drive for reading and writing on a removable nonvolatile disk (such as a "floppy disk") and an optical disk drive for reading and writing on a removable nonvolatile optical disk (such as a Compact Disc-Read Only Memory (CD-ROM), Digital Video Disc-Read Only Memory (DVD-ROM), or other optical media) can be provided. In these cases, each drive can be connected to bus 18 through one or more data media interfaces. Memory 28 may include at least one program product 40 having a set of program modules 42 configured to perform the functions of the embodiments of the present invention. The program product 40, which can be stored in, for example, memory 28, such program modules 42 include, but are not limited to, one or more application programs, other program modules, and program data, and the implementation of a network environment may be included in each or some combination of these examples. Program modules 42 generally execute the functions and / or methods in the embodiments described in the present invention.

[0150] The electronic device 12 can also communicate with one or more external devices 14 (such as a keyboard, a mouse, a camera, etc. and a display), and can also communicate with one or more devices that enable a user to interact with the electronic device 12, and / or communicate with any device that enables the electronic device 12 to communicate with one or more other computing devices (such as a network card, a modem, etc.). Such communication can be carried out through the input / output (I / O) interface 22. Moreover, the electronic device 12 can also communicate with one or more networks (such as a Local Area Network (LAN), a Wide Area Network (WAN)) and / or a public network, such as the Internet) through the network adapter 20. As shown in the figure, the network adapter 20 communicates with other modules of the electronic device 12 through the bus 18. It should be understood that although not shown in the figure, other hardware and / or software modules can be used in combination with the electronic device 12, including but not limited to: microcode, device drivers, redundant processing units, external disk drive arrays, Redundant Arrays of Independent Disks (RAID) devices, magnetic tape drives, and data backup storage devices, etc.

[0151] The processor 16 executes various functional applications and data processing by running the programs stored in the memory 28, for example, implementing the binocular camera calibration method provided in the above embodiments of the present invention, including:

[0152] respectively determining calibration wall image pairs corresponding to the first distance, the second distance, and the third distance of the binocular camera to be calibrated, wherein the calibration wall includes at least three calibration plates;

[0153] determining the corner image coordinates of the calibration plate corners of each calibration plate in each of the calibration wall image pairs;

[0154] determining the three-dimensional coordinates of the corners of each calibration plate corner, and based on the corner image coordinates and the three-dimensional coordinates of the corners, determining the external camera parameters and the internal camera parameters corresponding to the binocular camera to be calibrated.

[0155] Of course, those skilled in the art can understand that the processor can also implement the technical solutions of the binocular camera calibration method provided in any embodiment of the present invention.

[0156] Embodiment Six

[0157] Embodiment Six of the present invention also provides a computer-readable storage medium, on which a computer program is stored, and when the program is executed by a processor, it implements the steps of the binocular camera calibration method provided in any embodiment of the present invention. The method includes:

[0158] Determine the calibration wall image pairs corresponding to the first distance, the second distance, and the third distance of the binocular camera to be calibrated respectively, where the calibration wall includes at least three calibration plates;

[0159] Determine the corner image coordinates of the calibration plate corners of each calibration plate in each of the calibration wall image pairs;

[0160] Determine the three-dimensional coordinates of the calibration plate corners. Based on the corner image coordinates and the three-dimensional coordinates of the corners, determine the external camera parameters and the internal camera parameters corresponding to the binocular camera to be calibrated.

[0161] The computer storage medium of the embodiments of the present invention may adopt any combination of one or more computer-readable media. The computer-readable medium may be a computer-readable signal medium or a computer-readable storage medium. The computer-readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination of the above. More specific examples (a non-exhaustive list) of the computer-readable storage medium include: an electrical connection having one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In this document, the computer-readable storage medium may be any tangible medium that contains or stores a program, and the program can be used by or in combination with an instruction execution system, apparatus, or device.

[0162] The computer-readable signal medium may include a data signal propagated in a baseband or as part of a carrier wave, which carries the computer-readable program code. Such a propagated data signal may take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the above. The computer-readable signal medium may also be any computer-readable medium other than the computer-readable storage medium, and the computer-readable medium can send, propagate, or transmit a program for use by or in combination with an instruction execution system, apparatus, or device.

[0163] The program code contained on the computer-readable medium can be transmitted by any appropriate medium, including but not limited to wireless, wire, optical cable, RF, etc., or any suitable combination of the above.

[0164] Computer program code for performing the operations of the embodiments of the present invention may be written in one or more programming languages or combinations thereof. The programming languages include object-oriented programming languages such as Java, Smalltalk, C++, and also include conventional procedural programming languages such as the "C" language or similar programming languages. The program code may be executed entirely on the user's computer, partially on the user's computer, executed as a stand-alone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In the case of a remote computer, the remote computer may be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., by connecting through the Internet using an Internet service provider).

[0165] Note that the above is only the preferred embodiment of the present invention and the technical principles applied. Those skilled in the art will understand that the present invention is not limited to the specific embodiments described herein. Various obvious changes, re-adjustments, and substitutions can be made by those skilled in the art without departing from the protection scope of the present invention. Therefore, although the present invention has been described in more detail through the above embodiments, the present invention is not limited to the above embodiments. Without departing from the concept of the present invention, more other equivalent embodiments may be included, and the scope of the present invention is determined by the scope of the appended claims.

Claims

1. A binocular camera calibration method, characterized in that, The method includes: Determining calibration wall image pairs corresponding to a first distance, a second distance, and a third distance of a binocular camera to be calibrated respectively, where the calibration wall includes at least three calibration plates; wherein, the calibration plates are evenly distributed within the full field of view of the binocular camera to be calibrated; Determining the corner image coordinates of the calibration plate corners of each calibration plate in each of the calibration wall image pairs; Determining the three-dimensional coordinates of the calibration plate corners, and based on the corner image coordinates and the three-dimensional coordinates of the corners, determining the external camera parameters and internal camera parameters corresponding to the binocular camera to be calibrated; The determining the three-dimensional coordinates of the calibration plate corners includes: Obtaining the relative three-dimensional coordinate relationship between the calibration plate corners in each calibration plate, and based on the relative three-dimensional coordinate relationship, determining the three-dimensional coordinates of the calibration plate corners in the calibration wall image pair corresponding to the first distance; Based on the three-dimensional coordinates of the calibration plate corners in the calibration wall image pair corresponding to the first distance, respectively determining the three-dimensional coordinates of the calibration plate corners in the calibration wall image pair corresponding to the second distance and the calibration wall image pair corresponding to the third distance; The determining the external camera parameters and internal camera parameters corresponding to the binocular camera to be calibrated based on the corner image coordinates and the three-dimensional coordinates of the corners includes: Determining the projection matrix of the binocular camera to be calibrated based on the corner image coordinates and the three-dimensional coordinates of the corners; Calculating the external camera parameters and internal camera parameters corresponding to the binocular camera to be calibrated based on the projection matrix.

2. The method according to claim 1, characterized in that, The calibration wall includes a reference calibration plate and at least two reference calibration plates, and the method further includes: Converting the calibration plate corners other than the calibration plate corners on the reference calibration plate in the calibration wall image pair corresponding to the first distance to the three-dimensional coordinate system of the reference calibration plate in the calibration wall image pair corresponding to the first distance, to obtain the three-dimensional reference coordinates of the calibration plate corners; Determining the corner reprojection image coordinates corresponding to the three-dimensional reference coordinates of the corners based on the external camera parameters and the internal camera parameters; Calculating the reprojection residuals based on the corner image coordinates and the corner reprojection image coordinates, and optimizing the external camera parameters and / or the internal camera parameters according to the calculation result of the reprojection residuals.

3. The method according to claim 2, wherein The converting the calibration plate corners other than the calibration plate corners on the reference calibration plate in the calibration wall image pair corresponding to the first distance to the three-dimensional coordinate system of the reference calibration plate in the calibration wall image pair corresponding to the first distance, to obtain the three-dimensional reference coordinates of the calibration plate corners includes: Determining the position conversion relationship between the reference calibration plate and each of the reference calibration plates; Based on the position conversion relationship, determining the three-dimensional reference coordinates of the calibration plate corners other than the calibration plate corners on the reference calibration plate in the calibration wall image pair corresponding to the first distance, when converted to the three-dimensional coordinate system of the reference calibration plate in the calibration wall image pair corresponding to the first distance.

4. The method according to claim 3, characterized in that, Determining, based on the position conversion relationship, the corner three-dimensional reference coordinates of other calibration plate corner points on the reference calibration plate in the calibration wall image pair corresponding to the first distance, excluding the calibration plate corner points on the reference calibration plate in the calibration wall image pair corresponding to the first distance, includes: Determining the measured working distance spacing and each deviation angle corresponding to the binocular camera to be calibrated; Based on the measured working distance spacing, each deviation angle, and the position conversion relationship, determining the corner three-dimensional reference coordinates of other calibration plate corner points on the reference calibration plate in the calibration wall image pair corresponding to the first distance, excluding the calibration plate corner points on the reference calibration plate in the calibration wall image pair corresponding to the first distance; Correspondingly, optimizing the external camera parameters and / or the internal camera parameters according to the calculation result of the reprojection residual includes: Optimizing at least one of the external camera parameters, the internal camera parameters, the measured working distance spacing, each deviation angle, and the position conversion relationship according to the calculation result of the reprojection residual.

5. The method according to claim 1, characterized in that Determining the corner image coordinates of the calibration plate corner points of each calibration plate in each calibration wall image pair includes: Obtaining the corner image coordinates of the calibration plate corner points of each calibration plate in each calibration wall image pair; Performing clustering processing on each corner image coordinate, and respectively determining the binocular calibration plate pairing results of each calibration wall image pair based on the clustering processing result; Updating the corner image coordinates of each calibration plate corner point based on the binocular calibration plate pairing result.

6. A binocular camera calibration device, characterized in that, The device includes: An image determination module, configured to respectively determine calibration wall image pairs corresponding to the first distance, the second distance, and the third distance of the binocular camera to be calibrated, where the calibration wall includes at least three calibration plates; and each calibration plate is evenly distributed in the full field of view of the binocular camera to be calibrated; An image coordinate determination module, configured to determine the corner image coordinates of the calibration plate corner points of each calibration plate in each calibration wall image pair; An internal and external parameter determination module, configured to determine the corner three-dimensional coordinates of each calibration plate corner point, and determine the external camera parameters and the internal camera parameters corresponding to the binocular camera to be calibrated based on the corner image coordinates and the corner three-dimensional coordinates; The internal and external parameter determination module is further configured to: obtain the three-dimensional coordinate relative relationship between the calibration plate corner points in each calibration plate, and determine the corner three-dimensional coordinates of each calibration plate corner point in the calibration wall image pair corresponding to the first distance based on the three-dimensional coordinate relative relationship; Based on the corner three-dimensional coordinates of each calibration plate corner point in the calibration wall image pair corresponding to the first distance, respectively determine the corner three-dimensional coordinates of each calibration plate corner point in the calibration wall image pair corresponding to the second distance and the calibration wall image pair corresponding to the third distance; The internal and external parameter determination module is further configured to: determine the projection matrix of the binocular camera to be calibrated based on the corner image coordinates and the corner three-dimensional coordinates; Calculating the external camera parameters and the internal camera parameters corresponding to the binocular camera to be calibrated based on the projection matrix.

7. An electronic device, characterized in that, The electronic device includes: One or more processors; A storage device, configured to store one or more programs, When the one or more programs are executed by the one or more processors, the one or more processors implement the binocular camera calibration method according to any one of claims 1-5.

8. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the program is executed by the processor, the binocular camera calibration method according to any one of claims 1-5 is implemented.

Images