Binocular camera self-calibration method, computer device, storage medium, program product and mobile platform

Abstract

The application discloses a binocular camera self-calibration method, a computer device, a storage medium, a program product and a mobile platform. The binocular camera comprises a first camera and a second camera, and the first camera and the second camera are not rigidly connected. The method comprises the following steps: acquiring a first image collected by the first camera and a second image collected by the second camera; determining whether the first camera and the second camera are abnormal in calibration according to the first image and the second image; and correcting the external parameters of the first camera and the second camera when it is determined that the first camera and the second camera are abnormal in calibration. The application checks the calibration abnormality of the binocular camera, corrects the external parameters when it is determined that the calibration is abnormal, and solves the problem that the external parameters of the binocular camera change due to reasons such as vibration and temperature difference, and the data collected by the binocular camera is inaccurate.

Description

Binocular camera self-calibration method, computer device, storage medium, program product and mobile platform TECHNICAL FIELD

[0001] The present application relates to the technical field of binocular cameras, and in particular to a binocular camera self-calibration method, a computer device, a storage medium, a program product and a mobile platform. BACKGROUND

[0002] A perception sensor is one of the core links of an intelligent robot. The intelligent robot includes, but is not limited to, an intelligent vehicle, a wheeled robot, a tracked robot, a biped robot, a quadruped robot, and the like.

[0003] Taking an intelligent vehicle with an automatic driving function or a high-order auxiliary driving function as an example, the perception sensor mainly shoulders two responsibilities of detecting dynamic and static obstacles around the vehicle and understanding road elements and structures. The detection of dynamic and static obstacles includes the detection of common traffic participants such as vehicles and pedestrians, and the detection of unconventional obstacles such as construction barriers and fallen branches. The understanding of road elements and structures includes the identification and understanding of road infrastructure such as lane lines, traffic lights, signboards, ground arrows, and road topology. Based on the results of environmental perception, the intelligent driving system can make a decision on how the vehicle should move. The perception sensor is the "eyes" of the intelligent vehicle. In related technologies, a binocular camera is used as the "eyes" of the intelligent vehicle. The binocular camera collects left and right images to obtain the parallax of each pixel point, and then reconstructs three-dimensional information based on the principle of triangulation to identify obstacles.

[0004] The inventors have found in the implementation of the present application that the main factors affecting the perception performance of the binocular camera are the intrinsic parameters and the extrinsic parameters of the binocular camera. However, the intrinsic parameters of the binocular camera will not change after the binocular camera is calibrated. However, the extrinsic parameters of the binocular camera are easily affected by external environmental factors (for example, the vibration of the vehicle, the change of the environmental temperature and humidity, etc., which will cause the change of the extrinsic parameters of the binocular camera).

[0005] In related technologies, a rigid connection structure is used between the two cameras to ensure the stability of the extrinsic parameters of the binocular camera. However, the rigidly connected binocular camera has the following problems: the rigid connection structure increases the cost of the binocular camera, and if different binocular distances and different vehicle installation environments need to be matched, different models of rigidly connected binocular cameras need to be produced; the rigidly connected binocular camera has a large volume, occupies a large space on the vehicle, and has poor installation flexibility.

[0006] Therefore, there is an urgent need for a technical solution that can solve the technical problems existing in the related technologies described above.

[0007] SUMMARY

[0008] Embodiments of the present application provide a binocular camera self-calibration method, a computer device, a storage medium, a program product and a mobile platform, which are used to solve at least one of the above technical problems.

[0009] In a first aspect, embodiments of the present application provide a binocular camera self-calibration method, the binocular camera comprising a first camera and a second camera, the first camera and the second camera being not rigidly connected, the method comprising:

[0010] obtaining a first image collected by the first camera and a second image collected by the second camera;

[0011] determining whether the first camera and the second camera are abnormal in calibration according to the first image and the second image;

[0012] correcting extrinsic parameters of the first camera and the second camera when it is determined that the first camera and the second camera are abnormal in calibration.

[0013] In some embodiments, determining whether the first camera and the second camera are abnormal in calibration according to the first image and the second image comprises:

[0014] dividing the first image and the second image into a plurality of sub-images respectively; calculating a calibration error value corresponding to the first image and the second image according to errors between matching pixel points in the plurality of sub-images of the first image and the plurality of sub-images of the second image; and determining that the first camera and the second camera are abnormal in calibration when the calibration error value exceeds a preset error threshold.

[0015] In some embodiments, calculating a calibration error value corresponding to the first image and the second image according to errors between matching pixel points in the plurality of sub-images of the first image and the plurality of sub-images of the second image comprises:

[0016] calculating a plurality of sub-image matching errors according to matching pixel points in the plurality of sub-images of the first image and the plurality of sub-images of the second image; and determining the calibration error value corresponding to the first image and the second image according to the plurality of sub-image matching errors.

[0017] In some embodiments, the binocular camera self-calibration method further comprises: determining a predetermined number of pixel points for each sub-image of the first image; and determining a predetermined number of pixel points for each sub-image of the second image.

[0018] calculating a plurality of sub-image matching errors according to matching pixel points in the plurality of sub-images of the first image and the plurality of sub-images of the second image comprises:

[0019] According to the matching pixel points in the predetermined number of pixel points of each sub-image of the first image and the predetermined number of pixel points of each sub-image of the second image, a plurality of sub-image matching errors corresponding to the matching pixel points are calculated.

[0020] In some embodiments, determining a calibration error value corresponding to the first image and the second image according to the plurality of sub-image matching errors comprises: calculating an average value of the plurality of sub-image matching errors as the calibration error value of the first image and the second image.

[0021] In some embodiments, before the first image and the second image are divided into a plurality of sub-images, the method further comprises: determining abnormal pixel points in the first image and the second image, and filtering out the abnormal pixel points.

[0022] In some embodiments, determining abnormal pixel points in the first image and the second image comprises: determining a plurality of pixel point error values of a plurality of pairs of matching pixel points in the first image and the second image; and determining abnormal pixel points in the plurality of pairs of matching pixel points according to the plurality of pixel point error values.

[0023] In some embodiments, when it is determined that the first camera and the second camera are calibrated abnormally, correcting the extrinsic parameters of the first camera and the second camera comprises:

[0024] An initial rotation matrix corresponding to the first camera and the second camera is obtained by using epipolar constraint; and a target disparity deviation value is searched in a preset disparity deviation range by using a grid search method, so as to correct the initial rotation matrix.

[0025] In some embodiments, the preset disparity deviation range comprises n preset disparity deviation values.

[0026] The searching of the target disparity deviation value in the preset disparity deviation range by using the grid search method comprises: for an i-th preset disparity deviation value, calculating coordinate information of a plurality of pixel points at two different time instants.

[0027] For the i-th preset disparity deviation value, an i-th group of error values is calculated according to the coordinate information of the plurality of pixel points at the two different time instants; wherein i takes a value from 1 to n.

[0028] A target disparity deviation value is determined according to a distribution of the first to n-th groups of error values.

[0029] In some embodiments, for the i-th preset disparity deviation value, the calculation of the coordinate information of the plurality of pixel points at the two different time instants comprises:

[0030] For the i-th preset parallax deviation value, first coordinate information of the plurality of pixel points at the first time is calculated; second coordinate information of the plurality of pixel points at the second time is determined according to relative pose information of the vehicle at the first time and the second time, and the first coordinate information.

[0031] In some embodiments, the target parallax deviation value is determined according to a distribution of the first to the n-th groups of error values, including: determining a number of error values smaller than a preset error threshold value in the plurality of error values of the m-th group of error values, wherein m is an integer from 1 to n, to obtain n error value numbers; and determining a preset parallax deviation value corresponding to a maximum value in the n error value numbers as the target parallax deviation value.

[0032] In the second aspect, the present application further provides a computer device, including a memory, a processor and a computer program stored in the memory, and the processor executes the computer program to implement the steps of the binocular camera self-calibration method according to any one of the above embodiments.

[0033] In some embodiments, the processor executes the computer program to implement the following steps: when it is detected that the images captured by the first camera and / or the second camera have no valid information, generating prompt information to remind the user.

[0034] In some embodiments, the processor executes the computer program to implement the following steps: recording the changes of the extrinsic parameters of the first camera and the second camera, and generating an extrinsic parameter change curve.

[0035] In the third aspect, the present application further provides a computer readable storage medium, which stores a computer program / instruction, and the computer program / instruction is executed by a processor to implement the steps of the binocular camera self-calibration method according to any one of the above embodiments.

[0036] In the fourth aspect, the present application further provides a computer program product, which includes a computer program / instruction, and the computer program / instruction is executed by a processor to implement the steps of the binocular camera self-calibration method according to any one of the above embodiments.

[0037] In the fifth aspect, the present application further provides a mobile platform, which is installed with the computer device according to any one of the above embodiments.

[0038] In some embodiments, the first camera and the second camera are installed at the front of the mobile platform. The baseline of the first camera and the second camera is determined according to the height and / or width of the detected object.

[0039] In some embodiments, the baseline of the first camera and the second camera ranges from 0.2m to 1.2m, and the field of view angle of the first camera and the second camera is 120°.

[0040] The application realizes self-calibration of a binocular camera with non-rigid connection. In one aspect, the two cameras of the binocular camera do not need a rigid connection structure, thereby saving the cost of the structure and reducing the space occupied by the binocular camera installation, and improving the flexibility of the binocular camera installation configuration. Meanwhile, by checking the calibration abnormality of the binocular camera and correcting the external parameter when determining the calibration abnormality, the problem that the external parameter between the binocular cameras changes due to reasons such as vibration and temperature difference change, thereby causing errors in the data collected by the binocular camera, is solved. BRIEF DESCRIPTION OF DRAWINGS

[0041] In order to more clearly illustrate the technical solutions of the embodiments of the application, the following will briefly introduce the drawings needed to be used in the embodiment description. Obviously, the drawings in the following description are some embodiments of the application, and other drawings can also be obtained by those skilled in the art without creative labor on the basis of these drawings.

[0042] FIG. 1 is a flowchart of an embodiment of the binocular camera self-calibration method of the application;

[0043] FIG. 2 is a schematic diagram of alignment of two matching points on the first image and the second image in the application;

[0044] FIG. 3 is a flowchart of another embodiment of constructing a language model in the application;

[0045] FIG. 4 is a schematic diagram of an embodiment of dividing sub-images in the application;

[0046] FIG. 5 is a flowchart of another embodiment of constructing a language model in the application;

[0047] FIG. 6 is a flowchart of another embodiment of constructing a language model in the application;

[0048] FIG. 7 is a schematic diagram of an embodiment of grid search used in the application;

[0049] FIG. 8 is a schematic diagram of depth change of feature points at t1 and t2 in the application;

[0050] FIG. 9 is a schematic diagram of baseline distance and observable length change of a blocked vehicle in a blocked scene in the application;

[0051] FIG. 10 is a curve of ranging error change with depth in different baseline value conditions in the application;

[0052] FIG. 11 is a schematic diagram of an embodiment of the binocular camera self-calibration system of the application;

[0053] FIG. 12 is a schematic diagram of an embodiment of the binocular camera self-calibration method combined with interactive design in the application;

[0054] FIG. 13 is a structural schematic diagram of an embodiment of a computer device of the present application. DETAILED DESCRIPTION

[0055] In order to make the objects, technical solutions and advantages of the embodiments of the present application clearer, the technical solutions in the embodiments of the present application will be described clearly and completely below with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are only some of the embodiments of the present application, rather than all the embodiments of the present application. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative work fall within the scope of protection of the present application. It should be noted that the embodiments in the present application and the features in the embodiments can be combined with each other without conflict.

[0056] It should also be noted that, in this document, the terms “comprising” and “including” not only include those elements, but also include other elements not explicitly listed, or further include elements inherent in the process, method, article or device. Without more limitations, the elements defined by the statement “comprising” do not exclude the presence of other identical elements in the process, method, article or device comprising the elements.

[0057] The present application provides a binocular camera self-calibration method. The method can be executed by a computer device to realize self-calibration of a binocular camera. The computer device can be configured on a mobile platform (for example, a vehicle, a robot and a drone, etc.). Taking a vehicle as an example, the computer device can be a domain controller of the vehicle. It should be noted that the above is only an example, and the present application is not limited thereto.

[0058] As shown in FIG. 1, the embodiments of the present application provide a binocular camera self-calibration method, wherein the binocular camera includes a first camera and a second camera, and there is no rigid connection between the first camera and the second camera. The binocular camera self-calibration method includes:

[0059] S10, acquiring a first image collected by the first camera and a second image collected by the second camera. The first camera can be a left camera, and the second camera can be a right camera. The first camera and the second camera can be independently installed at the front of the vehicle to shoot the scene in front of the vehicle and obtain the depth information of the objects (for example, a vehicle in front, a pedestrian, a lane line, a sign and a road edge, etc.) in the scene in front, for high-order auxiliary driving or automatic driving, etc. The first image and the second image are images shot by the first camera and the second camera at the same time under their respective angles of view.

[0060] S20, determining whether the first camera and the second camera are abnormal in calibration according to the first image and the second image. For example, whether the first camera and the second camera are abnormal in calibration is determined according to the alignment of the matching points in the first image and the second image. When the alignment of the matching points in the first image and the second image indicates that the calibration error between the first image and the second image exceeds a preset error threshold, it is determined that the first camera and the second camera are abnormal in calibration; when the alignment of the matching points in the first image and the second image indicates that the calibration error between the first image and the second image does not exceed the preset error threshold, it is determined that the first camera and the second camera are normal in calibration.

[0061] As shown in FIG. 2, it is a determination diagram of the alignment of two matching points on the first image and the second image in the present application. The left rectangle in the figure represents the first image, which includes two pixel points represented by a pentagram and a cross; the right rectangle in the figure represents the second image, which also includes two pixel points represented by a pentagram and a cross; the two pixel points corresponding to the two pentagrams in the first image and the second image are a pair of matching pixel points, and the two pixel points corresponding to the two crosses in the first image and the second image are another pair of matching pixel points. As can be seen from FIG. 2, there is an error error between the two pairs of matching pixel points in the Y-axis direction (the upward and downward direction in the figure). Taking the pixel point p1 (x1, y1) represented by the pentagram in the first image and the pixel point p2 (x2, y2) represented by the pentagram in the second image as an example. If the y difference of the pixel points p1 and p2: |y1-y2|<0.5 pixels, it indicates that the first image and the second image are aligned in the y direction; otherwise, it indicates that the first image and the second image are not aligned, thereby determining that the first camera and the second camera are abnormal in calibration.

[0062] S30, when it is determined that the first camera and the second camera are abnormal in calibration, correcting the extrinsic parameters of the first camera and the second camera. For example, the extrinsic parameters of the first camera and the second camera include a rotation matrix between the first camera and the second camera; when it is determined that the first camera and the second camera are abnormal in calibration, the rotation matrix between the first camera and the second camera is corrected.

[0063] The method of the embodiments of the present application realizes the self-calibration of the binocular camera with non-rigid connection. On the one hand, there is no rigid connection structure between the two cameras of the binocular camera, which saves the cost of the structure and reduces the space occupied by the installation of the binocular camera, thereby improving the flexibility of the installation configuration of the binocular camera. At the same time, by checking the abnormal calibration of the binocular camera and correcting the extrinsic parameters when it is determined that the calibration is abnormal, the problem that the extrinsic parameters between the binocular cameras change due to reasons such as vibration and temperature difference change, thereby causing errors in the data collected by the binocular camera, is solved.

[0064] Fig. 3 is a flow diagram of another embodiment of the self-calibration method of the binocular camera according to the present application. In the embodiment, determining whether the first camera and the second camera are abnormal in calibration comprises:

[0065] S21, dividing the first image and the second image into a plurality of sub-images respectively.

[0066] For example, the first image and the second image are divided into a plurality of sub-images respectively according to predetermined sizes. Fig. 4 is a diagram of an embodiment of the sub-image division according to the present application. In Fig. 4, the image is divided into a matrix with a width of 5 and a height of 4. The predetermined size can be a predetermined width and a predetermined height (for example, the predetermined width is 30 pixels and the predetermined height is 40 pixels), which is not limited in the present application.

[0067] S22, calculating a calibration error value corresponding to the first image and the second image according to errors between matching pixel points in the plurality of sub-images of the first image and the plurality of sub-images of the second image. For example, a plurality of sub-image matching errors are calculated according to matching pixel points in the plurality of sub-images of the first image and the plurality of sub-images of the second image, and a calibration error value corresponding to the first image and the second image is determined according to the plurality of sub-image matching errors (for example, the average of the plurality of sub-image matching errors is calculated as the calibration error value of the first image and the second image).

[0068] For example, the first image is a left-eye image and the second image is a right-eye image. Correspondingly, the plurality of sub-images of the first image include a plurality of left-eye sub-images, and the plurality of sub-images of the second image include a plurality of right-eye sub-images. Further, taking a first left-eye sub-image in the plurality of left-eye sub-images and a first right-eye sub-image in the plurality of right-eye sub-images as an example, a plurality of error values between a plurality of pairs of matching pixel points in the first left-eye sub-image and the first right-eye sub-image are calculated (the calculation method of the error value between the matching pixel points can refer to the calculation method of the error in the Y-axis direction between the two pairs of matching pixel points in the foregoing embodiment), and the average of the plurality of error values is determined as the matching error of the first left-eye sub-image and the first right-eye sub-image. Then, the matching error values corresponding to the remaining sub-images in the plurality of left-eye sub-images and the plurality of right-eye sub-images are calculated in the same way.

[0069] S23, determining that the first camera and the second camera are abnormal in calibration when the calibration error value exceeds a preset error threshold.

[0070] In the embodiments of the present application, the whole image (the first image and the second image) is divided into a plurality of sub-images, and then the matching pixel points in each sub-image are used to determine the calibration error, so as to determine whether the first camera and the second camera are calibrated abnormally, thereby improving the accuracy of the determination result.

[0071] In some embodiments, the binocular camera self-calibration method further comprises: determining a predetermined number of pixel points for each sub-image of the first image; and determining a predetermined number of pixel points for each sub-image of the second image. For example, the method for determining a predetermined number of pixel points for each sub-image of the first image comprises:

[0072] Taking the first sub-image of the first image as an example, a pixel point in the first image is obtained, and it is determined whether the pixel point falls into the first sub-image region of the first image;

[0073] If yes, it is further determined whether the pixel points that have fallen into the first sub-image region have reached a predetermined number; if no, the pixel point is determined as a pixel point of the first sub-image region; if yes, one pixel point is selected and deleted from the pixel points that have fallen into the first sub-image region, and the pixel point is determined as a pixel point of the first sub-image region;

[0074] If no, another pixel point in the first image is obtained and the same processing steps are performed;

[0075] After the predetermined number of pixel points of the first sub-image in the first image are collected, the predetermined number of pixel points of the remaining sub-images in the first image are collected in the above-mentioned manner.

[0076] For example, the bucket collector is used to realize the screening and collecting process of the pixel points, and the specific implementation is as follows: the bucket collector is obtained by dividing the original image into a plurality of small regions (buckets), when a pixel point falls into a certain region, the pixel point is tried to be added to the bucket corresponding to the certain region, when the bucket is not full, the pixel point is put into the bucket, and when the bucket is saturated (for example, the number of pixel points added to the bucket has reached a predetermined number), the retention probability p of the pixel point is calculated: in =N max / N add , wherein N max is the maximum capacity of the bucket, N add is the number of all pixel points that try to be added to the bucket, and both are adjustable parameters obtained according to the algorithm result, memory and computing power balance; for a bucket, N max is a fixed value, and N addis obtained according to historical statistics. A 0-1 floating point number is randomly generated (for example, a random number is generated using a uniform distribution, which can be implemented based on a relevant standard library in a corresponding programming language, which is not limited in the present application), and if it is less than the reserved probability p in A pixel point is randomly selected from the pixel points already saved in the bucket and deleted, and a new pixel point is used to replace the deleted pixel point. In this way, it can be ensured that the pixel points collected in the time domain are randomly distributed.

[0077] Correspondingly, the matching errors of the plurality of sub-images are calculated according to the matching pixel points in the plurality of sub-images of the first image and the plurality of sub-images of the second image, including: calculating the matching errors of the plurality of sub-images according to the matching pixel points in the predetermined number of pixel points of each sub-image of the first image and the predetermined number of pixel points of each sub-image of the second image.

[0078] Exemplarily, the first image is a left eye image, and the second image is a right eye image. Correspondingly, the plurality of sub-images of the first image includes a plurality of left eye sub-images, and the plurality of sub-images of the second image includes a plurality of right eye sub-images. Further, taking a first left eye sub-image in the plurality of left eye sub-images and a first right eye sub-image in the plurality of right eye sub-images as an example. A plurality of error values between a plurality of pairs of matching pixel points in the predetermined number of pixel points of the first left eye sub-image and the predetermined number of pixel points of the first right eye sub-image are calculated (wherein the calculation method of the error value between the matching pixel points can refer to the calculation method of the error in the Y-axis direction between the two pairs of matching pixel points in the foregoing embodiment), and the average value of the plurality of error values is determined as the matching error of the first left eye sub-image and the first right eye sub-image. Then the matching error values corresponding to the remaining sub-images in the plurality of left eye sub-images and the plurality of right eye sub-images are calculated in the same way.

[0079] The inventors found in the process of implementing the present application that noise matching feature points can be introduced in the process of obtaining Harris feature points from the first image and the second image and obtaining matching feature points (for example, left-right eye matching pixel point pairs) in the first image and the second image using the KLT (Kanade-Lucas-Tomasi) tracking algorithm. The usual method is to use the fundamental ransac (Random Sample Consensus, an iterative algorithm for estimating the parameters of a mathematical model from a data set containing outliers, used in computer vision and robotics, such as image registration, three-dimensional reconstruction, etc.), but this way takes a long time to call and occupies resources. In order to avoid the influence of the above-mentioned noise matching feature points on the accuracy of determining whether the first camera and the second camera are abnormally calibrated, the inventors propose the following scheme:

[0080] Before dividing the first image and the second image into a plurality of sub-images, the following steps are further performed: determining abnormal pixel points in the first image and the second image, and filtering out the abnormal pixel points. Illustratively, a plurality of pixel point error values of a plurality of pairs of matching pixel points in the first image and the second image are determined; and according to the plurality of pixel point error values, abnormal pixel points in the plurality of pairs of matching pixel points are determined.

[0081] Illustratively, since the sensor can be considered as having a calibrated intrinsic parameter, the vehicle vibration caused change in the extrinsic parameter usually does not deviate too much, so a pixel point error value of a pair of matching pixel points is randomly taken as a center error, and then the difference between the pixel point error values of all other matching pixel points and the center error is determined, and if the difference is within a certain range, the matching pixel point is considered as an inlier. For example, there are three pairs of left-right matching pixel points: (pl1, pr1), (pl2, pr2), and (pl3, pr3); wherein the error of (pl1, pr1) is the y difference y1 of pl1 and pr1, the error of (pl2, pr2) is the y difference y2 of pl2 and pr2, and the error of (pl3, pr3) is the y difference y3 of pl3 and pr3; the y difference y1 of (pl1, pr1) is randomly selected as the center error; then the differences between y2 and y3 and y1 are determined, and if the differences are within a certain range, the corresponding matching pixel point pair is considered as an inlier, otherwise the corresponding matching pixel point is considered as an outlier. Finally, the state when the inliers are the most is selected as the final inlier point. For example, when the y difference y2 of (pl2, pr2) is selected as the center error, all other matching pixel points are determined as 300 inliers and 100 outliers, then the y difference y2 of (pl2, pr2) is selected as the center error, and all matching pixel points having a difference within a certain range from y2 are the final inlier points (301 in total).

[0082] As shown in FIG. 5, it is a flowchart of another embodiment of a binocular camera self-calibration method of the present application. When it is determined that the first camera and the second camera are calibrated abnormally, the extrinsic parameters of the first camera and the second camera are corrected, including:

[0083] S31, obtaining an initial rotation matrix corresponding to the first camera and the second camera by using epipolar constraint.

[0084] S32, searching for a target disparity deviation value in a preset disparity deviation range by using a grid search method, so as to correct the initial rotation matrix.

[0085] For step S31, a first camera and the second camera initial rotation matrix R is obtained by using epipolar constraint with binocular stereo track points, which can optimize the epipolar error to the minimum, but the convergence in the disparity (yaw of horizontal disparity and pitch of vertical disparity) direction is not good (cost is not sensitive to it), so the grid search in the next step is needed to optimize the deviation in the disparity direction. The pitch refers to the angle between the optical axis and the ground when the camera is installed. When the camera is installed towards the front, the pitch is 0.

[0086] After rectification, the feature points of the left and right eyes satisfy the following epipolar constraint: x0 T Ex1=0, where x0, x1 are the normalized coordinates of the matching points of the left and right eyes. The binocular coordinate relationship is: x1=Rx0+t, and the essential matrix is: E=[t X ]R, where [t X ] represents the skew-symmetric matrix corresponding to the translation vector. The epipolar constraint for the left and right eyes can be obtained as follows:

[0087] Let x′1=[t X ]x1, and the formula (1) can be obtained by substituting it into the formula (1):

[0088] x′1Rx0=0,

[0089] The following optimization problem is designed in the application: the geometric meaning is to optimize the perpendicular distance from the point to the straight line:

[0090] Since the modulus is invariant after the rotation of a matrix, the denominator can eliminate R, and after simplification, it is equivalent to optimizing e:

[0091] The residual error of the i-th observation point is:

[0092] Ceres (an open source tool for solving nonlinear optimization problems) is used for solving, and the CPU load is high. The analytical form of the Jacobian matrix is given as follows:

[0093] The optimization objective function has a geometric meaning of the perpendicular distance from the feature point to the epipolar line. Solving the optimization objective function can obtain the initial rotation matrix R. When the binocular yaw angle deviates, the epipolar line can still be aligned, and the objective function is not sensitive. Therefore, only by optimizing the epipolar line, the yaw direction rotation cannot be well calibrated. Therefore, a grid search method is used to search for a target disparity deviation value in a preset disparity deviation range, so as to correct the initial rotation matrix.

[0094] As shown in FIG. 6, it is a flowchart of another embodiment of a binocular camera self-calibration method of the application. The preset disparity deviation range includes n preset disparity deviation values. In this embodiment, step S32, the grid search method is used to search for a target disparity deviation value in the preset disparity deviation range, including:

[0095] S321, for the i th preset disparity deviation value, the coordinate information of a plurality of pixel points at two different times is calculated.

[0096] For example, for the i th preset disparity deviation value, the first coordinate information of a plurality of pixel points at the first time is calculated; according to the relative pose information of the vehicle at the first time and the second time, and the first coordinate information, the second coordinate information of a plurality of pixel points at the second time is determined.

[0097] As shown in FIG. 7, it is a schematic diagram of an embodiment of the grid search used in the application. A plurality of groups of 3 frames of images shown in FIG. 7 are collected, and a grid search is performed on the disparity shift in a range. For each disparity shift, the 3D coordinates of the track points at the t1 time can be calculated, and the number of inliers can be calculated by projecting the 3D coordinates to the left eye at the t2 time. Finally, the disparity shift with the most inliers is selected as the optimal disparity shift, so as to determine the final binocular R.

[0098] Wherein, the grid search is a parameter adjustment method: among all candidate parameter selections, every possibility is tried by loop iteration, and the best parameter is the final result. The principle is like finding the maximum value in an array (why it is called grid search, because taking a model with two parameters as an example, parameter a has 3 possibilities, and parameter b has 4 possibilities, all possibilities can be represented as a 3*4 table, and each cell is a grid, and the loop process is like traversing and searching in each grid, so it is called grid search).

[0099] S322. For the i-th preset disparity deviation value, calculate the i-th group of error values ​​based on the coordinate information of the multiple pixels at two different times; where i takes values ​​from 1 to n.

[0100] S323. Determine the target disparity deviation value based on the distribution of error values ​​from group 1 to group n.

[0101] For example, the number of error values ​​less than a preset error threshold among the multiple error values ​​of the m-th group of error values ​​is determined, where m takes the value from 1 to n, resulting in n error value counts; the preset disparity deviation value corresponding to the maximum value among the n error value counts is determined as the target disparity deviation value.

[0102] In some embodiments, it is assumed that the binocular baseline has been aligned by the epipolar constraint, but there is a bias in the disparity direction. This bias is calculated using the following formula: bf / (d1+bias)-bf / (d2+bias)=δz.

[0103] For example, b is the baseline length of the binocular camera, f is the focal length of the binocular camera, the disparity of a certain feature point is d1, after the car moves a certain distance, the disparity of that point becomes d2, and δz is the change in depth at that point. When the car is traveling in a perfectly straight line, δz can be simply considered as the distance the car has traveled.

[0104] By continuously tracking the feature points and adding new feature points, multiple sets (d1, d2, δz) are obtained. For each set of observations, a bias can be calculated. The bias distribution is statistically analyzed, and the most likely value is selected.

[0105] Figure 8 shows a schematic diagram of the depth changes of feature points at times t1 and t2 in this application. Exemplarily, the depth change δz of the feature points is obtained by the following method:

[0106] The 3D coordinates of feature point P at time t1 are calculated using the original bi-target positioning parameters (to obtain z1). The relative poses of t1 and t2 are projected onto time t2 to obtain the 3D coordinates at time t2 (to obtain z2), thus obtaining δz.

[0107] For example, for a selected disparity shift, the triangulation method, commonly used in computer vision, can be used to obtain the 3D coordinates of the point at time t1. Then, using the vehicle's motion information between times t1 and t2, the 3D coordinates at time t1 are transformed to time t2. Finally, the difference between the 3D coordinates projected onto the point at time t2 and the 2D coordinates obtained using the KLT algorithm can be used to determine the error value.

[0108] Exemplarily, assuming that there are 100 points, 10 candidate parameters of disparity shift are selected, and after enumeration of 90 points, it is considered that the error is minimum when disparity shift = X1; 5 points consider that the error is minimum when disparity shift = X2, and other points consider that the error is minimum when disparity shift = X3, X4, …. Then it is considered that the correct value of disparity shift at this time should be X1.

[0109] In some embodiments, the application further provides a computer device, comprising a memory, a processor and a computer program stored in the memory, and the processor executes the computer program to implement the steps of the binocular camera self-calibration method in any of the above embodiments.

[0110] In some embodiments, the processor executes the computer program to implement the following steps: when it is detected that the image captured by the first camera and / or the second camera has no valid information, a prompt information is generated to remind the user. In this embodiment, when the lens is dirty due to rain, haze, etc., and the driver has not cleaned it in time, the image has no valid information. At this time, the system extracts the information of the image quality monitoring module to remind the user to clean it in time. Exemplarily, the valid information can be a detection object (for example, a front vehicle, a pedestrian, a lane line, a signboard and a road edge, etc.) in the image. Correspondingly, when the image captured by the first camera and / or the second camera has no detection object, it is considered that there is no valid information, so that it is determined that the lens of the first camera and / or the second camera is dirty or blocked, and needs to be cleaned.

[0111] In some embodiments, the processor executes the computer program to implement the following steps: recording the change of the extrinsic parameter of the first camera and the second camera, and generating an extrinsic parameter change curve.

[0112] The self-calibration computer device for the non-rigid binocular camera in the embodiments of the application can monitor the calibration error in real time. For example, after the vehicle runs for 1 day, due to vehicle vibration, temperature difference change, etc., the extrinsic parameter between the binocular cameras changes slightly. The application can detect and self-calibrate in a short time (for example, the change can be found in 0.1 seconds, and the self-calibration can be completed in 0.5 seconds), update the calibration parameter, ensure the normal work of the system, and record the change of the extrinsic parameter, and display the extrinsic parameter change curve on the HMI human-computer interaction interface.

[0113] In some embodiments, the application further provides a computer readable storage medium, which stores a computer program / instruction, and the computer program / instruction is executed by a processor to implement the steps of the binocular camera self-calibration method in any of the above embodiments.

[0114] In some embodiments, the present application also provides a computer program product comprising computer programs / instructions for implementing the steps of the binocular camera self-calibration method according to any of the above embodiments when executed by a processor.

[0115] In some embodiments, the present application also provides a mobile platform installed with the computer device according to any of the above embodiments.

[0116] In some embodiments, the first camera and the second camera are installed at the front of the mobile platform. For example, in the case of a mobile platform being a vehicle, the first camera and the second camera are installed on the front windshield of the vehicle. In some embodiments, the baseline of the first camera and the second camera is determined according to the height and / or width of the detected object.

[0117] In some embodiments, the baseline of the first camera and the second camera ranges from 0.2m to 1.2m, and the field of view angle of the first camera and the second camera is 120°.

[0118] As shown in FIG. 9, it is a schematic diagram of the baseline distance and the observable length of the cut-in vehicle in the cut-in scenario according to the present application. Exemplarily, the installation design constraints of the non-rigid binocular camera according to the present application will be described below in combination with FIG. 9: the baseline of the non-rigid binocular camera ranges from 0.2m to 1.2m, the left camera is maximally offset to the left by b = 0.51m relative to the current integrated installation mode, and the field of view angle of the left camera and the right camera is FOV = 120°. Wherein,

[0119] When b = 0.51m (baseline = 1.2m), the observable length of the right vehicle can be calculated to decrease by d = b * tan(30°) = 0.30m, and similarly, the observable length of the left vehicle increases by d = 0.30m. Assuming that the cut-in vehicle is just pressed to the line, the lateral distance to the center of the lane is 3.75m / 2 = 1.875m. When the head of the right cut-in vehicle is 1m away from the head of the host vehicle, the observable vehicle length decreases from 1.85m to 1.55m. When the head of the right cut-in vehicle is 1.5m away from the head of the host vehicle, the observable vehicle length decreases from 2.35m to 2.05m.

[0120] When b = 0.11m (baseline = 0.4m), the observable length of the right vehicle can be calculated to decrease by d = b * tan(30°) = 0.06m, and similarly, the observable length of the left vehicle increases by d = 0.06m. Assuming that the cut-in vehicle is just pressed to the line, the lateral distance to the center of the lane is 3.75m / 2 = 1.875m. When the head of the right cut-in vehicle is 1m away from the head of the host vehicle, the observable vehicle length decreases from 1.85m to 1.79m. When the head of the right cut-in vehicle is 1.5m away from the head of the host vehicle, the observable vehicle length decreases from 2.35m to 2.29m.

[0121] Further, from the above installation position design analysis, the lateral position of the binocular camera has little effect on the sensing range capability. In some embodiments, point cloud level objects are detected, which need to meet the following conditions:

[0122] struct obstacle{

[0123] float Ay; / / object height

[0124] float Ax; / / object width

[0125] } obs

[0126] float h = 1.554f; / / camera optical center to ground intercept

[0127] float lambda = 1.5f; / / detection significance control coefficient

[0128] float p = 1.5f; / / binocular camera minimum detectable pixel size, corresponding to binocular camera detection capability;

[0129] · Obstacle segmentation capability

[0130] · Binocular detection capability

[0131] Where b is the baseline length of the binocular camera, f is the focal length of the binocular camera, and lambda is the obstacle detection threshold (determined by the detection algorithm, which is not limited in the present application).

[0132] Combining the two formulas, we get: And when the problem is "slim type", such as isolation columns, single ice cream tubes, etc., the obstacle detection distance is more limited by formula (2). The binocular detection capability, assuming the length-width ratio of the ice cream tube is 3:1, then after the baseline exceeds 0.153m, theoretically increasing the baseline will no longer improve the obstacle detection capability. Considering that the structure installation cannot guarantee the pitch to be 0, and the value of lambda, the actual installation baseline can be taken as 0.4-0.6m.

[0133] As shown in FIG. 10, for different baseline values in the present application, the ranging error changes with the depth. In FIG. 10, two curves are shown when the baseline is 0.4m and 0.18m, respectively. The curve with smaller slope corresponds to the baseline value of 0.4m, and the curve with larger slope corresponds to the baseline value of 0.18.

[0134] As shown in FIG. 11, it is a schematic diagram of an embodiment of the binocular camera self-calibration system of the present application. It includes: an odometer Odo, which calculates the distance δz of the vehicle movement according to the detected wheel speed data; a first camera and a second camera, which are used to collect image data; a camera driving and wheel speed sensor data forwarding module HAL, which is used to send data from the sensor to the system; a gpu_service, which is used to convert the image from the raw data to the format that can be used by the algorithm; a feature point extraction and tracking module KLT, the output result of which is shared by the calibration module and the perception module, which is used for feature point extraction and tracking of the image data; an OSCC module, which is used to execute the binocular camera self-calibration method of the present application; a Maphex-generation, the input of which is the updated external parameter of the binocular camera of the OSCC module, and the output is a pre-check table of binocular distortion correction, which is updated online to the gpu_service, which is used to serve the gpu_service. The gpu_service and the pre-check table of distortion correction are used to pre-process the image. The specific pre-processing method can refer to the related prior art, which is not limited in the present application; a Perception: perception module, which can perform depth calculation according to the output result of the feature point extraction and tracking module KLT.

[0135] As shown in FIG. 12, it is a schematic diagram of an embodiment of the binocular camera self-calibration method combined with interactive design in the present application. The calibration checking tool module is used to execute the binocular camera self-calibration method in the foregoing embodiments of the present application, and the information such as the frequency and amplitude of the change of the external parameter R of the binocular camera is displayed through the HMI. The display method includes but is not limited to the update curve of the calibration external parameter R and the calibration abnormal user reminder (camera dirt, loose, etc.).

[0136] It should be noted that, for the foregoing method embodiments, in order to simply describe, they are all expressed as a series of actions, but those skilled in the art should know that the present application is not limited by the action sequence described, because according to the present application, certain steps can be performed in other order or simultaneously.

[0137] FIG. 13 is a schematic diagram of the hardware structure of the computer device for executing the binocular camera self-calibration method according to another embodiment of the present application. As shown in FIG. 13, the device includes:

[0138] one or more processors 1310 and memories 1320, of which one processor 1310 is taken as an example in FIG. 13. The device for executing the binocular camera self-calibration method can also include: an input device 1330 and an output device 1340.

[0139] The processor 1310, the memory 1320, the input device 1330 and the output device 1340 can be connected through a bus or other means, of which the connection through the bus is taken as an example in FIG. 13.

[0140] The memory 1320, as a non-volatile computer readable storage medium, can be used to store non-volatile software programs, non-volatile computer executable programs and modules, such as program instructions / modules corresponding to the binocular camera self-calibration method in the embodiments of the present application. The processor 1310 executes various function applications and data processing of the server by running the non-volatile software programs, instructions and modules stored in the memory 1320, that is, implements the binocular camera self-calibration method in the above method embodiments. The memory 1320 can include a program storage area and a data storage area, wherein the program storage area can store an operating system and at least one application required by a function; the data storage area can store data created according to the use of the binocular camera self-calibration device, etc. In addition, the memory 1320 can include a high-speed random access memory, and can also include a non-volatile memory, such as at least one magnetic disk storage device, a flash memory device, or other non-volatile solid-state memory device. In some embodiments, the memory 1320 can optionally include a memory remotely arranged with respect to the processor 1310, and these remote memories can be connected to the binocular camera self-calibration device through a network. Examples of the above network include but are not limited to the Internet, an intranet, a local area network, a mobile communication network, and a combination thereof.

[0141] The input device 1330 can receive input digital or character information, and generate signals related to user settings and function control of the binocular camera self-calibration device. The output device 1340 can include a display device such as a display screen.

[0142] The one or more modules are stored in the memory 1320, and when executed by the one or more processors 1310, perform the binocular camera self-calibration method in any of the above method embodiments.

[0143] The above product can perform the method provided by the embodiments of the present application, and has the corresponding function modules and beneficial effects of performing the method. Technical details not described in detail in the embodiments can be referred to the method provided by the embodiments of the present application.

[0144] The computer device of the embodiments of the present application exists in various forms, including but not limited to:

[0145] (1) Mobile communication device: the feature of this kind of device is to have mobile communication function, and to provide voice and data communication as the main target. This kind of terminal includes: smart phone (such as iPhone), multimedia phone, functional phone, and low-end phone, etc.

[0146] (2) Ultra-mobile personal computer device: This kind of device belongs to the category of personal computer, has the functions of calculation and processing, and generally has the feature of mobile Internet. This kind of terminal includes PDA, MID and UMPC device, etc., such as iPad.

[0147] (3) Server: A device providing calculation service, the constitution of the server includes processor, hard disk, memory, system bus, etc. The server is similar to the general computer architecture, but since it needs to provide high reliable service, it has higher requirements in processing capability, stability, reliability, security, scalability, manageability, etc.

[0148] (4) Other electronic devices with data interaction function.

[0149] Through the description of the above embodiments, those skilled in the art can clearly understand that each embodiment can be realized by means of software plus a general hardware platform, and of course it can also be realized by hardware. Based on such understanding, the above technical solutions or the part that contributes to the related art can be embodied in the form of software product, which can be stored in a computer readable storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., and includes a plurality of instructions to make a computer device (which can be a personal computer, server, or network device, etc.) execute the method described in each embodiment or some part of the embodiment.

[0150] Finally, it should be noted that: the above examples are only used to illustrate the technical solutions of the present application, and not to limit them; although the present application has been described in detail with reference to the foregoing examples, those skilled in the art should understand that: it can still modify the technical solutions recorded in the foregoing examples, or make equivalent replacement for some technical features; and these modifications or replacements do not make the essence of the corresponding technical solutions deviate from the spirit and scope of the technical solutions of the embodiments of the present application.

Claims

1. A self-calibration method for a binocular camera, characterized in that, The binocular camera includes a first camera and a second camera, wherein there is no rigid connection between the first camera and the second camera, and the method includes: Acquire a first image captured by the first camera and a second image captured by the second camera; Determine whether the first camera and the second camera are calibrated abnormally based on the first image and the second image; When the calibration of the first camera and the second camera is determined to be abnormal, the external parameters of the first camera and the second camera are corrected.

2. The method according to claim 1, characterized in that, The step of determining whether the first camera and the second camera are calibrated abnormally based on the first image and the second image includes: The first image and the second image are each divided into multiple sub-images; Based on the error between matching pixels in multiple sub-images of the first image and multiple sub-images of the second image, calculate the calibration error value corresponding to the first image and the second image; When the calibration error value exceeds a preset error threshold, the calibration of the first camera and the second camera is determined to be abnormal.

3. The method according to claim 2, characterized in that, Based on the errors between matching pixels in multiple sub-images of the first image and multiple sub-images of the second image, a calibration error value corresponding to the first image and the second image is calculated, including: Based on the matching pixel points in multiple sub-images of the first image and multiple sub-images of the second image, calculate the corresponding multiple sub-image matching error; Based on the matching errors of the multiple sub-images, a calibration error value corresponding to the first image and the second image is determined.

4. The method according to claim 3, characterized in that, Also includes: A predetermined number of pixels are determined for each sub-image of the first image; a predetermined number of pixels are determined for each sub-image of the second image; Based on the matching pixel points in multiple sub-images of the first image and multiple sub-images of the second image, calculate the corresponding multiple sub-image matching errors, including: Based on a predetermined number of pixels in each sub-image of the first image and a predetermined number of matching pixels in each sub-image of the second image, the corresponding sub-image matching errors are calculated.

5. The method according to claim 3 or 4, characterized in that, Determining the calibration error value corresponding to the first image and the second image based on the multiple sub-image matching errors includes: calculating the average value of the multiple sub-image matching errors and determining it as the calibration error value of the first image and the second image.

6. The method according to any one of claims 2-4, characterized in that, Before dividing the first image and the second image into multiple sub-images, the method further includes: identifying abnormal pixels in the first image and the second image, and filtering out the abnormal pixels.

7. The method according to claim 6, characterized in that, Determining abnormal pixels in the first image and the second image includes: Determine multiple pixel error values ​​for multiple pairs of matching pixels in the first image and the second image; determine abnormal pixels among the multiple pairs of matching pixels based on the multiple pixel error values.

8. The method according to any one of claims 1-4, characterized in that, When the calibration of the first camera and the second camera is determined to be abnormal, the extrinsic parameters of the first camera and the second camera are corrected, including: An initial rotation matrix corresponding to the first camera and the second camera is obtained by using epipolar constraints; a target disparity deviation value is searched within a preset disparity deviation range using a grid search method to correct the initial rotation matrix.

9. The method according to claim 8, characterized in that, The preset parallax deviation range includes n preset parallax deviation values; The method of searching for the target disparity deviation value within a preset disparity deviation range using a grid search includes: For the i-th preset disparity deviation value, calculate the coordinate information of multiple pixels at two different times; For the i-th preset disparity deviation value, the i-th group of error values ​​is calculated based on the coordinate information of the multiple pixels at two different times; where i takes values ​​from 1 to n. The target disparity deviation value is determined based on the distribution of error values ​​from group 1 to group n.

10. The method according to claim 9, characterized in that, For the i-th preset disparity deviation value, calculate the coordinate information of multiple pixels at two different times, including: For the i-th preset disparity deviation value, calculate the first coordinate information of multiple pixels at the first time point; Based on the relative pose information of the vehicle at the first and second time points, and the first coordinate information, the second coordinate information of multiple pixels at the second time point is determined.

11. The method according to claim 9, characterized in that, The target disparity deviation value is determined based on the distribution of error values ​​from group 1 to group n, including: determining the number of error values ​​less than a preset error threshold among multiple error values ​​in group m, where m takes values ​​from 1 to n, resulting in n error value counts; and determining the preset disparity deviation value corresponding to the maximum value among the n error value counts as the target disparity deviation value.

12. A computer device, comprising a memory, a processor, and a computer program stored in the memory, characterized in that, The processor executes the computer program to implement the steps of the method according to any one of claims 1-11.

13. The computer device according to claim 12, characterized in that, The processor executes the computer program to perform the following steps: when it detects that the images captured by the first camera and / or the second camera have no valid information, it generates a prompt message to remind the user.

14. The computer device according to claim 12 or 13, characterized in that, The processor executes the computer program to perform the following steps: record the changes in extrinsic parameters of the first camera and the second camera, and generate extrinsic parameter change curves.

15. A computer-readable storage medium having a computer program / instructions stored thereon, characterized in that, When the computer program / instructions are executed by the processor, they implement the steps of the method described in any one of claims 1-11.

16. A computer program product comprising a computer program / instructions, characterized in that, When the computer program / instructions are executed by the processor, they implement the steps of the method described in any one of claims 1-11.

17. A mobile platform, characterized in that, The device is equipped with a first camera, a second camera, and a computer device as described in any one of claims 12-14.

18. The mobile platform according to claim 17, characterized in that, The first camera and the second camera are mounted in front of the mobile platform.

19. The mobile platform according to claim 17, characterized in that, The baselines of the first camera and the second camera are determined based on the height and / or width of the object being detected.

20. The mobile platform according to claim 17, characterized in that, The baseline values ​​of the first camera and the second camera range from 0.2 to 1.2 m, and the field of view of the first camera and the second camera is 120°.

Images