A real-time binocular camera self-calibration method

Abstract

The application discloses a real-time binocular camera self-calibration method, comprising the following steps: 1, acquiring two adjacent images obtained by a binocular camera during movement of a mobile device according to a sampling period, and marking the two adjacent images as a first image and a second image respectively; 2, extracting a first static key point in the first image and a second static key point in the second image respectively; 3, calculating an angle rotation matrix R and a coordinate translation matrix T according to IMU information during movement of the mobile device; 4, performing first coordinate transformation on the second static key point by using a three-dimensional coordinate transformation matrix, and performing second coordinate transformation on the first static key point by using an inter-frame motion transformation matrix; and 5, introducing a correction parallax offset into the first coordinate transformation result and the second coordinate transformation result respectively, generating a correction function calculation formula, and calculating the correction parallax offset under a preset condition; the correction parallax offset is used for correcting a parallax value d of the binocular camera.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of image processing, and in particular to a real-time binocular camera self-calibration method. BACKGROUND

[0002] The principle of binocular camera: it uses the direct difference of the transverse coordinates of the target point in the left and right views (i.e. disparity) and the inverse proportional relationship between the distance of the target point to the imaging plane to realize distance measurement. That is, if the disparity of the target point is known, the distance Z of the target point to the camera can be known. Specifically, the solving method of Z is as follows: Z = fB / D, wherein f represents the focal length of the camera, B represents the center distance of the binocular camera, and D represents the disparity of the target point.

[0003] Binocular rectification requires that binocular rectification is based on the monocular intrinsic parameter data (focal length, imaging origin, distortion coefficient) obtained after camera calibration and the relative position relationship of binocular (rotation matrix and translation vector), respectively, to eliminate distortion and line alignment of left and right views, so that the imaging origin coordinates of left and right views are consistent, the optical axes of two cameras are parallel, the left and right imaging planes are coplanar, and the epipolar lines are aligned. Thus, any pixel point in one image (such as the right view) and the pixel point corresponding to the same object in the other image (such as the left view) must have the same line number, and only one-dimensional search is needed at the corresponding line number of left and right views.

[0004] For binocular cameras, during the entire life cycle after factory calibration, due to environmental temperature and other factors, deviation may occur in the alignment of left and right views. This phenomenon is usually called binocular disparity degradation, which will cause errors in disparity matching and affect the distance measurement function.

[0005] For example, patent CN112861940A based on a neural network method proposes a binocular disparity estimation model, which includes: obtaining a sample left image, a sample right image, and a disparity label; taking the sample left image and the sample right image as the input of a binocular disparity estimation model, the binocular disparity estimation model includes a feature extraction network module, a matching cost calculation module, a single-scale cost aggregation module, a multi-scale cost aggregation module, and a disparity regression model; calculate the error between the disparity label and the estimated disparity to train the binocular disparity estimation model, and solve the problems of large calculation amount, long time consumption, poor effect of object edges and non-textured areas. Although the error between the disparity label and the estimated disparity is calculated to improve the training of the disparity estimation model, the calculation effect of the binocular disparity estimation algorithm is improved, but the influence of binocular disparity degradation is not considered in this process.

[0006] And patent CN111225201A proposes a parallax correction method, including: collecting two original images including a target object through a binocular camera; determining the first parallax of the target object in the imaging area of the two original images; adjusting the position of the imaging area in the two original images according to the first parallax and the preset parallax; determining the target image based on the imaging area after position adjustment. This patent can correct the parallax of the binocular camera, avoid the additional calculation caused by calibrating the binocular camera to correct the parallax, and improve the imaging consistency of the binocular camera. In the above patent, only the position of the imaging area in the two original images is adjusted according to the first parallax and the preset parallax, and then adjusted and determined, but when the binocular parallax degenerates, the preset parallax is essentially changing, and this changing value is difficult to determine, at this time the imaging will also have errors, and the correction is not accurate.

[0007] In summary, the parallax matching error caused by binocular parallax degradation and the inaccurate calibration result in the prior art still need to be further solved, and it is of great significance to solve the above technical problems. SUMMARY

[0008] To this end, the present application proposes the following technical solutions:

[0009] A real-time binocular camera self-calibration method, the binocular camera is installed on a mobile device, comprising the following steps:

[0010] Step 1: According to the sampling period, obtain two adjacent frames of images obtained by the binocular camera during the movement of the mobile device, and mark the two adjacent frames of images as the first image and the second image respectively;

[0011] Step 2: Extract the first static key points in the first image and the second static key points in the second image respectively;

[0012] Step 3: According to the IMU information during the movement of the mobile device, calculate the inter-frame motion transformation matrix, which includes the angle rotation matrix R and the coordinate translation matrix T;

[0013] Step 4: Perform first coordinate transformation on the second static key points using the three-dimensional coordinate transformation matrix, and perform second coordinate transformation on the first static key points using the inter-frame motion transformation matrix;

[0014] Step 5: Introduce a correction parallax offset in the first coordinate transformation result and the second coordinate transformation result respectively to generate a correction function calculation formula, and calculate the correction parallax offset that meets the preset condition;

[0015] Wherein, the correction parallax offset is used to correct the parallax value d of the binocular camera.

[0016] Preferably, before step 4, further comprising:

[0017] A threshold value for extracting static key points is preset, denoted as a preset threshold value, the first static key points and the second static key points selected in step 1 are accumulated in quantity, and it is judged whether the quantity of the first static key points and the second static key points is greater than or equal to the preset threshold value, if yes, step 4 is executed, if no, the image is discarded and step 1 is re-executed.

[0018] Preferably, the preset condition in step 5 is that the value result of the correction function calculation formula is 0.

[0019] Step 5 specifically includes:

[0020] The first static key points and the second static key points are paired;

[0021] The paired first static key points and the second static key points are respectively subjected to first coordinate transformation and second coordinate transformation, and a correction parallax offset is introduced, and the calculation formula corresponding to the correction function F(offset) calculation formula is:

[0022]

[0023] In the formula, B represents the center distance of the left and right cameras in the binocular camera; d i,t , d i,t+1 is the parallax value corresponding to the i-th group of static key points at the t+1 and t moments; c x , c y represents the offset of the camera optical axis in the image pixel coordinate system, in units of pixels; f represents the focal length of the camera; (u i,t , v i,t ), (u i,t+1 , v i,t+1 ) are the pixel coordinates corresponding to the i-th group of static key points at the t+1 and t moments;

[0024] When the value result of the correction function calculation formula is 0, the correction parallax calculation result corresponding to each pair of static key point groups is denoted as a correction parallax intermediate value;

[0025] The correction parallax intermediate values are accumulated, the average value of the correction parallax intermediate values is calculated, and the average value calculation result is denoted as a correction parallax offset.

[0026] Preferably, the target function corresponding to the preset condition is:

[0027]

[0028] Step 5 specifically includes:

[0029] pairing the first static key point and the second static key point;

[0030] The paired first static key point and the second static key point are respectively subjected to first coordinate transformation and second coordinate transformation, and a correction parallax offset is introduced, and the calculation formula corresponding to the correction function F(offset) is as follows:

[0031]

[0032] In the formula, B represents the center distance of the left and right cameras in the binocular camera; d i,t , d i,t+1 is the parallax value corresponding to the i-th group of static key points at the t+1 and t moments; c x , c y represents the offset of the camera optical axis in the image pixel coordinate system, in units of pixels; f represents the focal length of the camera; (u i,t , v i,t ), (u i,t+1 , v i,t+1 ) are the pixel coordinates corresponding to the i-th group of static key points at the t+1 and t moments;

[0033] The correction parallax offset satisfying the target function is calculated.

[0034] Preferably, in the process of selecting static key points, step 2 also filters the adjacent two frames of images, and the filtering method is:

[0035] Based on the deep learning method, the image information contained in the adjacent two frames of images is recognized, and the region of interest in the adjacent two frames of images is further obtained. When it is determined that the image information in the region of interest is of a first target, the corresponding image in the region of interest is cut out, and then the corresponding image corner points in the adjacent two frames of images after cutting out are recorded as static key points; wherein the first target is a moving object.

[0036] Preferably, in the process of selecting static key points, step 2 filters the adjacent two frames of images, and the filtering method is:

[0037] Based on the deep learning method, the image information contained in the adjacent two frames of images is recognized, and the region of interest in the adjacent two frames of images is further obtained. When it is determined that the image information in the region of interest is of a second target, the image corner points of the corresponding image in the region of interest are obtained and recorded as static key points; wherein the second target is a static object.

[0038] Preferably, the moving device is a vehicle, and the calculation method of the angle rotation matrix R in step 3 is as follows:

[0039]

[0040] wherein p, r, q are respectively the pitch angle, roll angle, yaw angle in the attitude angle.

[0041] Preferably, the solving of the attitude angles r, p, q comprises the following steps:

[0042] Step 3-1, by installing an accelerometer on the vehicle, reading the acceleration values of the accelerometer around the X, Y, Z axes at t+1 time as follows: x，t+1 , α y，t+1 , α z，t+1 And by the following formula to solve the r acc,t+1 and p acc,t+1 at t+1 time:

[0043]

[0044] wherein g represents the value of the gravitational acceleration;

[0045] Step 3-2: by installing an accelerometer and a gyroscope on the vehicle, reading the acceleration values of the accelerometer around the X, Y, Z axes at any t time as follows: x，t , α y，t , α z，t And by the following formula to solve the r acc,t , p acc,t at t time:

[0046]

[0047] Reading the angular velocity ω xt , ω yt , ω zt of the gyroscope around the X, Y, Z axes at any t time, by the following formula to solve the r gyvo , p gyvo , q gyvo at t+1 time:

[0048]

[0049] Step 3-3, based on the results of step 3-1, step 3-2, the attitude is fused:

[0050]

[0051] wherein K represents the proportionality coefficient.

[0052] Preferably, the moving device is a vehicle, and the coordinate translation matrix T in step 3 is calculated by the vehicle speed signal to obtain the relative displacement matrix T within t to t+1 time;

[0053]

[0054] wherein T x and T z are the lateral and longitudinal components of the vehicle speed, respectively, T y is the displacement component in the vertical direction;

[0055] A gyroscope is installed on the vehicle to read the angular velocities ω x，t+1 , ω y，y+1 , ω y，y+1 of the rotation around the X, Y, Z axes at the time t+1, and to solve the matrix T by the following formula:

[0056]

[0057] wherein V represents the vehicle speed; ΔT is the time interval from the time t to the time t+1.

[0058] Preferably, the first static key point and the second static key point are image corner points in the first image and the second image, respectively.

[0059] The beneficial effects of the present application are:

[0060] (1) The present application proposes a real-time binocular camera self-calibration method, which selects image corner points of stationary objects in adjacent two images as key points for correction, accumulates the selected static key points and the corresponding inter-frame motion transformation matrix in the adjacent two images, then calculates the mean value of the accumulated parallax offset, and takes the obtained mean value as the final parallax offset, which ensures the accuracy of the data, and then corrects the parallax value of the binocular camera through the calculated correction parallax offset, solving the problem that the image corner points of moving objects such as pedestrians and vehicles in the selected image information form noise interference to the binocular camera correction result, affecting the accuracy of the binocular camera correction result.

[0061] (2) The present application solves the problem of parallax matching error caused by binocular parallax degradation, greatly improving the measurement accuracy of the binocular camera.

[0062] The above description is only a summary of the technical solutions of the present application, in order to more clearly understand the technical means of the present application, so as to implement according to the content of the specification, and in order to make the above and other purposes, characteristics and advantages of the present application more obvious and easy to understand, the following will be described in detail with the preferred embodiments of the present application and with the help of the accompanying drawings.

[0063] According to the detailed description of the specific embodiments of the present application in the following text combined with the accompanying drawings, those skilled in the art will be more clear about the above and other purposes, advantages and characteristics of the present application. BRIEF DESCRIPTION OF DRAWINGS

[0064] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the prior art, the present application will be further described in detail below with reference to the embodiments in the drawings, but does not constitute any limitation on the present application.

[0065] Figure 1 is a schematic diagram of the step flow of the binocular camera self-calibration method of the present application;

[0066] Figure 2 is a schematic diagram of the coordinate system of the binocular stereo camera parameter calibration of the present application. DETAILED DESCRIPTION

[0067] In order to make the objects, technical solutions and advantages of the embodiments of the present application clearer, the technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are only a part of the embodiments of the present application, rather than all the embodiments of the present application. In the following description, specific details such as specific configurations and components are provided only to help a comprehensive understanding of the embodiments of the present application. Therefore, those skilled in the art should clearly understand that various changes and modifications can be made to the embodiments described herein without departing from the scope and spirit of the present application. In addition, in order to be clear and concise, the description of known functions and structures is omitted in the embodiments.

[0068] The term "and / or" herein is only a description of the association relationship of the associated objects, which means that there can be three kinds of relationships, for example, A and / or B can mean that there are three cases of A alone, B alone, and A and B together. The term "and" herein is a description of another association relationship of the associated objects, which means that there can be two kinds of relationships, for example, A and B can mean that there are two cases of A alone and A and B together. In addition, the character " / " herein generally means that the associated objects before and after the character " / " are in an "or" relationship.

[0069] It should also be noted that, in this document, the relationship terms such as first and second are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply that there is any such actual relationship or order between these entities or operations. Moreover, the terms "include", "contain" or any other variants thereof are intended to cover non-exclusive inclusion.

[0070] Embodiment 1

[0071] The present embodiment provides a real-time binocular camera self-calibration method, which solves the problems of parallax matching error caused by parallax degradation of the binocular camera and inaccurate calibration results.

[0072] For the convenience of understanding, the sampling period is set in the embodiment, that is, the sampling time corresponding to the adjacent two frames of images of the binocular camera is set as t time and t+1 time respectively, and then the adjacent two frames of images at t time and t+1 time are sequentially sampled according to the sampling period, wherein the adjacent two frames of images in the sampling are sequentially recorded as the first image and the second image.

[0073] It should be noted that the size of the sampling period can be artificially set as needed.

[0074] In combination with the accompanying Figure 1 , the Figure 1 is a schematic diagram of the step flow of the binocular camera self-calibration method of the present application. In the embodiment, a vehicle is exemplarily illustrated as a mobile device, wherein the vehicle is installed with a binocular camera, and the binocular camera comprises left and right two cameras. The binocular camera disparity calibration method in the embodiment comprises the following steps:

[0075] Step 1, according to the sampling period, acquiring the adjacent two frames of images obtained by the binocular camera at t time and t+1 time during the driving of the vehicle, and recording the adjacent two frames of images as the first image and the second image respectively, wherein the adjacent two frames of images can be collected by any one of the left and right cameras;

[0076] Step 2, extracting the first static key point in the first image and the second static key point in the second image respectively.

[0077] Preferably, the first static key point and the second static key point in the embodiment are image corner points in the first image and the second image respectively.

[0078] In the embodiment, the image information contained in the adjacent two frames of images is recognized by image recognition, and the recognized image information can be pedestrians, vehicles, lane lines, words on the road, roadside traffic signboards, etc.

[0079] In the embodiment, considering the uniqueness and rotation invariance of the corner points, the corner points are used as static key points in the correction process, so as to match and position the feature values of the corner points between images.

[0080] It should be noted that in the embodiment, the corner points in the first image and the second image can be paired by pixel point matching to form a static key point group, and the specific process is not described again.

[0081] However, it is found that the correction result of the binocular camera is not ideal in the correction process, especially in a complex environment with many pedestrians and vehicles. Through research on image data, researchers found that the image corner points of moving objects such as pedestrians and vehicles would form noise interference to the correction result of the binocular camera.

[0082] However, when trying to select only the image corner points of static objects (such as lane lines, traffic signs) as the correction basis, the accuracy of the correction result of the binocular camera is significantly improved.

[0083] Therefore, in the process of step 2 of static key point selection, filtering processing needs to be performed on the adjacent two frames of images (the first and second images). The filtering processing can adopt the following methods:

[0084] One method is based on a deep learning method, which identifies the image information contained in the adjacent two frames of images, and further obtains the region of interest in the adjacent two frames of images. When it is determined that the image information in the region of interest is a first target, the corresponding image in the region of interest is cut out, and then the corresponding image corner points in the adjacent two frames of images after cutting out are recorded as static key points. The first target can be a moving object such as a pedestrian or a vehicle.

[0085] Another method is based on a deep learning method, which identifies the image information contained in the adjacent two frames of images, and further obtains the region of interest in the adjacent two frames of images. When it is determined that the image information in the region of interest is a second target, the image corner points of the corresponding image in the region of interest are obtained and recorded as static key points. The second target is a static object such as a lane line, a traffic sign, or a character.

[0086] Further, w static key points are selected from the adjacent two frames of images read in step 1, and the static key point pixel coordinate matrices A t+1 , A t of t+1 and t time points are stored.

[0087] The key point pixel disparity sets (d 1,t+1 , d 2,t+1 ... d i,t+1 ... d w,t+1 ) and (d 1,t , d 2,t ... d i,t ... d w,t ) of t+1 and t time points are stored, i=1, 2, …, w, i is the serial number of the key point, that is, the corner points with the same serial number at t+1 and t time points form a static key point group, and the serial number of the static key point group is also i.

[0088]

[0089]

[0090] In the above formula, u i,t+1 represents the X-axis pixel coordinate of any ith static key point at t+1 time, and v i,t+1represents the pixel coordinate of any i-th static key point on the Y-axis at t+1 time;

[0091] u i,t represents the pixel coordinate of any i-th static key point on the X-axis at t time, i,t represents the pixel coordinate of any i-th static key point on the Y-axis at t time;

[0092] d i,t is the disparity value of any i-th static key point at t time, i,t+1 is the disparity value of any i-th static key point at t+1 time.

[0093] Step 3, according to the IMU information in the vehicle driving process, the inter-frame motion transformation matrix is calculated, and the inter-frame motion transformation matrix includes an angle rotation matrix R and a coordinate translation matrix T.

[0094] In step 3, the angle rotation matrix R represents the angle rotation matrix of the camera coordinate system at t time to the camera coordinate system of the binocular camera at t+1 time;

[0095] The angle rotation matrix R from t to t+1 time is calculated as follows:

[0096]

[0097] In formula (3), p, r, and q are respectively the pitch angle, roll angle, and yaw angle in the attitude angle.

[0098] In the process of automatic driving of the vehicle, through the inertial measurement unit IMU or the combination of the accelerometer and the gyroscope installed on the vehicle, the attitude fusion method can be used to obtain the attitude angle and the acceleration of the vehicle. The attitude angles r, p, and q in formula (3) are calculated by the following formula:

[0099]

[0100] It should be noted that the various coordinate systems involved in the present embodiment have been unified and aligned through coordinate system transformation, and the origin of the unified coordinate system is set at the position of the vehicle head. The process of coordinate system transformation is not repeated here.

[0101] Further, the specific calculation of the attitude angles p, r, and q in the angle rotation matrix R in step 3 is as follows:

[0102] Step 3-1, calculate r through the accelerometer acc,t+1 and p acc,t+1 ;

[0103] When the accelerometer rotates a certain attitude, the gravity acceleration will generate corresponding components on the 3 axes of acceleration, the essence is that the (0, 0, g) in the geodetic coordinate system is the coordinate in the new accelerometer coordinate system, and the 3 values read by the accelerometer are the new coordinates (α x , α y , α z ).

[0104]

[0105] Wherein, M z , M y , M x represent the rotation matrix corresponding to the angle of rotation around the Z, Y, X axis coordinate axis at t time; g represents the gravity acceleration value; by solving the above equation, r acc , p acc angle can be solved.

[0106] By installing the accelerometer on the vehicle, the acceleration values of the accelerometer around the X, Y, Z axes at t+1 time are: α x，t+1 , α y，t+1 , α z，t+1 ; by using formula (5), r acc,t+1 , p acc,t+1 at t+1 time can be solved, as follows:

[0107]

[0108] Step 3-2, calculate the attitude angle r gyvo , p gyvo , q gyvo by the gyroscope;

[0109] By the accelerometer and the gyroscope installed on the vehicle, the acceleration values of the accelerometer around the X, Y, Z axes at any t time are read α x，t+1 , α y，t+1 , α z，t+1 , and r acc,t , p acc,t at t time are solved by the following formula:

[0110]

[0111] The angular velocity of rotation around the 3 axes measured by the gyroscope can be obtained by integrating the angular velocity. Specifically, the angular velocity of rotation around the X, Y, Z axes measured by the gyroscope at any t time is ω xt , ω yt , ω zt , and the angular velocity used for attitude update satisfy the following relationship:

[0112]

[0113] Simplify the above formula, you can get r gyvo , p gyvo , q gyvo As follows:

[0114]

[0115] Step 3-3, based on the results of step 3-1, step 3-2, the pose is fused:

[0116]

[0117] Wherein, K is the proportion coefficient. The proportion coefficient needs to be adjusted according to the actual, preferably with 0.4.

[0118] In addition, it should be noted that:

[0119] The accelerometer, gyroscope measured X axis, Y axis, Z axis mentioned in the application are all X axis, Y axis, Z axis in the camera coordinate system:

[0120] The definition of the camera coordinate system is as follows:

[0121] 1) Establish a three-dimensional coordinate system with the depth camera as the origin, and the depth camera as the origin of the camera coordinate system,

[0122] 2) X axis is along the horizontal direction of the depth camera-color camera, and Z axis is perpendicular to X axis and points to the direction of shooting;

[0123] 3) Y axis is perpendicular to X and Z;

[0124] 4) The recommended right-hand rule coordinate system.

[0125] Further, the coordinate translation matrix T in step 3 represents the relative displacement matrix of the camera coordinate system. Since the IMU information cannot provide position information, an additional signal providing position or speed is needed, and the typical signal is the vehicle speed signal or GPS signal. Through the vehicle speed signal, the relative displacement matrix T in t to t+1 can be obtained;

[0126]

[0127] Wherein, T x , T y , T z Is the component of the camera coordinate system origin at t to the camera coordinate system origin at t+1.

[0128] Through the gyroscope installed on the vehicle, the angular velocity ω measured by the gyroscope around the X, Y, Z axes of the vehicle at t+1 is readx，t+1 , ω y，y+1 , ω y，y+1 , the solution formula is as follows:

[0129]

[0130] Wherein, V represents the vehicle speed (can be obtained from the speedometer, directly take t+1 time of vehicle speed), ΔT is the time interval from t time to t+1 time.

[0131] Step 4, the first coordinate transformation is carried out on the second static key point by using the three-dimensional coordinate transformation matrix, and the second coordinate transformation is carried out on the first static key point by using the interframe motion transformation matrix.

[0132] Wherein, before step 4, the number of the first static key point and the second static key point extracted from step 1 is also judged, and the specific is:

[0133] The threshold of the extracted static key point is set in advance, that is, the preset threshold. The first static key point and the second static key point selected from step 1 are accumulated with the first image and the second image, and are corresponding with the corresponding angle rotation matrix R and coordinate translation matrix T, then whether the number of the first static key point and the second static key point is greater than or equal to the preset threshold is judged respectively, if yes, step 4 is executed; if no, the image is discarded, and step 1 is reexecuted. Wherein, the discarded image can be the first image, also can be the first image and the second image, determined by the sampling period / interval.

[0134] Through the pre-setting and judging of the number of static key points, the problem that the camera correction effect is not good due to the small number of corner points, the unstable key points of the collected image information and other factors in the process of processing image corner points is avoided.

[0135] Step 5, the correction parallax offset is introduced into the first coordinate transformation result and the second coordinate transformation result respectively, the correction function calculation formula is generated, and the correction parallax offset under the preset condition is calculated; wherein, the correction parallax offset is used for correcting the parallax value d of the binocular camera.

[0136] It should be noted that the correction result can be set to the binocular camera in real time to take effect, and the configuration file on the storage is updated, and after power off and starting again, it still takes effect.

[0137] For any pixel point in the image, based on the binocular ranging principle, through the form of three-dimensional coordinate transformation, it can be converted from two-dimensional coordinate to a point in three-dimensional coordinate system.

[0138] The two-dimensional coordinates of any pixel point in the image are set as (u, v), and the corresponding three-dimensional coordinates of the point are (px , p y , p z , the corresponding calculation formula is:

[0139]

[0140] In the formula, B represents the center distance of the left and right cameras, d is the parallax value; f represents the focal length of the camera; c x , c y represents the offset of the camera optical axis in the image pixel coordinate system.

[0141] Similarly, after obtaining the angle rotation matrix R and the coordinate translation matrix T, if the binocular camera does not need to be corrected, the three-dimensional coordinates of the same pixel point (static key point) in the next frame image at the t+1 time can be calculated based on the three-dimensional coordinates of the point in the last frame image at the t time through the following calculation formula:

[0142]

[0143] For an ideal binocular camera without correction, the relationship of the corresponding static key points in the adjacent two frames of images is equal, which can be brought into the above formula.

[0144] That is, in the ideal state, the relationship of the corresponding static key points in the adjacent two frames of images is equal, as shown in the following formula:

[0145]

[0146] However, due to environmental temperature, vibration and other factors, the physical structure of the camera deforms, so that the relationship of the corresponding static key points in the adjacent two frames no longer conforms to the above formula, combined with the attached Figure 2 , as shown in the following formula: Figure 2 is the coordinate system diagram of the binocular stereo camera external parameter calibration of the application.

[0147] Based on the above formula, for the adjacent two frames of images, the pixel coordinates of the i-th second static key point in the second image at the t+1 time are first coordinate transformed in the form of three-dimensional coordinate transformation, and the correction parallax offset is introduced to convert the two-dimensional pixel coordinates into the coordinate points in the three-dimensional space coordinate system. The first coordinate transformation can adopt the following formula:

[0148]

[0149] In the formula, B represents the center distance of the left and right cameras in the binocular camera; d i,t+1 is the parallax value corresponding to the i-th group of static key points at the t+1 time; c x , c yrepresents the offset of the camera optical axis in the image pixel coordinate system, in units of pixels; f represents the focal length of the camera; (u i,t+1 i,t+1 is the pixel coordinate corresponding to the i th group of static key points at the t+1 th moment;

[0150] For the i th first static key point in the first image at the t th moment, based on the inter-frame motion transformation matrix, a second coordinate transformation is performed, and a corrected parallax offset is introduced to calculate the three-dimensional coordinates, and the following formula is used:

[0151]

[0152] In the formula, B represents the center distance of the left and right two cameras in the binocular camera; d i,t is the parallax value corresponding to the i th group of static key points at the t th moment; c x , c y represents the offset of the camera optical axis in the image pixel coordinate system, in units of pixels; f represents the focal length of the camera; (u i,t , v i,t is the pixel coordinate corresponding to the i th group of static key points at the t th moment;

[0153] Therefore, the corrected parallax offset is calculated as follows:

[0154]

[0155] In the formula, B represents the center distance of the left and right two cameras; c x , c y represents the offset of the camera optical axis in the image pixel coordinate system, in units of pixels; f represents the focal length of the camera; B, c x and c y can obtain initial values through stereo calibration, and are known quantities.

[0156] Further, in step 5, the above-mentioned corrected parallax offset is calculated. The accumulated static key points and the corresponding inter-frame motion transformation matrix, i.e. the angle rotation matrix R and the coordinate translation matrix T, in the corresponding adjacent two frames of images, are respectively brought into the formula F(offset) to calculate the value result of the corrected function calculation formula. When the value result is 0, the corrected parallax calculation result corresponding to each pair of static key point groups is obtained, a plurality of corrected parallax intermediate values are obtained, and then the calculation of the average value is used, i.e. the corrected parallax intermediate values are accumulated to calculate the average value of the corrected parallax intermediate values. The average value calculation result is recorded as the corrected parallax offset, so as to ensure the accuracy of the data and reduce the correction error.

[0157] ​Further, a target function can be set by using the least square method, and the correction parallax offset corresponding to the correction parallax offset calculation formula is solved by optimizing the target function.

[0158] The target function corresponding to the preset condition is as follows, and the optimization target is set as the minimum overall deviation.

[0159]

[0160] The specific solving process is not described here.

[0161] Then, the first static key point and the second static key point are paired, and the paired first static key point and the second static key point are sequentially subjected to the first coordinate transformation and the second coordinate transformation, and the correction parallax offset is introduced, and the calculation formula corresponding to the correction function calculation formula is:

[0162]

[0163] The correction parallax offset satisfying the target function is calculated.

[0164] In addition, the convergence function calculation method can also be used to solve the correction parallax offset corresponding to the correction function F(offset) calculation formula in the embodiment, and the specific process is not described here.

[0165] It should be noted that:

[0166] The t+1 time and t time key point pixel coordinate matrix A in step 2 t+1 , A t If the read pixel is in the left image, c x , c y in formula (15) is also the offset of the left camera optical axis in the left image pixel coordinate system; the t+1 time and t time key point pixel coordinate matrix A in step 2 t+1 , A t If the read pixel is in the right image, c x , c y in formula (15) is also the offset of the right camera optical axis in the right image pixel coordinate system.

[0167] The above embodiments are preferred embodiments of the present application, which are only used to facilitate the description of the present application and do not limit the present application in any form. Any person skilled in the art can make partial changes or modifications to the equivalent embodiments within the scope of the technical features disclosed in the present application without departing from the technical features of the present application, and the changes or modifications still belong to the scope of the technical features of the present application.

Claims

1. A real-time binocular camera self-calibration method, the binocular camera is installed on a mobile device, characterized in that, The method comprises the following steps: Step 1: According to a sampling period, obtaining two adjacent images acquired by a binocular camera during movement of a mobile device, and denoting the two adjacent images as a first image and a second image respectively; Step 2: Extracting a first static key point in the first image and a second static key point in the second image respectively; Step 3: Calculating an inter-frame motion transformation matrix according to IMU information during movement of the mobile device, wherein the inter-frame motion transformation matrix comprises an angle rotation matrix R and a coordinate translation matrix T; Step 4: Performing first coordinate transformation on the second static key point by using a three-dimensional coordinate transformation matrix, and performing second coordinate transformation on the first static key point by using the inter-frame motion transformation matrix; specifically, after obtaining the angle rotation matrix R and the coordinate translation matrix T, if the binocular camera does not need to be corrected, then a three-dimensional coordinate of a same pixel point in a next frame image at a t+1 moment can be calculated based on a three-dimensional coordinate of the point in a previous frame image at a t moment by using the following calculation formula: ， Step 5: Introducing a correction parallax offset into the first coordinate transformation result and the second coordinate transformation result respectively to generate a correction function F calculation formula, and calculating the correction parallax offset that satisfies a preset condition; The correction parallax offset is used for correcting a parallax value d of the binocular camera.

2. The real-time binocular camera self-calibration method of claim 1, wherein, Before the step 4, the method further comprises the following steps: A threshold value for extracting a static key point is set as a preset threshold value, the number of the first static key points and the second static key points selected in the step 1 is accumulated, and it is determined whether the number of the first static key points and the second static key points is greater than or equal to the preset threshold value; if yes, the step 4 is executed; if no, the image is discarded, and the step 1 is re-executed.

3. The real-time binocular camera self-calibration method of claim 1, wherein, The preset condition in the step 5 is that a value result of the correction function F calculation formula is 0; The step 5 specifically comprises the following steps: Pairing the first static key points and the second static key points; Performing first coordinate transformation and second coordinate transformation on the paired first static key points and second static key points respectively, and introducing the correction parallax offset, wherein a corresponding calculation formula of the correction function F calculation formula is: ， In the formula, B represents the center distance of the left and right cameras in the binocular camera; d i,t , i,t+1 is the disparity value corresponding to the i th group of static key points at the t+1, t moment; c x , y represents the offset of the camera optical axis in the image pixel coordinate system, in units of pixels; f represents the focal length of the camera; (u i,t , v i,t ), (u i,t+1 , v i,t+1 ) are the pixel coordinates corresponding to the i th group of static key points at the t+1, t moment; When the value result of the correction function F calculation formula is 0, a correction parallax calculation result corresponding to each pair of static key point groups is denoted as a correction parallax intermediate value; The correction parallax intermediate values are accumulated, an average value of the correction parallax intermediate values is calculated, and a calculation result of the average value is denoted as the correction parallax offset.

4. The real-time binocular camera self-calibration method of claim 3, wherein, The target function corresponding to the preset condition is: ， The step 5 specifically comprises the following steps: Pairing the first static key points and the second static key points; Performing first coordinate transformation and second coordinate transformation on the paired first static key points and second static key points respectively, and introducing the correction parallax offset, wherein a corresponding calculation formula of the correction function F calculation formula is: ， In the formula, B represents the center distance of the left and right cameras in the binocular camera; d i,t , d i,t+1 is the disparity value corresponding to the ith group of static key points at the t+1, t moment; c x , c y represents the offset of the camera optical axis in the image pixel coordinate system, in units of pixels; f represents the focal length of the camera; (u i,t , v i,t ), (u i,t+1 , v i,t+1 ) are the pixel coordinates corresponding to the ith group of static key points at the t+1, t moment; Calculating the correction parallax offset that satisfies the target function.

5. The real-time binocular camera self-calibration method according to any one of claims 1 to 4, characterized in that, The step 2 further filters the adjacent two images in the process of selecting the static key points, and the filtering method is: Based on a deep learning method, image information contained in the adjacent two images is recognized, and a region of interest in the adjacent two images is further obtained, when the image information in the region of interest is determined as a first target, the corresponding image in the region of interest is cut out, and then the corresponding image corner points in the adjacent two images after cutting out are recorded as static key points; wherein the first target is a moving object.

6. The real-time binocular camera self-calibration method according to any one of claims 1 to 4, characterized in that, The step 2 filters the adjacent two images in the process of selecting the static key points, and the filtering method is: Based on a deep learning method, image information contained in the adjacent two images is recognized, and a region of interest in the adjacent two images is further obtained, when the image information in the region of interest is determined as a second target, image corner points of the corresponding image in the region of interest are obtained and recorded as static key points; wherein the second target is a static object.

7. The real-time binocular camera self-calibration method of claim 1, wherein, The moving device is a vehicle, and the calculation method of the angle rotation matrix R in the step 3 is as follows: ， Wherein, p, r, q are respectively pitch angle, roll angle and yaw angle in the attitude angle.

8. The real-time binocular camera self-calibration method of claim 7, wherein, The solution of the attitude angle r, p, q includes the following steps: Step 3-1, by installing an accelerometer on the vehicle, reading the acceleration values of the accelerometer around the X-axis, Y-axis, Z-axis at t+1 time as: , , and solving by the following formula to obtain the t+1 time and : ， Wherein, g represents the value of gravity acceleration; Step 3-2: Read the acceleration values of the accelerometer around the X, Y, Z axes at any time t through the accelerometer and gyroscope installed on the vehicle , , and the time t is solved by the following formula , : ， reading the angular velocity of rotation of the gyroscope around the X, Y, Z axes at any instant t , , , r at instant t+1 is obtained by solving the following equations gyvo , p gyvo , q gyvo : ， Step 3-3, fusing the attitude based on the results of step 3-1 and step 3-2: ， Wherein, K represents the proportionality coefficient.

9. The real-time binocular camera self-calibration method of claim 1, wherein, The moving device is a vehicle, and the coordinate translation matrix T in the step 3 is calculated by a vehicle speed signal to obtain a relative displacement matrix T within t to t+1 time; ， wherein, and Vx and Vy are respectively the lateral and longitudinal components of the vehicle speed, is the displacement component in the vertical direction; The vehicle is provided with a gyroscope, which reads the angular velocity of rotation around the X, Y, Z axes at time t+1 , , ; and the matrix T is solved by the following equation: ， wherein V represents a vehicle speed; is a time interval from time t to time t+1.

10. The real-time binocular camera self-calibration method according to any one of claims 1 or 2 or 3, characterized in that, The first static key point and the second static key point are image corner points in the first image and the second image respectively.

Images