Key point-based automatic calibration system and method for extrinsic parameters of vehicle-mounted monocular cameras
Through the key point-based automatic calibration system of the vehicle-mounted single-eye camera external parameters, the automatic calibration of the camera external parameters is achieved using feature point extraction and relative motion solution, and the simplicity and applicability of the vehicle-mounted camera calibration is solved, and it is suitable for all Install the angled camera and correct the deflection caused by vehicle vibration.
Patent Information
- Application Number
- CN202210550682.8
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2022-05-18
- Publication Date
- 2024-11-12
- Estimated Expiration
- 2042-05-18
AI Technical Summary
The prior art cannot simply and efficiently calibrate the vehicle-mounted cameras, especially when the camera is deflected slightly due to vehicle vibration or installation deviation, and cannot be suitable for calibration of side view cameras.
Using a key point-based automatic calibration system for external parameters of the vehicle monocular camera, the automatic calibration of external parameters of the camera is realized through feature point extraction matching, relative motion solution, homography solution and external parameters solution modules, without relying on calibration objects or lanes Line information, suitable for cameras of all mounting angles.
It realizes online dynamic calibration of camera external parameters, simplifies the operation process, reduces hardware requirements, is suitable for a variety of vehicle models, can correct camera deflection caused by vehicle vibration or installation deviation, and is suitable for front and rear view and side view cameras.
Smart Images

Figure CN114842093B_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the technical field of intelligent automobiles, and in particular to a key point-based automatic calibration system and method for extrinsic parameters of a vehicle-mounted monocular camera. Background Art
[0002] The vehicle camera is an important sensor of the vehicle vision system, which can provide the vehicle with driving environment perception capabilities and provide the basic data required for decision-making for the assisted driving system. The effective functioning of the vehicle vision system depends on the accurate calibration of the vehicle camera. However, the camera will deflect during the working cycle due to vehicle body vibration or vehicle maintenance, which will affect the accuracy of the vision system. Therefore, a simple and efficient camera extrinsic calibration method is needed.
[0003] The current camera calibration methods based on pure vision algorithms mainly include traditional calibration methods and automatic calibration methods.
[0004] Traditional calibration methods, such as the invention patent application for a vehicle-mounted surround view camera calibration method in a simple calibration environment with application number CN202110511210.7, use a calibration object of known size, such as a calibration plate, etc., to establish a correspondence between points on the calibration object with known coordinates and its image pixel points in a fixed location, and use a certain method to obtain the camera extrinsic parameters. Although this method has high calibration accuracy, its operation is relatively complicated, and it requires manual participation of high-precision calibration objects, and it is impossible to calibrate and correct the parameters of a moving vehicle-mounted camera.
[0005] Most of the currently commonly used automatic calibration methods are based on some parallel or orthogonal information in the camera image. For example, in the invention patent application of CN202011551440.8, a method for online calibration of a vehicle-mounted camera and a vehicle-mounted infotainment system, regular straight lane lines are used to calibrate the external parameters of the vehicle-mounted camera. The vehicle-mounted camera collects lane line images and detects lane lines based on the collected images to obtain the world coordinate linear equation of the lane lines. The current value of the external parameter of the vehicle-mounted camera is used as the initial value and the straightness of the world coordinate linear equation of the lane lines is used as the optimization target to optimize the current value of the external parameter of the vehicle-mounted camera, thereby achieving the external parameter calibration of the vehicle-mounted camera. Although this type of calibration method has a fast calculation speed, it is only applicable to front-view and rear-view cameras, and cannot be calibrated for sections without lane line information or side-view cameras. Summary of the invention
[0006] In view of the above-mentioned deficiencies in the prior art, the technical problem to be solved by the present invention is: how to provide an automatic calibration system and method for the extrinsic parameters of a vehicle-mounted monocular camera based on key points, which can correct small camera deflections caused by vehicle vibration or installation deviation without relying on calibration objects, lane lines or other information and human participation, and is also suitable for side-view camera calibration.
[0007] In order to solve the above technical problems, the present invention adopts the following technical solutions:
[0008] The key point-based vehicle-mounted monocular camera extrinsic automatic calibration system includes:
[0009] A feature point extraction and matching module is used to extract feature points in continuous frame images and match them;
[0010] The relative motion solving module is used to solve the relative motion between consecutive frames based on feature point pairs and epipolar geometry method;
[0011] The homography solving module is used to extract the feature points on the road surface and select the road surface feature points with high matching according to the homography relationship;
[0012] The external parameter solving module is used to solve the camera external parameters based on the relative motion of continuous frames and the road feature point pairs;
[0013] The result output module is used to accumulate the results obtained by the external parameter solution module, and after reaching a fixed amount, calculate the average and finally output it.
[0014] A method for automatic calibration of extrinsic parameters of a vehicle-mounted monocular camera based on key points, using the above-mentioned automatic calibration system for extrinsic parameters of a vehicle-mounted monocular camera based on key points, comprises the following steps:
[0015] Step 1) Initial value setting;
[0016] Step 2) feature point extraction and matching, wherein the feature point extraction and matching module extracts feature points in continuous frame images and matches them;
[0017] Step 3) solving the relative motion, the relative motion solving module uses the epipolar geometry method to solve the relative motion between consecutive frames based on the feature point pairs;
[0018] Step 4) extracting and matching road surface feature points, the homography solving module extracts and matches road surface feature points;
[0019] Step 5) homography solving, the homography solving module performs homography solving;
[0020] Step 6) extrinsic parameter calculation, wherein the extrinsic parameter calculation module calculates the camera extrinsic parameters based on the continuous frame relative motion and the road feature point pairs;
[0021] Step 7) cumulative output: the result output module accumulates the results obtained by the external parameter solving module, and after reaching a fixed amount, calculates the average and finally outputs it.
[0022] Preferably, the setting of the initial value in step 1) includes setting the world coordinates of the camera installation position and setting the initial value of the camera Euler angle;
[0023] The method for setting the camera installation position is as follows: establish a world coordinate system with the ground below the center of the rear axle of the vehicle as the origin of the world coordinate system, the forward direction of the vehicle as the positive direction of the X-axis of the world coordinate system, and the direction perpendicular to the ground and upward as the positive direction of the Z-axis of the world coordinate system, and determine the world coordinates of the camera installation position according to the established world coordinate system;
[0024] The method for setting the initial value of the camera Euler angle is: using the known state value of the camera Euler angle last time as the initial value of the camera Euler angle.
[0025] Preferably, the initial value setting in step 1) also includes the camera's continuous frame images Im1 and Im2, the vehicle body speed and the vehicle body angular velocity, the continuous frame images are all time stamped, the time difference between the two camera frames is calculated by the time stamps of the two continuous frame images, and the relative motion between the two camera frames is determined by solving the epipolar constraint between the two camera frames;
[0026] When the angular velocity of the vehicle body is greater than 1 degree / second, it is considered that the vehicle is not moving in a straight line and the calculation is exited.
[0027] Preferably, in step 2), when the feature point extraction and matching module extracts and matches feature points in consecutive frame images, it first performs corner point detection on the two input frames, then extracts the corner points in the images and their feature vectors, and then matches the feature points based on the Hamming distance between the feature vectors, and finally retains all successfully matched feature point pairs.
[0028] Preferably, in step 2), feature points in continuous frame images are extracted using an ORB corner point detection method, and the number of feature points extracted from each frame image is less than or equal to 3000;
[0029] Feature points are matched based on the Hamming distance between feature vectors, and the top 30% with the highest matching degree are selected as the final matching results.
[0030] Preferably, when solving the relative motion in step 3), first screen out the feature point pairs with high matching degree in step 2), then solve the essential matrix between the two frames of images based on the feature point pairs, and finally solve the relative rotation matrix and relative displacement vector between the two frames of images based on the essential matrix.
[0031] Preferably, in step 3), if the extracted feature point pairs are less than 8 pairs, the calculation is exited; if the extracted feature point pairs are greater than or equal to 8 pairs, the five-point method is used to solve the essential matrix between the two frames of images. After completing the essential matrix calculation, the relative rotation matrix and relative translation vector between the two frames of images are calculated based on the essential matrix.
[0032] Preferably, in step 3), it is also necessary to determine whether the relative motion state estimation of the camera is correct. If it is correct, step 4) is executed; if it is not correct, the calculation is exited;
[0033] The method to determine whether the relative motion state estimation of the camera is correct is:
[0034] 1) Calculate the initial rotation matrix R based on the initial Euler angles [ω0,θ0,τ0] e0 :
[0035]
[0036] 2) Calculate the relative translation vector tc0 based on the initial value; for a vehicle in a translational state, the normalized translation vector between two consecutive frames of images is t car =[-1,0,0], then according to the coordinate system rotation and translation transformation relationship, the relative
[0037] The translation vector tc0 is:
[0038] t c0 =R e0 *t car ;
[0039] 3) Compare the distance between the relative translation vector tc of the estimated value and the relative translation vector tc0 of the initial value, set a threshold, if the distance between the relative translation vector tc of the estimated value and the relative translation vector tc0 of the initial value is greater than the threshold, exit the calculation; if the distance between the relative translation vector tc of the estimated value and the relative translation vector tc0 of the initial value is less than or equal to the threshold, execute step 4).
[0040] Preferably, in step 4), when extracting and matching road feature points, a method of limiting the feature point extraction area is used to reduce the number of non-road feature points, and the feature points are matched based on vehicle body motion information and local image information.
[0041] Preferably, the width of the camera screen is set to W and the height is set to H, then the feature point extraction area of the front view camera and the rear view camera is The feature point extraction area of the side view camera is
[0042] Preferably, in step 4), the method for matching feature points based on vehicle body motion information and local image information is as follows: a camera model is established based on an initial input value, pixel coordinates of road feature points in the first frame image are subsequently converted into world coordinates, and the position of the same feature point in the second frame image is predicted based on the vehicle body motion information, and a square local image of the first frame image is captured with the feature point in the first frame image as the center and the first set length as the side length as the image to be matched. After the position prediction is completed, the world coordinates of the predicted position are converted into pixel coordinates again, and a search area is constructed with the feature point as the center and the second set length as the radius. Feature points in the second frame image within the search area are searched. If there are corresponding feature points in the area, a square local image of the second frame image is captured with the feature point as the center and the first set length as the side length to match the local image of the feature point of the first frame image. The feature point with the highest matching degree is selected as the final matching point. If there are no corresponding feature points in the area, the current feature point matching fails and the next matching is started.
[0043] Preferably, when solving the homography in step 5), the homography relationship in the two frames of images is solved based on the feature point pairs successfully matched in step 4); for points located on the same plane in the two frames of images, their pixel coordinates satisfy the homography relationship, and the feature point pairs satisfying the homography are screened out by solving the homography;
[0044] If there are less than 4 pairs of successfully matched feature points in step 4), the homography solution fails and the calculation ends.
[0045] If there are at least 4 pairs of feature point pairs that are successfully matched in step 4), the least squares method is used to estimate the homography matrix, and all feature point pairs that satisfy the homography relationship are retained.
[0046] Preferably, when solving the camera extrinsic parameters in step 6), three orthogonal unit vectors are constructed based on the relative displacement vector in step 3) as three components of the camera extrinsic rotation matrix, and one undetermined parameter is retained; then, any two pairs of feature point pairs that are successfully matched in step 5) are selected to obtain all possible selection methods, and each group of selections is solved to obtain an undetermined parameter and its corresponding extrinsic rotation matrix, and the corresponding Euler angle can be obtained through the conversion relationship between the extrinsic rotation matrix and the Euler angle; if the difference between the obtained Euler angle and the initial value is greater than 5 degrees, it is considered to be an outlier and discarded; if the difference between the obtained Euler angle and the initial value is less than or equal to 5 degrees, all Euler angles that meet the requirements are retained and their average is calculated, and the corresponding extrinsic rotation matrix is calculated based on the average of the Euler angles as the camera Euler angles and extrinsic rotation matrix obtained by this calculation and calibration.
[0047] Preferably, the calculation method of the camera's extrinsic rotation matrix in step 6) is:
[0048] 1) Based on the relative translation vector tc obtained in step 3), construct a set of standard orthogonal vectors:
[0049]
[0050]
[0051] n2=r1×n1
[0052] Where: r 11 ,r 21 ,r 31 are the three components of r1;
[0053] Based on the above standard orthogonal vectors, a single-parameterized camera rotation matrix can be constructed:
[0054] R e (α)=[r1 -r1×r3(α) r3(α)]
[0055] r3(α)=cos(α)n1+sin(α)n2
[0056] Where: α is the unknown parameter, r1 is the first column of the rotation matrix, r3 is the third column of the rotation matrix, R e is the rotation matrix;
[0057] 2) Arrange and combine all the feature point pairs extracted in step 5) to obtain all possible extraction methods. If there are N pairs of feature point pairs in step 5), there are N extraction methods. kind;
[0058] 3) Taking the i-th combination as an example, solve the coordinates of its feature point in the camera coordinate system. If the pixel homogeneous coordinates of its feature point are m p11 ,m p12 ,m p21 ,m p22 , then its camera coordinate system coordinates are:
[0059]
[0060]
[0061]
[0062]
[0063] Where: K is the intrinsic parameter matrix of the camera, m c11 ,m c12 ,m c2 ,m c2 is the camera coordinate system coordinate of the feature point, [u c11 ,uc12 ,u c21 ,u c22 ],[v c1 ,v c12 ,v c2 v c22 ] are the x-axis component and y-axis component of each coordinate of the feature point camera coordinate system;
[0064] 4) Based on the coordinates of the feature point camera coordinate system, solve the rotation matrix parameters. The calculation method is:
[0065]
[0066]
[0067]
[0068] Preferably, in step 6), the calculation method of the dynamic calibration value of the camera Euler angle is:
[0069] If the camera extrinsic rotation matrix is obtained as:
[0070]
[0071] The obtained Euler angle is:
[0072]
[0073]
[0074]
[0075] Where: R e11 To R e3 is the rotation matrix R e The nine components of e ,θ e ,τ e They are the three components of the Euler angle;
[0076] If the difference between the calculated Euler angle and the initial value is greater than 5 degrees, it will be discarded. If it is less than or equal to 5 degrees, it will be retained. All Euler angles that meet the requirements are saved and the average is calculated, which is the dynamic calibration value of the camera Euler angle calculated this time.
[0077] Preferably, in step 7), the result output module accumulates the single Euler angles obtained in step 6), and when the accumulated data of the Euler angles reaches a set amount, the mean of all Euler angles is obtained, and the corresponding extrinsic parameter rotation matrix is calculated according to the mean of the Euler angles and output as the final calibration result.
[0078] Compared with the prior art, the present invention has the following advantages:
[0079] 1. The monocular camera automatic calibration method proposed in the present invention can realize the online dynamic calibration of the camera's extrinsic rotation matrix and Euler angles based on the camera's own image and vehicle body motion information. Compared with traditional calibration methods, this method does not require high-precision calibration objects or human participation, and can realize the calibration and compensation of the camera rotation matrix under the vehicle's motion state; compared with conventional automatic calibration methods, this method does not require the assistance of road lane lines, and can realize the calibration of cameras at all installation angles, not just front and rear view cameras. At the same time, this method has low hardware requirements and does not require strict implementation conditions. It has the advantages and characteristics of simplicity, efficiency, and accuracy, and is suitable for a variety of vehicle models.
[0080] 2. The present invention can realize automatic calibration of the camera rotation matrix or Euler angle under the condition of vehicle driving in a straight line, without the need for road lane line information, and can be used for calibration of front, rear and side view cameras.
[0081] 3. The present invention only relies on the camera to input continuous frame data and the vehicle's own motion information, and can realize the online calibration of the three rotation Euler angles of the camera. It does not need to rely on calibration plates, lane lines or other information and manual participation. It can correct small deflections of the camera caused by vehicle vibration or installation deviation. The method is highly flexible and simple and convenient to implement. BRIEF DESCRIPTION OF THE DRAWINGS
[0082] Figure 1 It is a flow chart of the method for automatic calibration of extrinsic parameters of a vehicle-mounted monocular camera based on key points of the present invention;
[0083] Figure 2 This is a typical feature point matching result diagram in the key point-based automatic calibration method for vehicle-mounted monocular camera extrinsic parameters of the present invention;
[0084] Figure 3 It is a flow chart of the calculation of relative motion in the method for automatic calibration of extrinsic parameters of a vehicle-mounted monocular camera based on key points of the present invention;
[0085] Figure 4 It is a flow chart of solving homography in the method for automatic calibration of extrinsic parameters of a vehicle-mounted monocular camera based on key points of the present invention;
[0086] Figure 5 The present invention is a flow chart of the calculation of external parameters in the method for automatic calibration of external parameters of a vehicle-mounted monocular camera based on key points. DETAILED DESCRIPTION
[0087] The present invention will be further described below in conjunction with the accompanying drawings and embodiments.
[0088] First of all, it should be noted that the algorithm involved in the present invention is based on the following four assumptions:
[0089] 1) The vehicle moves in a straight line;
[0090] 2) The camera only changes in installation angle relative to the vehicle body, but not in position;
[0091] 3) The difference between the initial value of the Euler angle and the true value does not exceed 5 degrees;
[0092] 4) The camera intrinsic parameters are known and do not change during the whole process; the camera extrinsic parameters refer to the parameters of the camera in the world coordinate system, such as the camera's position, rotation direction, etc.; the camera intrinsic parameters refer to the parameters related to the camera's own characteristics, such as the camera's focal length, pixel size, etc.
[0093] Specifically, the present invention provides a key point-based automatic calibration system for vehicle-mounted monocular camera extrinsic parameters, comprising:
[0094] A feature point extraction and matching module is used to extract feature points in continuous frame images and match them;
[0095] The relative motion solving module is used to solve the relative motion between consecutive frames based on feature point pairs and epipolar geometry method;
[0096] The homography solving module is used to extract the feature points on the road surface and select the road surface feature points with high matching according to the homography relationship;
[0097] The external parameter solving module is used to solve the camera external parameters based on the relative motion of continuous frames and the road feature point pairs;
[0098] The result output module is used to accumulate the results obtained by the external parameter solution module, and after reaching a fixed amount, calculate the average and finally output it.
[0099] In addition, the present invention also provides a method for automatic calibration of extrinsic parameters of a vehicle-mounted monocular camera based on key points, using the above-mentioned automatic calibration system for extrinsic parameters of a vehicle-mounted monocular camera based on key points, comprising the following steps, as shown in the attached figure: Figure 1 As shown:
[0100] Step 1) Initial value setting;
[0101] Step 2) Feature point extraction and matching: the feature point extraction and matching module extracts feature points in continuous frame images and matches them;
[0102] Step 3) solving the relative motion. The relative motion solving module uses the epipolar geometry method to solve the relative motion between consecutive frames based on the feature point pairs.
[0103] Step 4) Extracting and matching road feature points: the homography solving module extracts and matches road feature points;
[0104] Step 5) homography solving: the homography solving module performs homography solving;
[0105] Step 6) External parameter calculation: the external parameter calculation module calculates the camera external parameters based on the relative motion of continuous frames and the road feature point pairs;
[0106] Step 7) Cumulative output: The result output module accumulates the results obtained by the external parameter solution module, and after reaching a fixed amount, it calculates the average and finally outputs it.
[0107] In this embodiment, the initial value setting in step 1) includes the world coordinates of the camera installation position [x w0 ,y w0 ,z w0 ] and the initial value of the camera's Euler angle [ω0,θ0,τ0]; the Euler angle is a group of three independent angular parameters used to determine the position of a fixed-point rotating rigid body, consisting of the nutation angle ω, the precession angle (i.e., the precession angle) θ and the rotation angle τ, and is named after Euler who first proposed it.
[0108] The method for setting the camera installation position is as follows: the world coordinate system is established with the ground below the center of the rear axle of the vehicle as the origin of the world coordinate system, the vehicle forward direction as the positive direction of the world coordinate system X axis, and the direction perpendicular to the ground and upward as the positive direction of the world coordinate system Z axis, and the world coordinates of the camera installation position are determined according to the established world coordinate system [x w0 ,y w0 ,z w0 ], in specific implementation, the world coordinates of the camera installation position can be determined by manual measurement;
[0109] The method for setting the initial value of the camera Euler angle is: use the known state value of the camera Euler angle of the last time as the initial value of the camera Euler angle. If there is no known state, use the manual visual measurement method to determine its initial value.
[0110] In this embodiment, the data input required by the algorithm of the present invention is the camera continuous frame images Im1 and Im2, the vehicle body speed V and the angular velocity YawRate. The continuous frame images are time stamped, and the time difference dt between the two frames can be calculated. The algorithm determines the relative motion of the two frames by solving the epipolar constraint between the two frames. If the YawRate of the vehicle body is greater than 1 degree / second, it is considered that the vehicle is not moving in a straight line and the calculation is abandoned.
[0111] In this embodiment, in step 2), when the feature point extraction and matching module extracts and matches feature points in consecutive frame images, it first performs corner point detection on the two input frames, then extracts the corner points in the images and their feature vectors, and then matches the feature points based on the Hamming distance between the feature vectors, and finally retains all successfully matched feature point pairs.
[0112] Specifically, this algorithm first extracts feature points from the camera's continuous frame images Im1 and Im2, and the extraction algorithm uses ORB corner detection. ORB is the abbreviation of Oriented FAST and Rotated BRIEF. This feature detection operator is proposed based on the famous FAST feature detection and BRIEF feature descriptor. Its running time is much better than SIFT and SURF, and it can be applied to real-time feature detection. ORB feature detection is scale and rotation invariant, and is also invariant to noise and its perspective transformation. It first detects feature points in the image through the FAST operator, and then extracts the Steered BRIEF features of the feature points to describe the feature points. The feature is a one-dimensional vector. Finally, matching is performed based on the Hamming distance between feature vectors. The top 30% with the highest matching degree are selected as the final matching results. As shown in the attached figure Figure 2 The figure shows a typical feature point matching result. This algorithm limits the number of feature points extracted per frame to no more than 3000.
[0113] In this embodiment, when solving the relative motion in step 3), firstly, the feature point pairs with high matching degree in step 2) are screened out, and then the essential matrix E between the two frames of images is solved based on the feature point pairs, and finally, the relative rotation matrix Rc and the relative displacement vector tc between the two frames of images are solved based on the essential matrix E.
[0114] Specifically, as attached Figure 3 The figure shows the calculation flow chart of relative motion. After completing the feature point matching in step 2), the five-point method is used to solve the essential matrix E between the two frames. The solution of the essential matrix E requires at least 5 pairs of feature points. To ensure the accuracy of the solution, when the extracted feature point pairs are less than 8 pairs, the calculation is abandoned. If there are more than 8 pairs, the essential matrix E is estimated. After the calculation of the essential matrix E is completed, the relative rotation matrix Rc and the relative translation vector tc between the two frames can be calculated based on E.
[0115] At this time, it is necessary to determine whether the relative motion state of the camera is estimated correctly. If the vehicle is in a translational state, its relative rotation matrix should be a unit matrix. The present invention uses the following method to determine whether the relative motion estimation is correct:
[0116] 1) Calculate the relative translation vector tc0 based on the initial value. First, calculate its initial rotation matrix R based on the initial Euler angle [ω0,θ0,τ0] e0 :
[0117]
[0118] Then the relative translation vector tc0 based on the initial value is calculated. For a vehicle in a translation state, the normalized translation vector between two consecutive frames is t car=[-1,0,0], then according to the coordinate system rotation and translation transformation relationship, the relative translation vector tc0 of the camera is:
[0119] t c0 =R e0 *t car
[0120] 2) Compare the distance between the estimated value tc and the initial value tc0, and set the threshold to 0.15. c -t c0 If ‖>0.15, the estimation error is too large and the calculation is abandoned. If the threshold condition is met, the next step of calculation is performed.
[0121] In this embodiment, when extracting and matching road feature points in step 4), a method of limiting the feature point extraction area is used to reduce the number of non-road feature points, and the feature points are matched based on the vehicle body motion information and local image information.
[0122] Specifically, the calculation of camera extrinsic parameters depends on feature points located on the road plane, so feature points in the image need to be extracted again. The extraction algorithm also uses ORB corner detection, but the difference is that this detection needs to limit the detection area. If the camera screen width is W and the height is H, for the front and rear cameras, the feature point extraction area is For the side-view camera, the feature point extraction area is This can ensure that the extracted feature points are located on the road surface as much as possible.
[0123] In this embodiment, the method for matching feature points based on vehicle body motion information and local image information in step 4) is as follows: a camera model is established based on an initial input value, and then the pixel coordinates of the road feature point in the first frame image are converted into world coordinates, and the position of the same feature point in the second frame image is predicted based on the vehicle body motion information, and at the same time, a square local image of the first frame image is captured with the feature point in the first frame image as the center and the first set length as the side length as the image to be matched. After the position prediction is completed, the world coordinates of the predicted position are converted into pixel coordinates again, and a search area is constructed with the feature point as the center and the second set length as the radius, and the feature points in the second frame image within the search area are searched. If there are corresponding feature points in the area, a square local image of the second frame image is captured with the feature point as the center and the first set length as the side length to match the local image of the feature point of the first frame image, and the feature point with the highest matching degree is selected as the final matching point. If there are no corresponding feature points in the area, the current feature point matching fails and the next matching is started.
[0124] Specifically, after the feature point extraction is completed, the feature points in the two frames need to be matched. Since there are too many similar points on the road surface, if the Hamming distance of the BRIEF feature is directly used for matching, a large number of wrong matches will be caused, affecting subsequent calculations. Therefore, the present invention adopts a feature point matching algorithm based on vehicle body motion and image matching, which is as follows. Taking the i-th feature point in the first frame image as an example, the matching of other feature points is similar:
[0125] 1) Establish a pinhole camera model based on the initial values.
[0126] P i =K[R e0 t w ]P wi
[0127] Among them, K is the camera internal parameter matrix, which is a known parameter; R e0 is the initial rotation matrix; t w is the transfer vector, determined by the camera installation position:
[0128]
[0129] P i =[x i ,y i ,1] is the pixel homogeneous coordinate of the i-th feature point, P wi =[x wi ,y wi ,z wi ,1] is the corresponding world homogeneous coordinate.
[0130] 2) With [x i ,y i ] is the center, d is the side length, and the local area Rect1 in the first frame image is intercepted as the image to be matched.
[0131] 3) Calculate the world coordinates P of feature point i based on the camera model wi =[x wi ,y wi ,z wi ,1], since the feature point is located on the road surface, its Z coordinate z wi = 0. Then predict the position of the feature point in the next frame based on the vehicle body speed. The vehicle body speed is V, and the time interval between two frames is dt. Then the world coordinates of the predicted point are P wi_pre =[x wi -Vdt,y wi ,z wi ,1]. Then, according to the camera model, the pixel coordinates of the predicted point in the second frame image are deduced as P i_pre =[x i_pre ,y i_pre ,1].
[0132] 4) P i_pre As the center, r is the radius to establish a search area, and search for feature points of the second frame image in the area. If there are no feature points in the area, the current match fails and the next match begins. If there are n feature points in the area, these feature points are taken as the center, d is the side length, and the local area Rect21~Rect2 of the second frame image is intercepted. n as the image to be matched.
[0133] 5) Calculate Rect21~Rect2 n The absolute error and SAD between Rect1 and Rect1 are calculated as follows for the jth feature point in the region, 1≤j≤n:
[0134]
[0135] Calculate the SAD of all feature points to be matched, and select the point with the smallest SAD as the final matching result.
[0136] In this embodiment, when the homography is solved in step 5), as shown in the attached Figure 4 As shown, based on the successfully matched feature point pairs in step 4), the homography relationship in the two frames of images is solved; for points located on the same plane in the two frames of images, their pixel coordinates satisfy the homography relationship, and the feature point pairs satisfying the homography are screened out through homography solution, providing a basis for subsequent calculations;
[0137] If there are less than 4 pairs of successfully matched feature points in step 4), the homography solution fails and the calculation ends.
[0138] If there are at least 4 pairs of feature point pairs that are successfully matched in step 4), the least squares method is used to estimate the homography matrix, and all feature point pairs that satisfy the homography relationship are retained.
[0139] In this embodiment, when solving the camera extrinsic parameters in step 6), three orthogonal unit vectors are constructed based on the relative displacement vector in step 3) as three components of the camera extrinsic parameter rotation matrix, and one undetermined parameter is retained; then, any two pairs of feature point pairs that are successfully matched in step 5) are selected to obtain all possible selection methods, and each group of selections is solved to obtain an undetermined parameter and its corresponding extrinsic parameter rotation matrix, and the corresponding Euler angle can be obtained through the conversion relationship between the extrinsic parameter rotation matrix and the Euler angle; if the difference between the obtained Euler angle and the initial value is greater than 5 degrees, it is considered to be an abnormal value and discarded; if the difference between the obtained Euler angle and the initial value is less than or equal to 5 degrees, all Euler angles that meet the requirements are retained and their average is calculated, and the corresponding extrinsic parameter rotation matrix is calculated based on the average of the Euler angles as the camera Euler angles and extrinsic parameter rotation matrix obtained by this calculation and calibration.
[0140] Specific as attached Figure 5 As shown, the external parameter settlement in step 6) uses a parameterized method to solve the camera rotation matrix, which is achieved by single parameterizing the camera rotation matrix and solving a linear equation. The specific steps are as follows:
[0141] 1) Based on the relative translation vector tc obtained in step 3), construct a set of standard orthogonal vectors:
[0142]
[0143]
[0144] n2=r1×n1
[0145] Based on the above standard orthogonal vectors, a single-parameterized camera rotation matrix can be constructed:
[0146] R e (α)=[r1 -r1×r3(α) r3(α)]
[0147] r3(α)=cos(α)n1+sin(α)n2
[0148] Where α is the unknown parameter, r1 is the first column of the rotation matrix, r3 is the third column of the rotation matrix, R e is the rotation matrix.
[0149] 2) Solving the external parameter matrix requires extracting two pairs of feature points. All the feature point pairs extracted in step 5) are arranged and combined to obtain all possible extraction methods. If there are N pairs of feature point pairs in step 5), then there are N extraction methods. kind.
[0150] 3) Take the i-th combination as an example, and solve the coordinates of its feature point in the camera coordinate system. If the pixel homogeneous coordinates of its feature point are m p11 ,m p12 ,m p21 ,m p2 , then its camera coordinate system coordinates are:
[0151]
[0152]
[0153]
[0154]
[0155] Where: K is the intrinsic parameter matrix of the camera, m c11 ,m c12 ,m c21 ,m c2is the camera coordinate system coordinate of the feature point, [u c11 ,u c12 ,u c21 ,u c22 ],[v c11 ,v c12 ,v c21 v c22 ] are the x-axis component and y-axis component of each coordinate of the feature point camera coordinate system.
[0156] 4) Based on the coordinates of the feature point camera coordinate system, solve the rotation matrix parameters. The calculation method is:
[0157]
[0158]
[0159]
[0160] 5) Perform the above operation on all possible combinations to obtain all R e (α) Rotation matrix. Based on the relationship between Euler angle and rotation matrix, the camera's Euler angle is inferred. The specific method is as follows:
[0161] If the obtained external parameter rotation matrix is:
[0162]
[0163] The obtained Euler angle is:
[0164]
[0165]
[0166]
[0167] Where: R e11 To R e3 is the rotation matrix R e The nine components of e ,θ e ,τ e They are the three components of the Euler angle;
[0168] If the calculated Euler angle differs from the initial value by more than 5 degrees, it is discarded; if it differs from the initial value by less than 5 degrees, it is retained. All Euler angles that meet the requirements are stored and the average is calculated, which is the dynamic calibration value of the camera Euler angle calculated this time.
[0169] In this embodiment, in step 7), the camera extrinsic calibration is a continuous process, and its steady-state mean is obtained as the final result through repeated calculations within a certain period of time. The result output module accumulates the single Euler angles obtained in step 6). When the data accumulation of the Euler angles reaches a set amount, in this embodiment, the set amount is 500 groups, the mean of all Euler angles is obtained, and the corresponding extrinsic rotation matrix is calculated according to the mean of the Euler angles as the final calibration result output.
[0170] Compared with the prior art, the monocular camera automatic calibration method proposed in the present invention can realize the online dynamic calibration of the camera's extrinsic rotation matrix and Euler angles based on the camera's own image and vehicle body motion information. Compared with the traditional calibration method, the present method does not require high-precision calibration objects, nor does it require manual participation, and can realize the calibration and compensation of the camera rotation matrix under the vehicle's motion state; compared with the conventional automatic calibration method, the present method does not require the assistance of road lane lines, and can realize the calibration of cameras at all installation angles, not just front and rear view cameras. At the same time, the present method has low hardware requirements and does not require strict implementation conditions. It has the advantages and characteristics of simplicity, efficiency, and accuracy, and is suitable for a variety of vehicle models. The present invention can realize the automatic calibration of the camera rotation matrix or Euler angles under the condition that the vehicle is driving in a straight line, and does not require road lane line information, and can be used for the calibration of front, rear, and side view cameras. The present invention only relies on the camera to input continuous frame data and the vehicle's own motion information, and can realize the online calibration of the three rotation Euler angles of the camera. It does not need to rely on calibration plates, lane lines or other information and manual participation. It can correct small deflections of the camera caused by vehicle vibration or installation deviation. The method is highly flexible and simple and convenient to implement.
[0171] Finally, it should be noted that the above embodiments are only used to illustrate the technical solution of the present invention rather than to limit the technical solution. Those skilled in the art should understand that those modifications or equivalent substitutions of the technical solution of the present invention that do not depart from the purpose and scope of the technical solution should be included in the scope of the claims of the present invention.
Claims
1. The key point-based vehicle-mounted monocular camera extrinsic automatic calibration system is characterized by: include: A feature point extraction and matching module is used to extract feature points in continuous frame images and match them; The relative motion solving module is used to solve the relative motion between consecutive frames based on feature point pairs and epipolar geometry method; The homography solving module is used to extract the feature points on the road surface and select the road surface feature points with high matching according to the homography relationship; The extrinsic parameter solving module is used to solve the camera extrinsic parameters based on the relative motion of continuous frames and the road feature point pairs. When solving the camera extrinsic parameters, three orthogonal unit vectors are constructed based on the relative displacement vector as the three components of the camera extrinsic parameter rotation matrix, and one undetermined parameter is retained. Then, two pairs of feature point pairs are selected from the successfully matched pairs to obtain all possible selection methods. Each group of selections is solved to obtain an undetermined parameter and its corresponding extrinsic parameter rotation matrix. The corresponding Euler angle can be obtained through the conversion relationship between the extrinsic parameter rotation matrix and the Euler angle. If the difference between the obtained Euler angle and the initial value is greater than 5 degrees, it is considered to be an outlier and discarded. If the difference between the obtained Euler angle and the initial value is less than or equal to 5 degrees, all Euler angles that meet the requirements are retained and their average is calculated. The corresponding extrinsic parameter rotation matrix is calculated based on the average value of the Euler angle as the camera Euler angle and extrinsic parameter rotation matrix obtained by this calculation and calibration. The result output module is used to accumulate the results obtained by the external parameter solution module, and after reaching a fixed amount, calculate the average and finally output it.
2. A method for automatic calibration of extrinsic parameters of a vehicle-mounted monocular camera based on key points, characterized in that: The automatic calibration system for vehicle-mounted monocular camera extrinsic parameters based on key points as claimed in claim 1 comprises the following steps: Step 1) Initial value setting; Step 2) feature point extraction and matching, wherein the feature point extraction and matching module extracts feature points in continuous frame images and matches them; Step 3) solving the relative motion, the relative motion solving module uses the epipolar geometry method to solve the relative motion between consecutive frames based on the feature point pairs; Step 4) extracting and matching road surface feature points, the homography solving module extracts and matches road surface feature points; Step 5) homography solving, the homography solving module performs homography solving; Step 6) extrinsic parameter calculation, wherein the extrinsic parameter calculation module calculates the camera extrinsic parameters based on the continuous frame relative motion and the road feature point pairs; Step 7) cumulative output: the result output module accumulates the results obtained by the external parameter solving module, and after reaching a fixed amount, calculates the average and finally outputs it.
3. The method for automatic calibration of extrinsic parameters of a vehicle-mounted monocular camera based on key points according to claim 2, characterized in that: The setting of the initial value in step 1) includes the setting of the world coordinates of the camera installation position and the setting of the initial value of the camera Euler angle; The method for setting the camera installation position is as follows: establish a world coordinate system with the ground below the center of the rear axle of the vehicle as the origin of the world coordinate system, the forward direction of the vehicle as the positive direction of the X-axis of the world coordinate system, and the direction perpendicular to the ground and upward as the positive direction of the Z-axis of the world coordinate system, and determine the world coordinates of the camera installation position according to the established world coordinate system; The method for setting the initial value of the camera Euler angle is: using the known state value of the camera Euler angle last time as the initial value of the camera Euler angle.
4. The method for automatic calibration of extrinsic parameters of a vehicle-mounted monocular camera based on key points according to claim 3 is characterized in that: The initial value setting in step 1) also includes the camera's continuous frame images Im1 and Im2, the vehicle body speed and the vehicle body angular velocity. The continuous frame images are all time stamped. The time difference between the two camera frames is calculated by the time stamps of the two continuous frame images, and the relative motion between the two camera frames is determined by solving the epipolar constraint between the two camera frames. When the angular velocity of the vehicle body is greater than 1 degree / second, it is considered that the vehicle is not moving in a straight line and the calculation is exited.
5. The method for automatic calibration of extrinsic parameters of a vehicle-mounted monocular camera based on key points according to claim 4, characterized in that: In step 2), when the feature point extraction and matching module extracts and matches feature points in consecutive frame images, it first performs corner point detection on the two input frames, then extracts the corner points and their feature vectors in the images, and then matches the feature points based on the Hamming distance between the feature vectors, and finally retains all successfully matched feature point pairs.
6. The method for automatic calibration of extrinsic parameters of a vehicle-mounted monocular camera based on key points according to claim 5, characterized in that: In step 2), feature points in continuous frame images are extracted using the ORB corner detection method, and the number of feature points extracted from each frame image is less than or equal to 3000; Feature points are matched based on the Hamming distance between feature vectors, and the top 30% with the highest matching degree are selected as the final matching results.
7. The key point-based automatic calibration method for vehicle-mounted monocular camera extrinsic parameters according to claim 6, characterized in that: When solving the relative motion in step 3), first select the feature point pairs with high matching degree in step 2), then solve the essential matrix between the two frames of images based on the feature point pairs, and finally solve the relative rotation matrix and relative displacement vector between the two frames of images based on the essential matrix.
8. The key point-based automatic calibration method for vehicle-mounted monocular camera extrinsic parameters according to claim 7, characterized in that: In step 3), if the extracted feature point pairs are less than 8 pairs, the calculation is exited. If the extracted feature point pairs are greater than or equal to 8 pairs, the five-point method is used to solve the essential matrix between the two frames of images. After completing the essential matrix calculation, the relative rotation matrix and relative translation vector between the two frames of images are calculated based on the essential matrix.
9. The key point-based automatic calibration method for vehicle-mounted monocular camera extrinsic parameters according to claim 8, characterized in that: In step 3), it is also necessary to determine whether the relative motion state estimation of the camera is correct. If it is correct, execute step 4. If it is not correct, exit the calculation. The method to determine whether the relative motion state estimation of the camera is correct is: 1) Calculate the initial rotation matrix R based on the initial Euler angles [ω0,θ0,τ0] e0 : 2) Calculate the relative translation vector tc0 based on the initial value; for a vehicle in a translational state, the normalized translation vector between two consecutive frames of images is t car =[-1,0,0], then according to the coordinate system rotation and translation transformation relationship, the relative translation vector tc0 of the camera is: t c0 =R e0 *t car ; 3) Compare the distance between the relative translation vector tc of the estimated value and the relative translation vector tc0 of the initial value, set a threshold, if the distance between the relative translation vector tc of the estimated value and the relative translation vector tc0 of the initial value is greater than the threshold, exit the calculation; if the distance between the relative translation vector tc of the estimated value and the relative translation vector tc0 of the initial value is less than or equal to the threshold, execute step 4).
10. The key point-based automatic calibration method for vehicle-mounted monocular camera extrinsic parameters according to claim 9, characterized in that: In step 4), when extracting and matching road feature points, a method of limiting the feature point extraction area is used to reduce the number of non-road feature points, and the feature points are matched based on the vehicle body motion information and local image information.
11. The method for automatic calibration of extrinsic parameters of a vehicle-mounted monocular camera based on key points according to claim 10, characterized in that: Set the width of the camera screen to W and the height to H, then the feature point extraction area of the front view camera and rear view camera is The feature point extraction area of the side view camera is 12. The method for automatic calibration of extrinsic parameters of a vehicle-mounted monocular camera based on key points according to claim 11, characterized in that: The method for matching feature points based on vehicle body motion information and local image information in step 4) is as follows: a camera model is established based on an initial input value, and then the pixel coordinates of the road feature points in the first frame image are converted into world coordinates, and the position of the same feature point in the second frame image is predicted based on the vehicle body motion information, and at the same time, a square local image of the first frame image is captured with the feature point in the first frame image as the center and the first set length as the side length as the image to be matched. After the position prediction is completed, the world coordinates of the predicted position are converted into pixel coordinates again, and a search area is constructed with the feature point as the center and the second set length as the radius, and the feature points in the second frame image within the search area are searched. If there are corresponding feature points in the area, a square local image of the second frame image is captured with the feature point as the center and the first set length as the side length to match the local image of the feature points in the first frame image, and the feature point with the highest matching degree is selected as the final matching point. If there are no corresponding feature points in the area, the current feature point matching fails and the next matching starts.
13. The key point-based automatic calibration method for vehicle-mounted monocular camera extrinsic parameters according to claim 12, characterized in that: When solving the homography in step 5), the homography relationship in the two frames of images is solved based on the feature point pairs that are successfully matched in step 4); for points in the two frames of images that are located on the same plane, their pixel coordinates satisfy the homography relationship, and the feature point pairs that satisfy the homography are screened out through the homography solution; If there are less than 4 pairs of successfully matched feature points in step 4), the homography solution fails and the calculation ends. If there are at least 4 pairs of feature point pairs that are successfully matched in step 4), the least squares method is used to estimate the homography matrix, and all feature point pairs that satisfy the homography relationship are retained.
14. The key point-based automatic calibration method for vehicle-mounted monocular camera extrinsic parameters according to claim 13, characterized in that: The calculation method of the camera's external parameter rotation matrix in step 6) is: 1) Based on the relative translation vector t obtained in step 3) c , construct a set of standard orthogonal vectors r1, n1, n2, whose expressions are as follows: n2=r1×n1 Where: r 11 ,r 21 ,r 31 are the three components of r1; Based on the above standard orthogonal vectors, a single-parameterized camera rotation matrix can be constructed: R e (α)=[r1 -r1×r3(α) r3(α)] r3(α)=cos(α)n1+sin(α)n2 Where: α is the unknown parameter, r1 is the first column of the rotation matrix, r3 is the third column of the rotation matrix, R e is the rotation matrix; 2) Arrange and combine all the feature point pairs extracted in step 5) to obtain all possible extraction methods. If there are N pairs of feature point pairs in step 5), there are N extraction methods. kind; 3) Taking the i-th combination as an example, solve the coordinates of its feature point in the camera coordinate system. If the pixel homogeneous coordinates of its feature point are m p11 ,m p12 ,m p21 ,m p22 , then its camera coordinate system coordinates are: Where: K is the intrinsic parameter matrix of the camera, m c11 ,m c12 ,m c21 ,m c22 is the camera coordinate system coordinate of the feature point, [u c11 ,y c12 ,y c21 ,u c22 ],[v c11 ,v c12 ,v c21 v c2 ] are the x-axis component and y-axis component of each coordinate of the feature point camera coordinate system. 4) Based on the coordinates of the feature point camera coordinate system, solve the rotation matrix parameters. The calculation method is:
15. The key point-based automatic calibration method for vehicle-mounted monocular camera extrinsic parameters according to claim 14, characterized in that: In step 6), the calculation method of the dynamic calibration value of the camera Euler angle is: If the rotation matrix of the camera extrinsic parameters is obtained: The obtained Euler angle is: Where: R e11 To R e33 is the rotation matrix R e The nine components of e ,θ e ,τ e They are the three components of the Euler angle; If the difference between the calculated Euler angle and the initial value is greater than 5 degrees, it will be discarded. If it is less than or equal to 5 degrees, it will be retained. All Euler angles that meet the requirements are saved and the average is calculated, which is the dynamic calibration value of the camera Euler angle calculated this time.
16. The key point-based automatic calibration method for vehicle-mounted monocular camera extrinsic parameters according to claim 15, characterized in that: In step 7), the result output module accumulates the single Euler angles obtained in step 6). When the accumulated data of the Euler angles reaches a set amount, the mean of all Euler angles is obtained, and the corresponding external parameter rotation matrix is calculated according to the mean of the Euler angles as the final calibration result output.
