Vehicle-mounted foresight camera calibration method, electronic equipment and vehicle

Abstract

The invention discloses an external parameter calibration method for a vehicle-mounted foresight camera. The method comprises the following steps: screening an image sample sequence of a straight vehicle from an image stream shot by the vehicle-mounted foresight camera; obtaining a to-be-detected image from the image sample sequence of straight driving of the vehicle; obtaining a lane line linear equation and a first vanishing point in the to-be-detected image through a lane line geometric method, storing the lane line linear equation into a lane line sequence, and storing the first vanishing point into a linear vanishing point sequence; obtaining a second vanishing point of the to-be-detected image through a feature point method, and storing the second vanishing point into a feature vanishing point sequence; performing stability detection on the linear vanishing point sequence and the feature vanishing point sequence based on the variance of vanishing point coordinates, and selecting a vanishing point sequence with higher stability as a to-be-calibrated vanishing point sequence; and calibrating external parameters of the vehicle-mounted foresight camera by combining the lane line sequence, the to-be-calibrated vanishing point sequence and the internal parameters of the vehicle-mounted foresight camera.

Description

Technical Field

[0001] The present application relates to the field of intelligent driving technology, and in particular to a vehicle-mounted forward-looking camera calibration method, electronic equipment, and vehicle. Background Art

[0002] With the development of intelligent driving technology, on-board cameras, as a key perception tool, are playing an increasingly important role in intelligent driving tasks. Forward-facing cameras are typically installed at the front of the vehicle to perceive road conditions ahead. The external parameters of forward-facing cameras can affect the accuracy of many intelligent driving tasks, such as ranging and lane keeping. Therefore, calibration of these external parameters is crucial.

[0003] Using a calibration plate to calibrate the external parameters of a vehicle's forward-looking camera requires precise measurement of the plate's placement. This requires specialized personnel to measure and record parameters, making it unsuitable for post-production scenarios. Implementing extrinsic calibration of forward-looking cameras while the vehicle is in motion is a pressing issue. Summary of the Invention

[0004] The embodiments of the present application provide a vehicle-mounted forward-looking camera calibration method, electronic equipment, and vehicle, thereby solving the problem of difficulty in performing external parameter calibration of the vehicle-mounted forward-looking camera during driving.

[0005] In a first aspect, an embodiment of the present application provides a vehicle-mounted forward-looking camera calibration method, comprising:

[0006] Filtering a sequence of image samples of a vehicle traveling straight ahead from an image stream captured by a front-view camera on the vehicle;

[0007] Acquire an image to be detected from a sequence of image samples of the vehicle traveling straight;

[0008] Obtaining a lane line equation and a first vanishing point in the image to be detected by a lane line geometry method, storing the lane line equation in a lane line sequence, and storing the first vanishing point in a linear vanishing point sequence;

[0009] Obtaining a second vanishing point of the image to be detected by a feature point method and storing the result in a feature vanishing point sequence;

[0010] performing stability testing on the linear vanishing point sequence and the characteristic vanishing point sequence based on the variance of the vanishing point coordinates, and selecting a vanishing point sequence with higher stability as the vanishing point sequence to be calibrated;

[0011] The extrinsic parameters of the vehicle-mounted front-view camera are calibrated by combining the lane line sequence, the sequence of vanishing points to be calibrated, and the intrinsic parameters of the vehicle-mounted front-view camera.

[0012] Optionally, filtering a sequence of image samples of a vehicle traveling straight ahead from an image stream captured by a vehicle-mounted forward-looking camera includes:

[0013] A sequence of image samples of a vehicle traveling straight is filtered from the image stream according to lane line shape changes, lane line clarity, and lane flatness.

[0014] Optionally, before performing stability detection on the linear vanishing point sequence and the characteristic vanishing point sequence, the method further includes:

[0015] adjusting a cumulative number threshold of vanishing points in the linear vanishing point sequence and the characteristic vanishing point sequence based on at least one of lane line clarity, lane smoothness, and lane slope, wherein a higher lane clarity leads to a higher cumulative number threshold of vanishing points, a higher lane smoothness leads to a higher cumulative number threshold of vanishing points, and a higher lane slope leads to a lower cumulative number threshold of vanishing points;

[0016] After the cumulative number of vanishing points in the linear vanishing point sequence and the characteristic vanishing point sequence reaches the cumulative number threshold of vanishing points, acquiring the image to be detected is stopped.

[0017] Optionally, obtaining a lane line equation and a first vanishing point in the image to be detected by a lane line geometry method, storing the lane line equation in a lane line sequence, and storing the first vanishing point in a linear vanishing point sequence includes:

[0018] Performing distortion correction on the image to be detected to obtain a dedistorted image;

[0019] Extracting straight line edge points in the dedistorted image;

[0020] Optionally, performing straight line fitting on the straight line edge points to obtain at least four lane line equations, using the at least four lane line equations to represent the at least four lane lines, and storing the at least four lane line equations in a lane line sequence;

[0021] A plurality of intersections of the at least four lane lines are obtained, and an intersection fusion point of the plurality of intersections is stored as the first vanishing point in the linear vanishing point sequence, where the intersection fusion point is obtained by calculating a coordinate mean or clustering of the plurality of intersections.

[0022] Optionally, performing distortion correction on the image to be detected to obtain a dedistorted image includes:

[0023] Performing distortion correction on the image to be detected by using the RecRecNet algorithm;

[0024] The extracting straight line edge points from the dedistorted image includes:

[0025] Extracting straight line edge points in the dedistorted image using any one of EDTER, Canny, Sobel, and Laplace operators;

[0026] The performing straight line fitting on the straight line edge points includes:

[0027] Linear fitting is performed on the edge points of the line using a least squares method or a gradient descent method.

[0028] Optionally, obtaining the second vanishing point of the image to be detected by a feature point method includes:

[0029] Obtaining a feature matching pair between the image to be detected and a previous frame of the image to be detected by any one of the SIFT algorithm, ORB algorithm, nearest neighbor matching algorithm, SuperGlue algorithm, and LightGlue algorithm;

[0030] The two-dimensional coordinates of the feature matching pair are calibrated, and based on the calibrated feature matching pair, at least four straight lines are drawn using lane line equations, multiple intersection points of the at least four straight lines are obtained, and an intersection fusion point of the multiple intersection points is used as the second vanishing point. The intersection fusion point is obtained by calculating the coordinate mean or clustering of the multiple intersection points.

[0031] Optionally, obtaining the second vanishing point of the image to be detected by a feature point method includes:

[0032] Obtain multiple light loss amounts in the image to be detected and the previous frame image by using an optical flow method, extend the translation directions of the multiple light loss amounts in the two frames of image to obtain at least four extension lines, and obtain multiple intersection points of at least four straight lines where the at least four extension lines are located.

[0033] An intersection fusion point of the multiple intersection points is used as the second vanishing point, and the intersection fusion point is obtained by calculating a coordinate mean or a cluster center of the multiple intersection points.

[0034] Optionally, acquiring the image to be detected from the image sample sequence of the vehicle traveling straight includes:

[0035] Determine the sampling period of the image based on the sum of the computation time of detecting the lane line equation and the computation time of the vanishing point in a single image to be detected;

[0036] Calculating a maximum Euclidean distance of all vanishing points in the vanishing point sequence to be calibrated according to a sampling period, and deleting the first element in the lane line sequence, the linear vanishing point sequence, and the feature vanishing point sequence when the maximum Euclidean distance is greater than one twentieth of the width of the image to be detected;

[0037] A new image to be detected is obtained from the image sample sequence of the vehicle moving straight.

[0038] Optionally, based on the variance of vanishing point coordinates, performing stability detection on the linear vanishing point sequence and the characteristic vanishing point sequence, and selecting a vanishing point sequence with higher stability as the vanishing point sequence to be calibrated, includes:

[0039] Calculating a first variance of the coordinates of all vanishing points in the linear vanishing point sequence and a second variance of the coordinates of all vanishing points in the characteristic vanishing point sequence, and clearing the linear vanishing point sequence and the characteristic vanishing point sequence if a difference between the first variance and the second variance is greater than a variance threshold, wherein the variance threshold is one tenth of the width of the image to be detected;

[0040] If the difference between the first variance and the second variance is not greater than the variance threshold, the vanishing point sequence with the smaller variance is used as the vanishing point sequence to be calibrated.

[0041] Optionally, the calibrating the extrinsic parameters of the vehicle-mounted front-view camera by combining the lane line sequence, the sequence of vanishing points to be calibrated, and the intrinsic parameters of the vehicle-mounted front-view camera includes:

[0042] Storing lane line equations for at least two pairs of lane lines corresponding to each image to be detected in the lane line sequence;

[0043] For each image to be detected, the corresponding vanishing points in the vanishing point sequence to be calibrated are sequentially converted to the world coordinate system, and the roll value is optimized through a numerical iterative algorithm so that the at least two pairs of lane lines in the world coordinate system are rotated until they are equidistant and parallel.

[0044] In the dedistorted image corresponding to each image to be detected, according to the lane line equations of the at least two pairs of lane lines, the optimized roll value is converted using the vanishing point principle to obtain the pitch value and yaw value of the vehicle-mounted front-view camera. The optimized roll value is adjusted by binary search until the width difference of the at least two pairs of lane lines is less than 10 -5 The current roll value, the current pitch value, and the current yaw value are used as external parameters of the vehicle-mounted front-view camera.

[0045] In a second aspect, an embodiment of the present application provides an electronic device comprising: at least one memory and at least one processor, wherein the at least one memory stores executable code, and the at least one processor is used to execute the executable code in the at least one memory to implement the above-mentioned external parameter calibration method for a vehicle-mounted forward-looking camera.

[0046] The embodiments of the present application provide a method for calibrating extrinsic parameters of a vehicle-mounted forward-view camera, which does not require the additional provision of a calibration plate or the manual acquisition of calibrated initial extrinsic parameters. Furthermore, vanishing points are calculated for a straight-ahead vehicle image using a lane line geometry method and a feature point method, respectively. The vanishing points are stored in a vanishing point sequence, and the variance of the vanishing point coordinates in the vanishing point sequence is calculated to select a more stable vanishing point sequence for calculating the extrinsic parameters of the vehicle-mounted forward-view camera. The stability of the vanishing point sequence indicates that the vanishing points are substantially located at the same position in the image. This method combines information between lane lines and spatial feature points for mutual verification, avoiding the problem of large extrinsic parameter errors caused by the non-parallelism between the vehicle and the lane lines when using lane lines alone to calibrate the extrinsic parameters. This improves the accuracy of the extrinsic parameters of the vehicle-mounted forward-view camera, enhances reliability, and can better assist in subsequent intelligent driving tasks. BRIEF DESCRIPTION OF THE DRAWINGS

[0047] Figure 1 A schematic diagram of a calibration environment for a vehicle-mounted front-view camera according to an embodiment of the present application;

[0048] Figure 2 A schematic flow chart of a calibration method for a vehicle-mounted front-view camera according to an embodiment of the present application;

[0049] Figure 3 This is a schematic diagram of the process of obtaining the vanishing point using the lane line geometry method in an embodiment of the present application;

[0050] Figure 4 This is a schematic diagram of obtaining the vanishing point using the lane line geometry method in an embodiment of the present application;

[0051] Figure 5 A schematic diagram of obtaining vanishing points using the feature point method in an embodiment of the present application;

[0052] Figure 6 This is a schematic diagram of obtaining the vanishing point using the optical flow method in an embodiment of the present application;

[0053] Figure 7 A schematic diagram of the vehicle body world coordinate system in an embodiment of the present application;

[0054] Figure 8 Schematic diagram of the coordinate system conversion process in an embodiment of the present application;

[0055] Figure 9 Schematic diagram of the process of performing a binary search on roll in an embodiment of the present application. DETAILED DESCRIPTION

[0056] The present application will be described in detail below in conjunction with the specific embodiments shown in the accompanying drawings, but these embodiments do not limit the present application. Structural, methodological, or functional changes made by ordinary technicians in this field based on these embodiments are included in the scope of protection of the present application.

[0057] If the extrinsic parameter calibration of the vehicle-mounted forward-looking camera is performed directly based on the lane lines parallel to the driving direction, the vehicle's position relative to the lane lines may move due to changes in the lane lines, the influence of driving habits, uneven road surface, etc., causing the vehicle's driving direction to be not completely parallel to the lane lines. This will lead to inaccurate extrinsic parameter calibration and affect the final extrinsic parameter calibration accuracy.

[0058] See also Figure 1 , a schematic diagram of the calibration environment in which the vehicle-mounted forward-looking camera of an embodiment of the present application is located is taken in a road driving scenario.

[0059] The present invention provides a method for calibrating external parameters of a vehicle-mounted front-view camera, including the following steps:

[0060] Filtering a sequence of image samples of a vehicle traveling straight ahead from an image stream captured by a front-view camera on the vehicle;

[0061] Acquire an image to be detected from a sequence of image samples of a vehicle traveling straight ahead;

[0062] Obtaining the first vanishing point of the image to be detected by a lane line geometry method and storing it in a linear vanishing point sequence;

[0063] Obtain the second vanishing point of the image to be detected by the feature point method and store it in a feature vanishing point sequence;

[0064] Based on the variance of the vanishing point coordinates, a stability test is performed on the linear vanishing point sequence and the characteristic vanishing point sequence, and a sequence with higher stability is selected as the vanishing point sequence to be calibrated;

[0065] The extrinsic parameters of the vehicle-mounted front-view camera are calibrated by combining the vanishing point coordinates in the vanishing point sequence to be calibrated and the intrinsic parameters of the vehicle-mounted front-view camera.

[0066] In an embodiment of the present application, a sequence of image samples of a vehicle traveling straight ahead can be filtered from an image stream captured by a vehicle's forward-looking camera based on lane shape changes, lane clarity, and lane smoothness. For example, a deep learning algorithm can be used to detect lane shape in multiple consecutive images in the image stream. If the lane shape remains essentially unchanged, the vehicle is determined to have been traveling straight ahead during the corresponding time period of these images. For multiple images of a vehicle traveling straight ahead, if the deep learning algorithm detects that the lane clarity meets a preset clarity threshold and the lane smoothness meets a preset smoothness threshold, the image is stored in the sequence of image samples of the vehicle traveling straight ahead.

[0067] Optionally, a deep learning algorithm can be used to simultaneously detect the shape changes, lane line clarity, and lane smoothness of multiple consecutive images in the image stream, and several images in which the lane line shape remains basically unchanged, the lane line clarity meets a preset clarity threshold, and the lane smoothness meets a preset smoothness threshold are stored in an image sample sequence of the vehicle moving straight.

[0068] The embodiments of the present application provide a method for calibrating extrinsic parameters of a vehicle-mounted forward-view camera, which does not require the additional provision of a calibration plate or the manual acquisition of calibrated initial extrinsic parameters. Furthermore, vanishing points are calculated for a straight-ahead vehicle image using a lane line geometry method and a feature point method, respectively. The vanishing points are stored in a vanishing point sequence, and the variance of the vanishing point coordinates in the vanishing point sequence is calculated to select a more stable vanishing point sequence for calculating the extrinsic parameters of the vehicle-mounted forward-view camera. The stability of the vanishing point sequence indicates that the vanishing points are substantially located at the same position in the image. This method combines information between lane lines and spatial feature points for mutual verification, avoiding the problem of large extrinsic parameter errors caused by the non-parallelism between the vehicle and the lane lines when using lane lines alone to calibrate the extrinsic parameters. This improves the accuracy of the extrinsic parameters of the vehicle-mounted forward-view camera, enhances reliability, and can better assist in subsequent intelligent driving tasks.

[0069] See also Figure 2 As shown, the external parameter calibration method in the embodiment of the present application can be implemented in the following ways, but is not limited to:

[0070] S201: Acquire images to be detected from a sequence of image samples of a vehicle traveling straight ahead in a time sequence.

[0071] S202: Obtaining a lane line equation and a first vanishing point in the image to be detected using a lane line geometry method;

[0072] S203: Obtaining a second vanishing point in the image to be detected by a feature point method;

[0073] S204: Determine whether the queue length of the vanishing point sequence is greater than the cumulative threshold N of the number of vanishing points, that is, determine whether the cumulative number of vanishing points in the linear vanishing point sequence and the feature vanishing point sequence is greater than the cumulative threshold of the number of vanishing points;

[0074] If step S205 is executed,

[0075] Otherwise, the storage sequence operation is performed, that is, the first vanishing point in the image to be detected is stored in the linear vanishing point sequence, the lane line equation is stored in the lane line sequence, and the second vanishing point in the image to be detected is stored in the feature vanishing point sequence; and the process returns to S201.

[0076] S205: Determine whether the difference between the two sequences is greater than THR, that is, determine whether the difference between the variance of all vanishing point coordinates in the linear vanishing point sequence and the variance of all vanishing point coordinates in the feature vanishing point sequence is greater than the variance threshold; if so, execute S206; if not, execute S207;

[0077] S206: Clear all vanishing points in the two sequences and remind the driver to adjust the steering wheel.

[0078] For example, the driver can be prompted to keep the vehicle parallel to the lane through voice or display.

[0079] After clearing all the vanishing points of the two sequences, the process may return to step S201 to obtain a new batch of images to be detected.

[0080] S207: Using the vanishing point sequence with smaller variance, combined with the lane line sequence and the intrinsic parameters of the vehicle-mounted front view camera, calibrate the extrinsic parameters of the vehicle-mounted front view camera.

[0081] The vanishing point sequence with the smaller variance between the linear vanishing point sequence and the characteristic vanishing point sequence is used as the vanishing point sequence to be calibrated. The extrinsic parameters of the front-view camera are calibrated by combining the lane line sequence, the vanishing point sequence to be calibrated, and the intrinsic parameters of the onboard front-view camera. Since vanishing point sequences with large variances are unstable, choosing a vanishing point sequence with a smaller variance provides greater stability and accuracy, resulting in more accurate subsequent calibration results.

[0082] The aforementioned cumulative vanishing point threshold can be adjusted based on at least one of lane clarity, lane smoothness, and lane slope. The higher the lane clarity, the higher the cumulative vanishing point threshold; the higher the lane smoothness, the higher the cumulative vanishing point threshold; and the higher the lane slope, the lower the cumulative vanishing point threshold. For example, if the lane lines are unclear or uneven, the cumulative vanishing point threshold N = 5. If the lane lines are clear, the lane smoothness is high, and the lane slope is close to 0, the cumulative vanishing point threshold can be adjusted to 8, 9, 10, or even higher values.

[0083] In an embodiment of the present application, when the cumulative number of vanishing points in a vanishing point sequence reaches a threshold, the stability of the vanishing point sequence can be verified using the Euclidean distance between the vanishing points and the image width. The Euclidean distance between existing vanishing points in the vanishing point sequence is calculated. When the maximum Euclidean distance exceeds the verification threshold, the first vanishing point in the queue and the corresponding lane line equation are deleted from the sequence, and the process returns to the step of obtaining the image to be detected.

[0084] In step 203 , frames may be skipped to obtain images from the image sample sequence of the vehicle traveling straight ahead according to a sampling period. The sampling period may be determined based on the calculation time of the lane line equation and the calculation time of a single vanishing point.

[0085] Optionally, a sampling period for an image can be determined based on the sum of the computational time required to detect lane line equations and the computational time required to detect vanishing points in a single image to be detected. Based on the sampling period, a maximum Euclidean distance of all vanishing points in the vanishing point sequence to be calibrated is calculated. When the maximum Euclidean distance is greater than one twentieth of the width of the image to be detected, the first element in the lane line sequence, the linear vanishing point sequence, and the feature vanishing point sequence is deleted. A new image to be detected is obtained from the sample sequence of images of the vehicle traveling straight ahead.

[0086] The variance threshold in step 205 may be one tenth of the width of the image to be detected.

[0087] For example, the lane line geometry method can be used to obtain the first vanishing point in the image to be detected by the following method:

[0088] Step 301: Initialize the linear vanishing point sequence VP_line=[] and the lane line equation sequence LANE_line=[].

[0089] Step 302: Perform distortion correction on the image to be detected to obtain a dedistorted image.

[0090] A schematic diagram of the dedistorted image can be found in Figure 4 shown.

[0091] For example, the image to be detected may be subjected to distortion correction by using a polynomial distortion correction method, a RecRecNet (rectangular correction network) algorithm, or the like.

[0092] The RecRecNet algorithm is a deep learning model used for wide-angle image distortion correction. When the onboard front-view camera is a wide-angle camera, the RecRecNet algorithm can better correct distortion, thereby making subsequent lane line detection results more accurate.

[0093] Step 303: Extract the straight line edge points in the dedistorted image, perform straight line fitting on the straight line edge points using the lane line equation to obtain at least four straight lines, and store the at least four straight lines as at least four lane lines in the lane line sequence LANE_line.

[0094] Line edge points can be extracted using any of the following operators: EDTER, Canny, Sobel, or Laplacian. EDTER (Edge Detection with Transformer) is a two-stage edge detection model based on the Transformer architecture. It improves the accuracy of image edge detection by fusing global contextual information with local fine-grained features.

[0095] The Canny operator is a multi-stage edge extraction algorithm that can find the optimal edge detection that meets the following three conditions through variational methods:

[0096] Condition 1: The algorithm can identify as many actual edges in the image as possible;

[0097] Condition 2: The identified edges should be as close as possible to the actual edges in the actual image;

[0098] Condition 3: An edge in an image can only be identified once, and any image noise that may exist is not identified as an edge.

[0099] The Sobel operator is a traditional edge extraction algorithm based on image segmentation. It can detect edges by adding the weighted difference of the grayscale values ​​of the four areas of the top, bottom, left, and right of each pixel in the image and reaching an extreme value at the edge.

[0100] The Laplace operator can obtain the second-order derivative value by calculating the Laplace operator of the image, perform threshold processing on the second-order derivative value to obtain binary edge information, and then remove interference and noise through morphological operations.

[0101] The above process of fitting the edge points of the lane line using the lane line equation can be implemented using algorithms such as the least squares method, Hough transform, or gradient descent method.

[0102] For example, the sum of squared errors between the observed values ​​and the model predicted values ​​can be minimized using the least squares method, and the partial derivatives of the error function can be solved and set to zero to obtain the slope and intercept of the best fit line.

[0103] For example, the gradient descent method can be used to calculate multiple straight line edge points, continuously iteratively update the weights and intercepts, gradually reduce the loss function, and finally find the best fitting straight line.

[0104] Step 304: Obtain multiple intersection points of at least four straight lines, and store the intersection fusion point of the multiple intersection points as the first vanishing point in VP_line.

[0105] The intersection fusion point can be obtained by calculating the coordinate mean or clustering of multiple intersection points.

[0106] Exemplarily, the second vanishing point in the image to be detected may be obtained by a feature point method.

[0107] The feature point method can be implemented in the following two ways, but is not limited to:

[0108] Method 1:

[0109] A feature matching pair between the image to be detected and the previous frame of the image to be detected is obtained by using any of the following methods: SIFT (Scale-Invariant Feature Transform) algorithm, ORB (Oriented FAST and Rotated BRIEF) algorithm, nearest neighbor matching algorithm, SuperGlue algorithm, and LightGlue algorithm.

[0110] The SIFT algorithm extracts local feature points in an image that are invariant to scale, rotation, and illumination, and is used to perform feature matching between two images.

[0111] The ORB algorithm is an image feature detection and description algorithm that combines speed and robustness. It can add orientation calculation (Oriented) to FAST feature points through rotation invariance, and achieve matching stability by rotating the coordinate axes of the BRIEF descriptor. Through scale invariance, it processes features of different scales by constructing an image pyramid, thereby achieving feature matching between two images based on scale features.

[0112] The nearest neighbor matching algorithm can achieve feature matching between two images based on distance measurement.

[0113] SuperGlue and LightGlue are both image feature point extraction methods based on deep learning.

[0114] The SuperGlue algorithm constructs a graph structure between feature points and uses graph neural networks (GNN) to update node states and learn edge weights. The nodes in the graph structure represent feature points, and the edges represent the similarities between points. By constructing an image pyramid and performing feature extraction and matching at different levels, the feature matching of two images is achieved.

[0115] The LightGlue algorithm achieves fast and robust image feature matching by combining efficient feature extractors (such as SuperPoint) and optimized Transformer architecture.

[0116] The feature matching between the image to be detected and the previous frame of the image to be detected can refer to Figure 5 As shown. The two-dimensional coordinates of the feature matching pair are calibrated, and at least four straight lines are drawn based on the calibrated feature matching pair. Exemplarily, the two-dimensional coordinates of the feature matching pair can be calibrated using a polynomial distortion correction method, a RecRecNet algorithm, or the like. Multiple intersection points of the at least four straight lines are obtained, and the intersection fusion point of the multiple intersection points is used as the second vanishing point. The intersection fusion point is obtained by taking the coordinate mean or clustering of the multiple intersection points.

[0117] Method 2:

[0118] The optical flow method is used to obtain multiple light loss amounts in the image to be detected and the previous frame image, and the translation direction of the multiple light loss amounts in the two frames of images is extended to obtain at least 4 extension lines, and multiple intersection points of at least 4 straight lines where the at least 4 extension lines are located are obtained. The intersection fusion point of the multiple intersection points is used as the second vanishing point, and the intersection fusion point is obtained by calculating the coordinate mean or cluster center of the multiple intersection points.

[0119] For example, see Figure 6 As shown in the figure, the LK (Lucas-Kanada) method can be used to obtain the light loss from each pixel point Pt in the image to be detected to the corresponding pixel point Pt-1 in the previous image. These optical flow vectors can represent the displacement direction of each pixel point in two consecutive frames. When the camera moves horizontally, these optical flow vectors will converge towards the vanishing point. Therefore, after extending these light loss values, the intersection point Vp of the extended lines can be used as a candidate vanishing point. Multiple candidate vanishing points are clustered and fused as the vanishing point of the image to be detected.

[0120] See also Figure 7 and Figure 8 As shown, the extrinsic parameter calibration of the vehicle-mounted front-view camera is performed based on the N vanishing points in the vanishing point sequence to be calibrated and the lane line equation in the lane line sequence, which can be achieved in the following way.

[0121] In the lane line sequence, corresponding to each image to be detected, the lane line equations of at least two pairs of lane lines are stored;

[0122] For each image to be detected, the corresponding vanishing points in the vanishing point sequence to be calibrated are sequentially converted to the world coordinate system, and the roll angle roll value is optimized through a numerical iterative algorithm so that the at least two pairs of lane lines in the world coordinate system are rotated until they are equidistant and parallel.

[0123] In the dedistorted image corresponding to each image to be detected, the optimized roll value is converted through a relationship based on the lane line equations of at least two pairs of lane lines to obtain the pitch value and yaw value of the onboard forward-view camera. The optimized roll value, pitch value, and yaw value of the onboard forward-view camera are used as the extrinsic parameters of the onboard forward-view camera.

[0124] The above roll can represent the roll angle of the vehicle-mounted forward-looking camera in the world coordinate system, pitch represents the pitch angle of the vehicle-mounted forward-looking camera in the world coordinate system, and yaw represents the deflection angle of the vehicle-mounted forward-looking camera in the world coordinate system.

[0125] The initial roll value may be 0 or the roll value in the external parameters of the vehicle-mounted forward-looking camera stored before optimization, and is not limited here.

[0126] For example, see Figure 7 and Figure 8 As shown, the vehicle body world coordinate system Ow-XwYwZw where the vehicle-mounted forward-view camera is located can be converted into the camera coordinate system Oc-XcYcZc where the vehicle-mounted forward-view camera is located.

[0127] The first step is to rotate the vehicle body world coordinate system Ow-XwYwZw around Zw by yaw+pi / 2 so that the direction of Ow-XwYwZw is consistent with the coordinate system Ocw-XcwYcwZcw;

[0128] In the second step, the vehicle body world coordinate system Ow-XwYwZw is rotated along the Xw axis -pitch so that the Yw direction is consistent with the Zc direction;

[0129] The third step is to rotate the vehicle body world coordinate system Ow-XwYwZw along the Yw axis -roll so that the Xw direction is consistent with the Xc direction and the Zw direction is consistent with the -Yc direction;

[0130] Step 4: The vehicle body world coordinate system Ow-XwYwZw is rotated -90 degrees along the Xw axis, so that the directions of Zw and Zc are consistent, and the directions of Yw and Yc are consistent;

[0131] The fifth step is to translate the body world coordinate system Ow-XwYwZw coordinate system by T so that the body world coordinate system Ow-XwYwZw completely coincides with the camera coordinate system Oc-XcYcZc.

[0132] The above process can be implemented by referring to the following formula:

[0133]

[0134] Among them, R 33 is a 3*3 rotation matrix, T represents a 3*1 translation matrix, pitch represents the pitch angle of the vehicle-mounted front-view camera, yaw represents the deflection angle of the vehicle-mounted front-view camera, and roll represents the roll angle of the vehicle-mounted front-view camera.

[0135] (cam_x, cam_y, cam_z) represents the installation position of the vehicle's forward-looking camera in the vehicle's world coordinate system.

[0136] In an optional embodiment of the present application, the following formula 3 represents the conversion relationship between the pixel points of the dedistorted image obtained in step 302 and the world coordinate system, and Zw=0. Referring to formula 4, the coordinates of the points in the dedistorted image can be converted to the world coordinate system, and the homography matrix between the world coordinate system and the dedistorted image can be derived.

[0137]

[0138] Where α represents the depth scaling factor for converting a 3D image to a 2D image, fx and fy represent the intrinsic focal lengths of the vehicle's forward-looking camera, cx and cy represent the principal points of the vehicle's forward-looking camera, r represents the value in the rotation matrix R33, t represents the value in the translation parameter T, (u, v) represents the coordinates of the pixel point in the dedistorted image, and (Xw, Yw) represents the coordinates of the point in the dedistorted image converted to the world coordinate system.

[0139] Rotate the four lane lines in the image to be detected in the world coordinate system. When the four lane lines are parallel to the Xw coordinate axis in the world coordinate system, the angle between the lane lines and the Ycw axis is the yaw angle. At this time, the point on the lane line can be expressed as: (Ycw*tan(yaw),Ycw,0).

[0140] The coordinates of the vanishing point (u0, v0) on the lane line can be obtained by combining formulas 1-4 to find the values ​​of u0, v0 when the limit Ycw->∞ is reached.

[0141]

[0142] Where u0 represents the horizontal coordinate of the vanishing point, v0 represents the vertical coordinate of the vanishing point, fx and fy represent the focal lengths of the vehicle's forward-looking camera, cx and cy represent the coordinates of the principal point of the vehicle's forward-looking camera. pitch represents the pitch angle of the vehicle's forward-looking camera, yaw represents the yaw angle of the vehicle's forward-looking camera, and roll represents the roll angle of the vehicle's forward-looking camera.

[0143] When the roll angle roll, vanishing point coordinates (u0, v0) and camera intrinsic parameters (fx, fy, cx, cy) are known, the pitch angle and yaw angle of the vehicle-mounted forward-looking camera can be calculated as shown in the following formula 6:

[0144]

[0145] Where u0 represents the horizontal coordinate of the vanishing point, v0 represents the vertical coordinate of the vanishing point, fx and fy represent the focal lengths of the vehicle's forward-looking camera, cx and cy represent the coordinates of the principal point of the vehicle's forward-looking camera. pitch represents the pitch angle of the vehicle's forward-looking camera, yaw represents the yaw angle of the vehicle's forward-looking camera, and roll represents the roll angle of the vehicle's forward-looking camera.

[0146] Since the intersection of the four lane lines in the image to be detected by the on-board front-view camera can be fused to obtain the vanishing point, the coordinates of the vanishing point can be obtained. When the rotation angle of the on-board front-view camera is only related to the roll angle, the pitch and yaw can be obtained through roll conversion. In this way, the roll value can be optimized only through a numerical iterative algorithm. After rotating the lane lines so that the distance between two pairs of lane lines in the four lane lines is equal in width, the pitch and yaw can be calculated according to Formula 6. Then, the width difference of the left and right lane lines in the world coordinate system is obtained, and a binary search is performed on roll until the lane line width difference is less than 10 -5 Meters, the current roll value, current pitch value and current yaw value are used as the external parameters of the vehicle-mounted front view camera, thereby completing the angle external parameter calibration of the vehicle-mounted front view camera. For example, the width difference of each pair of lane lines can be obtained respectively, and then the two width differences are averaged to determine whether the average value is less than 10 -5 rice.

[0147] The initial roll value can be 0 or the roll value in the external parameters of the vehicle-mounted forward-looking camera stored before optimization, which is not limited here.

[0148] The process of binary search for roll is shown in Figure 9 As shown in the figure, the goal of the roll angle iteration is to make the left and right lane lines equal in width in the world coordinate system. The initial roll angle will make the left lane line much wider than the right lane line. As the binary search progresses, the roll angle continues to change. As the roll angle changes, the width of the left and right lane lines will also change accordingly, eventually achieving the effect of equal width.

[0149] The present application provides an extrinsic parameter calibration method for a vehicle-mounted forward-view camera. This method can remind the driver to adjust the steering wheel while the vehicle is in motion, select video frames in which the vehicle's direction of travel is parallel to the lane lines, and perform extrinsic parameter calibration without requiring an additional calibration plate or manually obtaining calibrated initial extrinsic parameters. By utilizing lane lines on the road and detected spatial feature points to complete the extrinsic parameter calibration for the vehicle-mounted forward-view camera, and by combining information between the lane lines and spatial feature points for mutual verification, the method avoids the problem of large extrinsic parameter errors caused by the non-parallelism between the vehicle and the lane lines when calibrating the extrinsic parameters using lane lines alone, thereby improving the accuracy of the extrinsic parameter calibration.

[0150] Based on the same inventive concept, an embodiment of the present application also provides an electronic device, comprising: at least one memory and at least one processor, the at least one memory storing executable code, and the at least one processor being used to execute the executable code in the at least one memory to implement the above-mentioned extrinsic parameter calibration method for a vehicle-mounted forward-looking camera.

[0151] An embodiment of the present application also provides a vehicle-mounted forward-looking camera for capturing an image stream; and a processor for executing the above-mentioned extrinsic parameter calibration method for the vehicle-mounted forward-looking camera.

[0152] The present application also provides a vehicle, including:

[0153] A forward-looking camera on the vehicle to capture an image stream; and

[0154] A processor is used to execute the above-mentioned extrinsic parameter calibration method for the vehicle-mounted front-view camera.

[0155] The above-mentioned vehicle-mounted forward-looking camera can be installed inside the vehicle windshield, on the roof, or outside the vehicle windshield, which is not limited here.

[0156] The at least one memory can be used to store a computer program, which may include instructions and data, and implement the steps of any of the above methods. The memory may be a random access memory, read-only memory, non-volatile memory, programmable ROM, erasable PROM, electrically erasable memory, flash memory, optical memory, registers, etc. Processor 801 may be a general-purpose processor, which can perform specific steps and / or operations by reading and executing a computer program stored in the memory. The general-purpose processor may use data stored in the memory during the execution of the steps and / or operations. The general-purpose processor may be a central processing unit, an ASIC, an FPGA, etc. The electronic device may also include a communication interface, which may include an input / output interface, a physical interface, and a logical interface for interconnecting devices within the network device. During implementation, the steps of the above method may be performed by hardware integrated logic circuits in the processor or by software instructions. The methods disclosed in conjunction with the embodiments of the present application can be directly implemented and executed by a hardware processor, or by a combination of hardware and software modules in the processor.

[0157] It should be understood that the processor in the embodiments of the present application may be a central processing unit (CPU), and the processor may also be other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field programmable gate arrays (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor may be a microprocessor or any conventional processor, etc.

[0158] It should also be understood that the memory in the embodiments of the present application may be a volatile memory or a non-volatile memory, or may include both volatile and non-volatile memories. Among them, the non-volatile memory may be a read-only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM), or a flash memory. The volatile memory may be a random access memory (RAM), which is used as an external cache. By way of example and not limitation, many forms of random access memory (RAM) are available, such as static RAM (SRAM), dynamic random access memory (DRAM), synchronous DRAM (SDRAM), double data rate synchronous dynamic random access memory (DDR SDRAM), enhanced synchronous dynamic random access memory (ESDRAM), synchronous link DRAM (SLDRAM), and direct rambus RAM (DR RAM).

[0159] The above embodiments can be implemented in whole or in part by software, hardware (such as circuits), firmware or any other combination. When implemented using software, the above embodiments can be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions or computer programs. When the computer instructions or computer program are loaded or executed on a computer, the process or function according to the embodiment of the present application is generated in whole or in part. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another computer-readable storage medium. For example, the computer instructions can be transmitted from one website, computer, server or data center to another website, computer, server or data center by infrared, microwave or other means. The computer-readable storage medium can be any available medium that can be accessed by a computer or a data storage device such as a server or data center that contains one or more available media sets. The available media can be magnetic media (for example, floppy disks, hard disks, tapes), optical media (for example, DVDs), solid-state drives, etc.

[0160] It should be understood that the term "and / or" as used herein simply describes a relationship between associated objects, indicating that three possible relationships exist. For example, "A and / or B" can represent: A alone, A and B together, or B alone. A and B can be singular or plural. Furthermore, the character " / " as used herein generally indicates an "or" relationship between the associated objects, but it may also indicate an "and / or" relationship. For specific understanding, please refer to the context.

[0161] In the present invention, "at least one" means one or more, and "plurality" means two or more. "At least one of the following" or similar expressions refers to any combination of these items, including any combination of single or plural items. For example, at least one of a, b, or c can mean: a, b, c, ab, ac, bc, or abc, where a, b, and c can be single or plural.

[0162] It should be noted that, in this document, relational terms such as first and second, etc., are used only to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any actual relationship or order between these entities or operations. Moreover, the terms "comprises," "comprising," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or device comprising a series of elements includes not only those elements, but also other elements not explicitly listed, or elements inherent to such process, method, article, or device. In the absence of further limitations, an element defined by the phrase "comprising a ..." does not exclude the presence of other identical elements in the process, method, article, or device comprising the element.

[0163] Each embodiment in this specification is described in a related manner. Similar portions between the embodiments can be referred to in conjunction with each other. Each embodiment focuses on the differences from other embodiments. In particular, the device embodiments are generally similar to the method embodiments, so their description is relatively simple. For related portions, refer to the description of the method embodiments.

[0164] The above are only preferred embodiments of the present application and are not intended to limit the scope of protection of the present application. Any modifications, equivalent replacements, improvements, etc. made within the spirit and principles of the present application are included in the scope of protection of the present application.

Claims

1. A method for calibrating external parameters of a vehicle-mounted front-view camera, characterized in that: include: Filtering a sequence of image samples of a vehicle traveling straight ahead from an image stream captured by a front-view camera on the vehicle; Acquire an image to be detected from a sequence of image samples of the vehicle traveling straight; Obtaining a lane line equation and a first vanishing point in the image to be detected by a lane line geometry method, storing the lane line equation in a lane line sequence, and storing the first vanishing point in a linear vanishing point sequence; Obtaining a second vanishing point of the image to be detected by a feature point method and storing the result in a feature vanishing point sequence; performing stability testing on the linear vanishing point sequence and the characteristic vanishing point sequence based on the variance of the vanishing point coordinates, and selecting a vanishing point sequence with higher stability as the vanishing point sequence to be calibrated; The extrinsic parameters of the vehicle-mounted front-view camera are calibrated by combining the lane line sequence, the sequence of vanishing points to be calibrated, and the intrinsic parameters of the vehicle-mounted front-view camera.

2. The method according to claim 1, wherein The step of filtering a sequence of image samples of a vehicle traveling straight ahead from an image stream captured by a vehicle-mounted front-view camera includes: A sequence of image samples of a vehicle traveling straight is filtered from the image stream according to lane line shape changes, lane line clarity, and lane flatness.

3. The method according to claim 1, wherein Before performing stability detection on the linear vanishing point sequence and the characteristic vanishing point sequence, the method further includes: adjusting a cumulative number threshold of vanishing points in the linear vanishing point sequence and the characteristic vanishing point sequence based on at least one of lane line clarity, lane smoothness, and lane slope, wherein a higher lane clarity leads to a higher cumulative number threshold of vanishing points, a higher lane smoothness leads to a higher cumulative number threshold of vanishing points, and a higher lane slope leads to a lower cumulative number threshold of vanishing points; After the cumulative number of vanishing points in the linear vanishing point sequence and the characteristic vanishing point sequence reaches the cumulative number threshold of vanishing points, acquiring the image to be detected is stopped.

4. The method according to claim 1, wherein Obtaining a lane line equation and a first vanishing point in the image to be detected by a lane line geometry method, storing the lane line equation in a lane line sequence, and storing the first vanishing point in a linear vanishing point sequence, including: Performing distortion correction on the image to be detected to obtain a dedistorted image; Extracting straight line edge points in the dedistorted image; Performing straight line fitting on the straight line edge points to obtain at least four lane line equations, using the at least four lane line equations to represent the at least four lane lines, and storing the at least four lane line equations in a lane line sequence; A plurality of intersections of the at least four lane lines are obtained, and an intersection fusion point of the plurality of intersections is stored as the first vanishing point in the linear vanishing point sequence, where the intersection fusion point is obtained by calculating a coordinate mean or clustering of the plurality of intersections.

5. The method according to claim 4, wherein The step of performing distortion correction on the image to be detected to obtain a dedistorted image includes: Performing distortion correction on the image to be detected by using the rectangular correction network RecRecNet algorithm; The extracting straight line edge points from the dedistorted image includes: Extracting straight line edge points in the dedistorted image using any one of a two-stage edge detection model EDTER, a Canny operator, a Sobel operator, or a Laplace operator; The performing straight line fitting on the straight line edge points includes: Linear fitting is performed on the edge points of the line using a least squares method or a gradient descent method.

6. The method according to claim 1, wherein Obtaining a second vanishing point of the image to be detected by a feature point method includes: Obtaining a feature matching pair between the image to be detected and a previous frame of the image to be detected by any one of a scale-invariant feature transform (SIFT) algorithm, a directional rapid rotation (ORB) algorithm, a nearest neighbor matching algorithm, a SuperGlue algorithm, and a LightGlue algorithm; The two-dimensional coordinates of the feature matching pair are calibrated, and based on the calibrated feature matching pair, at least four straight lines are drawn, multiple intersection points of the at least four straight lines are obtained, and the intersection fusion point of the multiple intersection points is used as the second vanishing point. The intersection fusion point is obtained by taking the coordinate mean or clustering of the multiple intersection points.

7. The method according to claim 1, wherein Obtaining a second vanishing point of the image to be detected by a feature point method includes: Obtain multiple light loss amounts in the image to be detected and the previous frame image by using an optical flow method, extend the translation directions of the multiple light loss amounts in the two frames of image to obtain at least four extension lines, and obtain multiple intersection points of at least four straight lines where the at least four extension lines are located. An intersection fusion point of the multiple intersection points is used as the second vanishing point, and the intersection fusion point is obtained by calculating a coordinate mean or a cluster center of the multiple intersection points.

8. The method according to any one of claims 1 to 7, wherein The acquiring of the image to be detected from the image sample sequence of the vehicle moving straight includes: Determine the sampling period of the image based on the sum of the computation time of detecting the lane line equation and the computation time of the vanishing point in a single image to be detected; Calculating a maximum Euclidean distance of all vanishing points in the vanishing point sequence to be calibrated according to a sampling period, and deleting the first element in the lane line sequence, the linear vanishing point sequence, and the feature vanishing point sequence when the maximum Euclidean distance is greater than one twentieth of the width of the image to be detected; A new image to be detected is obtained from the image sample sequence of the vehicle moving straight.

9. The method according to claim 1, wherein Based on the variance of the vanishing point coordinates, the linear vanishing point sequence and the characteristic vanishing point sequence are subjected to stability testing, and a vanishing point sequence with higher stability is selected as the vanishing point sequence to be calibrated, including: Calculating a first variance of the coordinates of all vanishing points in the linear vanishing point sequence and a second variance of the coordinates of all vanishing points in the characteristic vanishing point sequence, and clearing the linear vanishing point sequence and the characteristic vanishing point sequence if a difference between the first variance and the second variance is greater than a variance threshold, wherein the variance threshold is one tenth of the width of the image to be detected; If the difference between the first variance and the second variance is not greater than the variance threshold, the vanishing point sequence with the smaller variance is used as the vanishing point sequence to be calibrated.

10. The method according to claim 1, wherein The calibrating the extrinsic parameters of the vehicle-mounted front-view camera by combining the lane line sequence, the to-be-calibrated vanishing point sequence, and the intrinsic parameters of the vehicle-mounted front-view camera includes: Storing lane line equations for at least two pairs of lane lines corresponding to each image to be detected in the lane line sequence; For each image to be detected, the corresponding vanishing points in the vanishing point sequence to be calibrated are sequentially converted to the world coordinate system, and the roll angle roll value is optimized through a numerical iterative algorithm so that the at least two pairs of lane lines in the world coordinate system are rotated until they are equidistant and parallel. In the dedistorted image corresponding to each image to be detected, based on the lane line equations of the at least two pairs of lane lines, the optimized roll value is converted using the vanishing point principle to obtain the pitch angle value and yaw angle value of the vehicle-mounted front-view camera. The optimized roll value is adjusted by binary search until the width difference of the at least two pairs of lane lines is less than 10 -5 The current roll value, the current pitch value, and the current yaw value are used as external parameters of the vehicle-mounted front-view camera.

11. An electronic device, characterized in that: include: at least one memory and at least one processor, The at least one memory stores an executable code, and the at least one processor is configured to execute the executable code in the at least one memory to implement the method according to any one of claims 1 to 7 or 9.

12. A vehicle, characterized in that: include: A vehicle-mounted forward-looking camera for capturing image streams; as well as A processor, configured to execute the method according to any one of claims 1 to 7 or 9.

Images