Camera calibration method and device, equipment and storage medium

Abstract

The invention provides a camera calibration method and device, equipment and a storage medium, and the method comprises the steps: obtaining a preset number of lane line data pairs, and enabling any one of the lane line data pairs to be three-dimensional coordinate data of a reference lane line in two road images at a preset interval, the reference lane line comprises a left lane line and a right lane line of a vehicle where the camera is located; initial calibration parameters are obtained through calculation according to the mapping relation of the lane line data pairs; and generating a target calibration parameter based on the initial calibration parameter, wherein the target calibration parameter is used for calibrating the camera. According to the method, the calibration parameters of the camera are calculated according to the relationship between the lane line pairs spaced by the preset distance, so that the influence of the road scene on the calibration accuracy can be reduced, the calibration accuracy can be improved, and the applicability is wide.

Description

Technical Field

[0001] This application belongs to the technical field of autonomous driving, and particularly relates to a camera calibration method, device, equipment and storage medium. Background Art

[0002] On-vehicle cameras collect road images and provide real-time road conditions, lane line information, etc. to autonomous driving vehicles to assist the vehicles in making driving decisions. It can be seen that the accuracy of on-vehicle camera calibration has an important impact on the performance of autonomous driving.

[0003] A commonly used on-vehicle camera calibration scheme is to sense the vanishing point of the lane line. Then, the on-vehicle camera is calibrated by the relationship between the vanishing point back-projected onto the image coordinates and the optical center of the on-vehicle camera. In a scenario where the lane line is straight and the road is flat, the obtained vanishing point of the lane line is relatively accurate, and the obtained calibration parameters are also relatively accurate. However, in a scenario where the lane line is a turning line or the road has undulating slopes, the accuracy of the obtained vanishing point of the lane line is relatively poor, resulting in low accuracy of the obtained calibration parameters, thus affecting the calibration effect. It can be seen that the commonly used on-vehicle camera calibration method is greatly affected by the road scene and has poor applicability. Summary of the Invention

[0004] This application proposes a camera calibration method, device, equipment and storage medium, which can reduce the influence of the road scene on the calibration accuracy, is beneficial to improving the calibration accuracy, and has wide applicability.

[0005] The first aspect of the embodiments of this application proposes a camera calibration method, including:

[0006] Obtain a preset number of lane line data pairs. Any one lane line data pair is the three-dimensional coordinate data of the reference lane line in two road images separated by a preset distance. The reference lane line includes the left lane line and the right lane line of the vehicle where the camera is located;

[0007] Calculate initial calibration parameters according to the mapping relationship of each lane line data pair;

[0008] Generate target calibration parameters based on the initial calibration parameters, and the target calibration parameters are used to calibrate the camera.

[0009] In some embodiments of this application, the generating target calibration parameters based on the initial calibration parameters includes:

[0010] Determine the initial calibration parameters as the target calibration parameters;

[0011] Or,

[0012] Detect whether the total number of the obtained initial calibration parameters reaches a preset total number;

[0013] If the total number of the obtained initial calibration parameters reaches the preset total number, determine the average value of the preset total number of initial calibration parameters as the target calibration parameter;

[0014] If the total number of the obtained initial calibration parameters does not reach the preset total number, perform the operation of obtaining a preset number of lane line data pairs again.

[0015] In some embodiments of the present application, calculating the initial calibration parameter according to the mapping relationship of each lane line data pair includes:

[0016] For the k-th lane line data pair among the preset number of lane line data pairs, calculate the angle deviation corresponding to the k-th lane line data pair, where k is an integer greater than or equal to 1;

[0017] If the angle deviation meets the preset convergence condition, accumulate the angle deviation on the basis of the deviation accumulation value, and use the updated deviation accumulation value as the initial calibration parameter, where the deviation accumulation value is the deviation accumulation value updated based on the angle deviation corresponding to the (k - 1)-th lane line data pair;

[0018] If the angle deviation does not meet the preset convergence condition, iteratively adjust the external camera parameters after the (k - 1)-th adjustment using the angle deviation, and calculate the angle deviation corresponding to the (k + 1)-th lane line data pair according to the iteratively adjusted external camera parameters.

[0019] In some embodiments of the present application, calculating the angle deviation corresponding to the k-th lane line data pair includes:

[0020] Based on the external camera parameters after the (k - 1)-th adjustment, obtain the mapping relationship obtained by back-projecting the first lane line data in the k-th lane line data pair to the second lane line data, where the road image corresponding to the first lane line data is the road image after the road image corresponding to the second lane line data travels the preset distance;

[0021] Calculate the angle deviation between the first lane line data and the second lane line data according to the mapping relationship to obtain the angle deviation.

[0022] In some embodiments of the present application, the angle deviation includes a yaw angle deviation and a pitch angle deviation, and calculating the angle deviation between the first lane line data and the second lane line data according to the mapping relationship includes:

[0023] Calculate the lateral deviation and the longitudinal deviation between the first lane line data and the second lane line data;

[0024] The yaw angle deviation is calculated based on the lateral deviation, and the pitch angle deviation is calculated based on the longitudinal deviation and the lateral deviation.

[0025] In some embodiments of the present application, calculating the lateral deviation and the longitudinal deviation of the first lane line data and the second lane line data includes:

[0026] Calculating the lateral deviation and the longitudinal deviation of the first lane line data and the second lane line data includes:

[0027] Calculating the lateral deviation and the longitudinal deviation between the first left lane line data and the second left lane line data to obtain a first lateral deviation and a first longitudinal deviation; the first left lane line data is the data of the left lane line in the first lane line data, and the second left lane line data is the data of the left lane line in the second lane line data;

[0028] Calculating the lateral deviation and the longitudinal deviation between the first right lane line data and the second right lane line data to obtain a second lateral deviation and a second longitudinal deviation; the first right lane line data is the data of the right lane line in the first lane line data, and the second right lane line data is the data of the right lane line in the second lane line data;

[0029] Taking the sum of the first lateral deviation and the second lateral deviation as the lateral deviation of the first lane line data and the second lane line data, and taking the sum of the first longitudinal deviation and the second longitudinal deviation as the longitudinal deviation of the first lane line data and the second lane line data.

[0030] In some embodiments of the present application, calculating the lateral deviation and the longitudinal deviation between the first left lane line data and the second left lane line data to obtain a first lateral deviation and a first longitudinal deviation includes:

[0031] Calculating the average value of the lateral deviation between the first left lane line data and the second left lane line data corresponding to the first number of sampling points to obtain the first lateral deviation, and calculating the average value of the longitudinal deviation between the first left lane line data and the second left lane line data corresponding to the first number of sampling points to obtain the first longitudinal deviation;

[0032] Calculating the average value of the lateral deviation between the second left lane line data and the second left lane line data corresponding to the second number of sampling points to obtain the second lateral deviation, and calculating the average value of the longitudinal deviation between the first left lane line data and the second left lane line data corresponding to the second number of sampling points to obtain the second longitudinal deviation.

[0033] In some embodiments of the present application, calculating the yaw angle deviation based on the lateral deviation and calculating the pitch angle deviation based on the longitudinal deviation and the lateral deviation includes:

[0034] Calculating the average value of the lateral deviation within the preset distance, and the average value is the yaw angle deviation;

[0035] Calculating the pitch angle deviation according to the height of the camera from the ground, as well as the longitudinal deviation and the lateral deviation.

[0036] An embodiment of the second aspect of the present application provides a camera calibration device, including:

[0037] An acquisition module, configured to acquire a preset number of lane line data pairs, and any one lane line data pair is the three-dimensional coordinate data of a reference lane line in two road images spaced apart by a preset distance, and the reference lane line includes the left lane line and the right lane line of the vehicle where the camera is located;

[0038] A calculation module, configured to calculate initial calibration parameters according to the mapping relationship of each lane line data pair;

[0039] A generation module, configured to generate target calibration parameters based on the initial calibration parameters, and the target calibration parameters are used to calibrate the camera.

[0040] An embodiment of the third aspect of the present application provides an electronic device, including a memory, a processor, and a computer program stored on the memory and executable on the processor, and the processor runs the computer program to implement the method described in the first aspect above.

[0041] An embodiment of the fourth aspect of the present application provides a computer-readable storage medium, on which a computer program is stored, and the program is executed by a processor to implement the method described in the first aspect above.

[0042] The technical solutions provided in the embodiments of the present application at least have the following technical effects or advantages:

[0043] In an embodiment of the present application, a road image is obtained by a vehicle camera at preset intervals, and then the three-dimensional coordinate data of the reference lane lines in every two road images is used as a lane line data pair to calculate the calibration parameters of the camera. Among them, the reference lane lines include the left lane line and the right lane line of the vehicle where the camera is located. That is to say, the camera calibration method in the embodiment of the present application does not use the lane line vanishing point as the basic data for calibration, but uses the lane line data collected before and after the vehicle travels a preset distance as the basic data, so that it can be unaffected by the road scene. Further, initial calibration parameters are calculated according to the mapping relationship of each lane line data pair, and target calibration parameters are generated based on the initial calibration parameters, and the target calibration parameters are used to calibrate the camera. That is, by adopting the technical solution, the parameters of the camera are determined by the relative changes of the lane lines collected before and after the vehicle travels. Not only can relatively accurate basic data be obtained for any road scene, with wide applicability, but also calibration is performed through multiple lane line data pairs, which supports the correction of calibration parameters for lane lines that are not completely straight, and is beneficial to improving the accuracy of camera calibration.

[0044] Additional aspects and advantages of the present application will be given in part in the following description, become apparent in part from the following description, or be learned through the practice of the present application. BRIEF DESCRIPTION OF THE DRAWINGS

[0045] By reading the following detailed description of the preferred embodiments, various other advantages and benefits will become clear to those of ordinary skill in the art. The drawings are only for the purpose of showing the preferred embodiments and are not considered to be a limitation of the present application. Moreover, throughout the drawings, the same reference numerals are used to represent the same components.

[0046] In the drawings:

[0047] Figure 1 shows a flowchart of a camera calibration method provided by an embodiment of the present application;

[0048] Figure 2A shows a schematic diagram of the scene of lane lines in a 2D coordinate system provided by an embodiment of the present application;

[0049] Figure 2B shows Figure 2A the schematic diagram of the scene of the lane lines in a 3D coordinate system in ;

[0050] Figure 3A shows a schematic diagram of the scene where the first lane line data is projected onto the second lane line data provided by an embodiment of the present application;

[0051] Figure 3B shows a schematic diagram of the scene of lateral deviation calculation provided by an embodiment of the present application;

[0052] Figure 3C Shows a schematic diagram of the scenario for calculating the longitudinal deviation provided by an embodiment of the present application;

[0053] Figure 4 Shows a schematic structural diagram of a camera calibration device provided by an embodiment of the present application;

[0054] Figure 5 Shows a schematic structural diagram of an electronic device provided by an embodiment of the present application;

[0055] Figure 6 Shows a schematic diagram of a storage medium provided by an embodiment of the present application. Detailed implementation manners

[0056] Hereinafter, the exemplary embodiments of the present application will be described in more detail with reference to the accompanying drawings. Although the exemplary embodiments of the present application are shown in the drawings, it should be understood that the present application can be implemented in various forms and should not be limited by the embodiments set forth herein. On the contrary, these embodiments are provided so that the present application can be more thoroughly understood and the scope of the present application can be completely conveyed to those skilled in the art.

[0057] It should be noted that, unless otherwise specified, the technical terms or scientific terms used in the present application should have the ordinary meanings understood by those skilled in the art to which the present application belongs.

[0058] First, the technical scenario involved in the present application will be described.

[0059] The embodiments of the present application relate to the technology of calibrating the cameras of autonomous vehicles. With the development of technology, the cameras of autonomous vehicles are generally configured as bird eye view (BEV) cameras. As panoramic vision cameras, BEV cameras support data acquisition of a wide-angle field of view in the horizontal direction and pitch angle data acquisition in the vertical direction. Based on this, the parameter accuracy of the BEV camera in the direction perpendicular to the ground and the horizontal direction with respect to the ground has a great impact on the performance of the BEV camera. Calibrating the BEV camera can include yaw angle (Yaw) calibration and pitch angle (Pitch) calibration.

[0060] The camera calibration involved in the embodiments of the present application refers to estimating the external parameters (such as position, Yaw rotation angle, Pitch rotation angle, etc.) of the camera by collecting a series of feature points with known positions (such as lane lines) and poses, so as to calibrate the camera and obtain accurate road images.

[0061] Yaw refers to the rotation angle of the BEV camera in the horizontal direction.

[0062] Pitch refers to the rotational movement angle of the BEV camera in the vertical direction.

[0063] In a current BEV camera calibration scheme, by collecting BEV images of lane lines, the vanishing point of the lane lines in the BEV image is constructed. Furthermore, by back-projecting the BEV image from a 2D image into 3D coordinates, the camera is calibrated based on the relationship between the coordinates after back-projection of the vanishing point and the camera optical center. The vanishing point refers to a point where parallel lines converge in the camera vision as the field of view gradually recedes. In view of this, in the current BEV camera calibration scheme, for road scenarios where the lane lines are straight and the road is relatively flat, the calibration effect is relatively reasonable. For scenarios with turning lane lines or road undulations, due to inaccurate perception of the vanishing point, the calibration effect is relatively poor. In addition, the telephoto camera has a large perception error for long-distance lane lines, resulting in poor accuracy of the perceived vanishing point and relatively poor calibration effect.

[0064] To solve the above problems, in the technical solution provided by the embodiments of the present application, instead of using the lane line vanishing point as the basic data for calibration, the lane line data collected before and after the vehicle travels a preset distance is used as the basic data, and the parameters of the camera are determined by the relative changes of the lane lines collected before and after the vehicle travels. This method is not only not affected and restricted by the road scenario and has wide applicability, but also is conducive to improving the accuracy of camera calibration by calibrating multiple pairs of lane line data.

[0065] The execution subject of this technical solution can be any movable device that supports BEV camera calibration, including devices such as vehicles, ships, aircraft, or robots. Such devices can support the camera calibration of the embodiments of the present application through an autonomous driving system. The autonomous driving system can set execution modules for related functions such as lane line extraction, coordinate dimension conversion algorithms, and calibration algorithms.

[0066] The following describes a camera calibration method, device, and storage medium according to an embodiment of the present application with reference to the accompanying drawings. The technical solution of the present application will be described in detail below with specific embodiments. These specific embodiments can be combined with each other, and the same or similar concepts or processes may not be repeated in some embodiments. The embodiments of the present application will be described below with reference to the accompanying drawings.

[0067] See Figure 1 , Figure 1 which shows a flowchart of a camera calibration method provided by an embodiment of the present application. The method specifically includes the following steps:

[0068] Step S101, obtain a preset number of pairs of lane line data.

[0069] Among them, any lane line data pair is the 3D coordinate data of the reference lane line in two road images separated by a preset distance. The reference lane line includes the left lane line and the right lane line of the vehicle where the camera is located.

[0070] Exemplarily, the preset distance can be flexibly set based on the actual implementation scenario. For example, any value within 10 meters (m) to 15 m can be used as the preset distance.

[0071] In some embodiments, the vehicle can collect a road image every time it travels a preset distance in any road scenario. Here, the road image can be a 2D image in the BEV perspective. Then, the reference lane line in the 2D image and its coordinates in the image coordinate system (i.e., the 2D coordinate system) are identified. By back-projecting the coordinates of the reference lane line in the image coordinate system to the coordinates in the camera coordinate system (i.e., the 3D coordinate system), the lane line data of the corresponding reference lane line is obtained.

[0072] Exemplarily, after collecting the road image, a pre-trained lane line semantic segmentation model can be used to identify the left lane line and the right lane line of the vehicle in the road image, as well as the coordinate values of the sampling points of the left lane line and the right lane line. The left lane line and the right lane line of the vehicle are as Figure 2A shown. Further, the 3D coordinates corresponding to the 2D coordinates of the left lane line and the right lane line in Figure 2A can be calculated based on Inverse Perspective Mapping (IPM). The presentation of the lane line in the 3D coordinate system is as Figure 2B shown.

[0073] Calculating the 3D coordinates of the lane line using IPM can be implemented as follows: Assuming that the road surface is flat, the lane lines on the road surface are parallel to each other, and the Z-axis coordinate in the 3D coordinates of the lane line is 0, the 3D coordinates of the lane line can satisfy:

[0074]

[0075] Among them, refers to the homogeneous coordinate of the sampling point of the lane line transformed from the 2D coordinate system to the 3D coordinate system, and K -1 refers to the inverse of the camera intrinsic matrix, and [R|T] -1 refers to the inverse of the camera extrinsic matrix. refers to the homogeneous coordinate in the 2D coordinate system, and λ is a scalar, which refers to the distance from the camera optical center to the image plane.

[0076] Step S102, calculate the initial calibration parameters according to the mapping relationship of each lane line data pair.

[0077] Among them, the present technical solution can iteratively calculate the initial calibration parameters based on a preset number of lane line data pairs.

[0078] The processing procedures performed for each lane line data pair are similar. Taking any one of the preset number of lane line data pairs as an example, the process of calculating the initial calibration parameters in this embodiment will be described below. This any one lane line data is, for example, the k-th lane line data pair among the preset number of lane line data pairs, and k is an integer greater than or equal to 1.

[0079] Exemplarily, calculate the angle deviation corresponding to the k-th lane line data pair. If the angle deviation meets the preset convergence condition, accumulate the angle deviation on the basis of the deviation accumulation value, and use the updated deviation accumulation value as the initial calibration parameter. The deviation accumulation value is the deviation accumulation value updated based on the angle deviation corresponding to the (k - 1)-th lane line data pair. If the angle deviation does not meet the preset convergence condition, use the angle deviation to iteratively adjust the external camera parameters after the (k - 1)-th adjustment, and calculate the angle deviation corresponding to the (k + 1)-th lane line data pair according to the iteratively adjusted external camera parameters.

[0080] In some embodiments, the angle deviation meets the preset convergence condition, which can be realized as the value of the angle deviation being less than a preset threshold. In other embodiments, the angle deviation meets the preset convergence condition can be realized as k being a preset iteration number threshold.

[0081] The angle deviation corresponding to the k-th lane line data pair refers to the angle deviation between the first lane line data and the second lane line data in the k-th lane line data pair. Exemplarily, as Figure 3A shown, based on the external camera parameters after the (k - 1)-th adjustment, the mapping relationship obtained by back-projecting the first lane line data in the k-th lane line data pair to the second lane line data can be obtained. Figure 3A The dashed line in

[0082] is, for example, the projection line of the first lane line, and the solid line is, for example, the second lane line. Furthermore, according to the mapping relationship, calculate the angle deviation between the first lane line data and the second lane line data to obtain the angle deviation.

[0083] Further, the camera described in the embodiments of the present application is a BEV camera, and the BEV camera should be calibrated for Yaw and Pitch. Correspondingly, the angle deviation includes a yaw angle deviation (yaw d ) and a pitch angle deviation (pitch d ). Calculating the angle deviation between the first lane line data and the second lane line data may include: calculating the lateral deviation and the longitudinal deviation between the first lane line data and the second lane line data. Further, based on the lateral deviation, the yaw d is obtained, and based on the longitudinal deviation and the lateral deviation, the pitch d is obtained.

[0084] Combined with the foregoing description of the lane line data, each lane line data includes left lane line data and right lane line data. The left lane line data in the first lane line data is used as the first left lane line data, the right lane line data in the first lane line data is used as the first right lane line data, the left lane line data in the second lane line data is used as the second left lane line data, and the right lane line data in the second lane line data is used as the second right lane line data. Calculating the lateral deviation between the first lane line data and the second lane line data may include: calculating the lateral deviation between the first left lane line data and the second left lane line data to obtain a first lateral deviation, and calculating the lateral deviation between the first right lane line data and the second right lane line data to obtain a second lateral deviation. Then, the sum of the first lateral deviation and the second lateral deviation may be used as the lateral deviation between the first lane line data and the second lane line data. Correspondingly, calculating the longitudinal deviation between the first lane line data and the second lane line data may include: calculating the longitudinal deviation between the first left lane line data and the second left lane line data to obtain a first longitudinal deviation, and calculating the longitudinal deviation between the first right lane line data and the second right lane line data to obtain a second longitudinal deviation. Then, the sum of the first longitudinal deviation and the second longitudinal deviation may be used as the longitudinal deviation between the first lane line data and the second lane line data.

[0085] In some embodiments, the first lateral deviation may be the average value of the lateral deviations corresponding to the first number of sampling points between the first left lane line data and the second left lane line data; the first longitudinal deviation may be the average value of the longitudinal deviations corresponding to the first number of sampling points between the first left lane line data and the second left lane line data. The second lateral deviation may be the average value of the lateral deviations corresponding to the second number of sampling points between the first right lane line data and the second right lane line data; the second longitudinal deviation may be the average value of the longitudinal deviations corresponding to the second number of sampling points between the first right lane line data and the second right lane line data.

[0086] It should be understood that the above first number and second number may be the same or different. When the first number and the second number are the same, the interval distances between two adjacent sampling points on any one lane line may be equal, for example, it may be 5 m.

[0087] The following combines with the scenario diagram to illustrate the calculation processes of the lateral deviation and the longitudinal deviation of the present technical solution by taking the calculation of the lateral deviation as an example.

[0088] For example, referring to Figure 3B , P1 is the left lane line of the vehicle in the second lane line data, P2 is the right lane line of the vehicle in the second lane line data, Q1 is the projection of the left lane line of the vehicle in the first lane line data, and Q2 is the projection of the right lane line of the vehicle in the first lane line data. For any one side of the lane line, for example, sampling is performed at an interval of 5 m, and the lateral deviation of the projection of the first lane line data to the corresponding second lane line data corresponding to each sampling point is calculated. For example, the lateral deviation lb i of each sampling point can be obtained, where i is a positive integer. For any one side of the lane line, after obtaining the lateral deviations of n sampling points, the average value of the lateral deviations of the n sampling points is used as the lateral deviation corresponding to this side of the lane line, and n refers to the total number of sampling points. The lateral deviation of any one side of the lane line may satisfy: Correspondingly, the above first lateral deviation is The second lateral deviation is

[0089] It should be understood that the calculation processes of the first longitudinal deviation and the second longitudinal deviation are similar to the implementation manners of the Figure 3B illustrated embodiments. The difference is that for each sampling point on the second lane line, the longitudinal deviation of the projection of this sampling point to the first lane line is calculated. Details are not described herein.

[0090] Further, the lateral deviation between the first lane line data and the second lane line data can characterize the deviation of the yaw angle of the BEV camera, and the longitudinal deviation between the first lane line data and the second lane line data can characterize the deviation of the pitch angle of the BEV camera. After obtaining the lateral deviation and the longitudinal deviation between the first lane line data and the second lane line data, the yaw angle deviation can be calculated based on the lateral deviation and the pitch angle deviation can be calculated based on the longitudinal deviation and the lateral deviation

[0091] Exemplarily, the average value of the lateral deviation within the preset distance can be calculated, and this average value is the yaw angle deviation. For example, the yaw angle deviation of the k-th lane line data pair can satisfy: where L refers to the length value of the preset distance.

[0092] Since the data representation of the pitch angle is reflected by the deviation in the longitudinal direction, and for the sampling points of the lane lines, even if the deviation between the two lane lines can be calculated, it is not easy to determine the actual position where the sampling points should be projected. Therefore, according to the characteristic that the longitudinal (x) direction and the lateral (y) direction have scaling consistency, the deviation in the lateral (y) direction can be combined to calculate

[0093] Referring to Figure 3C , the height distance between the camera and the ground is, for example, h, and the actual position of a sampling point is, for example, position 1 in the figure, that is, the position of x. However, since the pitch angle to be calibrated by the camera is θ, the IPM projection position of this sampling point collected by the camera is position 2, that is, the position of x'. According to the similarity of triangles, it can be obtained that: Since the x direction and the y direction have scaling consistency, based on this, it can be determined that where y' is the lateral coordinate of the IPM projection position of this sampling point, and △x refers to the longitudinal deviation between position 1 and position 2.

[0094] Further, since the lateral deviation is easier to obtain than the longitudinal deviation, then, the of this sampling point can be obtained, where △(△y) refers to the lateral deviation of the corresponding sampling point projected onto the first lane line data and the second lane line data, and △x' refers to the longitudinal deviation of the corresponding sampling point projected onto the first lane line data and the second lane line data. y refers to the y value of the sampling point that is relatively close to the vehicle in the first lane line data and the second lane line data. The pitch of the k-th lane line data pair d can be the average value of θ of all sampling points included in the k-th lane line data pair over the preset distance.

[0095] The iterative process described in the embodiments of the present application may include, after obtaining the and , using and to update the yaw and pitch in the extrinsic camera parameters, and obtaining the latest yaw k+1 and pitch k+1 .

[0096]

[0097] Among them, yaw k and pitch k can be iteratively obtained according to the angle deviation of the (k - 1)-th lane line data pair. And, in the scenario of mapping the first lane line data of the k-th lane line data pair to the second lane line data, the extrinsic camera parameters are yaw k and pitch k as a reference. Based on this, it can be seen that the cumulative values of yaw d and pitch d obtained from a preset number of lane line data pairs are the calibration parameters corresponding to the preset number of lane line data pairs. Correspondingly, according to and update yaw d and pitch d :

[0098]

[0099]

[0100] It can be seen that, adopting this implementation method, camera calibration is performed based on the lane line data collected before and after the vehicle travels a preset distance as the basic data, so that it can be not affected by the road scene. In addition, based on a preset number of lane line data pairs, the extrinsic camera parameters are gradually adjusted based on the angle deviation corresponding to each lane line data pair. During the calibration process according to the new lane line data pair, the iteratively obtained yaw d and pitch d can be verified, thereby improving the accuracy of the calibration parameters.

[0101] Step S103, generating target calibration parameters based on the initial calibration parameters, where the target calibration parameters are used to calibrate the camera.

[0102] Among them, the target calibration parameters include target Yaw and target Pitch.

[0103] In some embodiments, the initial calibration parameters may be determined as the target calibration parameters. That is, the initial calibration parameters corresponding to a preset number of lane line data pairs may be used as the target calibration parameters.

[0104] Combined with the foregoing description of the IPM processing process, it can be seen that in the process of mapping the 2D coordinates of the lane lines to obtain 3D coordinates, it is premised that the road surface is flat and the lane lines on the road surface are parallel to each other. In actual implementation, the road scene is diverse, and it is difficult to obtain an absolutely flat road surface and absolutely parallel lane lines. Therefore, there are certain errors in the calibration parameters calculated using multiple lane line data pairs in one road scene.

[0105] In view of this, in some other embodiments, the total number of initial calibration parameters may be preset. After calculating the initial calibration parameters for the preset number of lane line data pairs in step S101, it is possible to detect whether the total number of the obtained initial calibration parameters reaches the preset total number. If the total number of the obtained initial calibration parameters reaches the preset total number, the average value of the preset total number of initial calibration parameters is determined as the target calibration parameter; if the total number of the obtained initial calibration parameters does not reach the preset total number, the operation of obtaining the preset number of lane line data pairs is performed again.

[0106] For example, the target calibration parameters are: Wherein, is the value of the target Yaw, is the value of the target Pitch, and N is the preset total number.

[0107] In this way, through multiple rounds of calibration, it is beneficial to correct the calibration parameters of non-fully straight lane lines, thereby improving the accuracy of camera calibration.

[0108] It can be seen that in the embodiments of the present application, a road image is obtained by the camera of the vehicle at intervals of a preset distance, and then the three-dimensional coordinate data of the reference lane lines in every two road images is used as a lane line data pair to calculate the calibration parameters of the camera. Among them, the reference lane lines include the left lane line and the right lane line of the vehicle where the camera is located. That is to say, the camera calibration method in the embodiments of the present application does not perform calibration based on the vanishing point of the lane line, but uses the lane line data collected before and after the vehicle travels a preset distance as the basic data, so that it can be unaffected by the road scene. Further, the initial calibration parameters are calculated according to the mapping relationship of each lane line data pair, and the target calibration parameters are generated based on the initial calibration parameters, and the target calibration parameters are used to calibrate the camera. That is, by adopting the technical solution, the parameters of the camera are determined by the relative changes of the lane lines collected before and after the vehicle travels. Not only can relatively accurate basic data be obtained for any road scene, with wide applicability, but also calibration is performed through multiple lane line data pairs, which supports the correction of the calibration parameters of non-perfectly straight lane lines, and is beneficial to improving the accuracy of camera calibration.

[0109] An embodiment of the present application also provides a camera calibration device, which is used to execute the camera calibration method provided in any of the above embodiments. As Figure 4 shown, the device includes: an acquisition module 41, a calculation module 42, and a generation module 43.

[0110] The acquisition module 41 is used to acquire a preset number of lane line data pairs. Any one lane line data pair is the three-dimensional coordinate data of the reference lane lines in two road images at intervals of a preset distance. The reference lane lines include the left lane line and the right lane line of the vehicle where the camera is located; the calculation module 42 is used to calculate the initial calibration parameters according to the mapping relationship of each lane line data pair; the generation module 43 is used to generate the target calibration parameters based on the initial calibration parameters, and the target calibration parameters are used to calibrate the camera.

[0111] Optionally, the generation module 43 is further used to determine the initial calibration parameters as the target calibration parameters;

[0112] Optionally, the generation module 43 is further used to detect whether the total number of the obtained initial calibration parameters reaches a preset total number;

[0113] If the total number of the obtained initial calibration parameters reaches the preset total number, the average value of the preset total number of initial calibration parameters is determined as the target calibration parameters;

[0114] If the total number of the obtained initial calibration parameters does not reach the preset total number, the operation of acquiring a preset number of lane line data pairs is performed again.

[0115] Optionally, the calculation module 42 is further configured to:

[0116] For the k-th lane line data pair among the preset number of lane line data pairs, calculate the angle deviation corresponding to the k-th lane line data pair, where k is an integer greater than or equal to 1;

[0117] If the angle deviation meets the preset convergence condition, accumulate the angle deviation on the basis of the deviation accumulation value, and use the updated deviation accumulation value as the initial calibration parameter, where the deviation accumulation value is the deviation accumulation value updated based on the angle deviation corresponding to the (k - 1)-th lane line data pair;

[0118] If the angle deviation does not meet the preset convergence condition, iteratively adjust the external camera parameters after the (k - 1)-th adjustment using the angle deviation, and calculate the angle deviation corresponding to the (k + 1)-th lane line data pair according to the iteratively adjusted external camera parameters.

[0119] Optionally, the calculation module 42 is further configured to: Based on the external camera parameters after the (k - 1)-th adjustment, obtain the mapping relationship obtained by back-projecting the first lane line data in the k-th lane line data pair to the second lane line data, where the road image corresponding to the first lane line data is the road image after the road image corresponding to the second lane line data has traveled the preset distance;

[0120] Calculate the angle deviation between the first lane line data and the second lane line data according to the mapping relationship to obtain the angle deviation.

[0121] Optionally, the angle deviation includes a yaw angle deviation and a pitch angle deviation, and the calculation module 42 is further configured to:

[0122] Calculate the lateral deviation and the longitudinal deviation between the first lane line data and the second lane line data;

[0123] Calculate the yaw angle deviation based on the lateral deviation, and calculate the pitch angle deviation based on the longitudinal deviation and the lateral deviation.

[0124] Optionally, the calculation module 42 is further configured to: The calculation of the lateral deviation and the longitudinal deviation between the first lane line data and the second lane line data includes:

[0125] Calculate the lateral deviation and the longitudinal deviation between the first left lane line data and the second left lane line data to obtain a first lateral deviation and a first longitudinal deviation; the first left lane line data is the data of the left lane line in the first lane line data, and the second left lane line data is the data of the left lane line in the second lane line data;

[0126] Calculate the lateral deviation and longitudinal deviation between the first right lane line data and the second right lane line data to obtain the second lateral deviation and the second longitudinal deviation; the first right lane line data is the data of the right lane line in the first lane line data, and the second right lane line data is the data of the right lane line in the second lane line data;

[0127] Take the sum of the first lateral deviation and the second lateral deviation as the lateral deviation between the first lane line data and the second lane line data, and take the sum of the first longitudinal deviation and the second longitudinal deviation as the longitudinal deviation between the first lane line data and the second lane line data.

[0128] Optionally, the calculation module 42 is further configured to:

[0129] Calculate the average value of the lateral deviations between the first left lane line data and the second left lane line data corresponding to the first number of sampling points to obtain the first lateral deviation, and calculate the average value of the longitudinal deviations between the first left lane line data and the second left lane line data corresponding to the first number of sampling points to obtain the first longitudinal deviation;

[0130] Calculate the average value of the lateral deviations between the second left lane line data and the second left lane line data corresponding to the second number of sampling points to obtain the second lateral deviation, and calculate the average value of the longitudinal deviations between the first left lane line data and the second left lane line data corresponding to the second number of sampling points to obtain the second longitudinal deviation.

[0131] Optionally, the calculation module 42 is further configured to:

[0132] Calculate the average value of the lateral deviation within the preset distance, and the average value is the yaw angle deviation;

[0133] Calculate the pitch angle deviation according to the height of the camera from the ground, and the longitudinal deviation and the lateral deviation.

[0134] The camera calibration device provided by the embodiments of the present application and the camera calibration method provided by the embodiments of the present application are based on the same inventive concept and have the same beneficial effects as the methods adopted, run or implemented by them.

[0135] The embodiments of the present application also provide an electronic device to execute the above camera calibration method. Please refer to Figure 5 It shows a schematic diagram of an electronic device provided by some embodiments of the present application. As Figure 5As shown, the electronic device 5 includes: a processor 500, a memory 501, a bus 502, and a communication interface 503. The processor 500, the communication interface 503, and the memory 501 are connected through the bus 502. A computer program that can run on the processor 500 is stored in the memory 501. When the processor 500 runs the computer program, it executes the camera calibration method provided in any of the foregoing embodiments of the present application.

[0136] Among them, the memory 501 may include a high-speed random access memory (RAM: Random Access Memory), and may also include a non-volatile memory, such as at least one disk memory. Through at least one communication interface 503 (which can be wired or wireless), a communication connection is established between this device network element and at least one other network element, and the Internet, wide area network, local area network, metropolitan area network, etc. can be used.

[0137] The bus 502 can be an ISA bus, a PCI bus, an EISA bus, etc. The bus can be divided into an address bus, a data bus, a control bus, etc. Among them, the memory 501 is used to store programs. After receiving an execution instruction, the processor 500 executes the program. The camera calibration method disclosed in any of the foregoing embodiments of the present application can be applied to the processor 500 or implemented by the processor 500.

[0138] The processor 500 may be an integrated circuit chip with the ability to process signals. In the implementation process, each step of the above method can be completed by the integrated logic circuit in the hardware of the processor 500 or the instructions in the form of software. The above-mentioned processor 500 may be a general-purpose processor, including a central processing unit (CPU for short), a network processor (NP for short), etc.; it may also be a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components. It can implement or execute the various methods, steps and logic block diagrams disclosed in the embodiments of the present application. The general-purpose processor may be a microprocessor or the processor may also be any conventional processor, etc. The steps of the method disclosed in combination with the embodiments of the present application can be directly embodied as being executed and completed by the hardware decoding processor, or executed and completed by a combination of the hardware and software modules in the decoding processor. The software module may be located in a mature storage medium in the art such as a random access memory, a flash memory, a read-only memory, a programmable read-only memory or an electrically erasable programmable memory, a register, etc. This storage medium is located in the memory 501, and the processor 500 reads the information in the memory 501 and combines its hardware to complete the steps of the above method.

[0139] The electronic device provided in the embodiment of the present application and the camera calibration method provided in the embodiment of the present application are based on the same inventive concept and have the same beneficial effects as the method adopted, run or implemented by it.

[0140] The embodiment of the present application also provides a computer-readable storage medium corresponding to the camera calibration method provided in the foregoing embodiment. Please refer to Figure 6 , which shows that the computer-readable storage medium is an optical disc 30, on which a computer program (i.e., a program product) is stored. When the computer program is run by a processor, it will execute the camera calibration method provided in any of the foregoing embodiments.

[0141] It should be noted that examples of the computer-readable storage medium may also include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other optical and magnetic storage media, which will not be elaborated here one by one.

[0142] The computer-readable storage medium provided in the above embodiment of the present application and the camera calibration method provided in the embodiment of the present application are based on the same inventive concept and have the same beneficial effects as the method adopted, run or implemented by the application program stored in it.

[0143] It should be noted that:

[0144] In the specification provided herein, a large number of specific details are set forth. However, it will be understood that embodiments of the present application may be practiced without these specific details. In some instances, well-known structures and techniques have not been shown in detail so as not to obscure the understanding of this specification.

[0145] Similarly, it should be understood that in order to streamline the present application and assist in understanding one or more of the various inventive aspects, in the above description of the exemplary embodiments of the present application, the various features of the present application are sometimes grouped together into a single embodiment, figure, or description thereof. However, the disclosed method should not be construed as reflecting an intention that the claimed subject matter of the present application requires more features than are expressly recited in each claim. Rather, as reflected in the following claims, the inventive aspects lie in less than all of the features of the single foregoing disclosed embodiment. Thus, the claims following the detailed description are hereby expressly incorporated into this detailed description, with each claim standing on its own as a separate embodiment of the present application.

[0146] In addition, those skilled in the art will appreciate that although some of the embodiments described herein include certain features included in other embodiments but not others, the combination of features of different embodiments is meant to be within the scope of the present application and forms different embodiments. For example, in the following claims, any one of the claimed embodiments can be used in any combination.

[0147] The above is only a preferred specific embodiment of the present application, but the protection scope of the present application is not limited thereto. Any changes or substitutions that can be easily conceived by those skilled in the art within the technical scope disclosed in the present application should be covered by the protection scope of the present application. Therefore, the protection scope of the present application shall be subject to the protection scope of the claims.

Claims

1. A camera calibration method, characterized in that, Including: Obtain a preset number of lane line data pairs. Any one lane line data pair is the three-dimensional coordinate data of the reference lane line in two road images with a preset distance interval. The reference lane line includes the left lane line and the right lane line of the vehicle where the camera is located; Calculate initial calibration parameters according to the mapping relationship of each lane line data pair; Generate target calibration parameters based on the initial calibration parameters. The target calibration parameters are used to calibrate the camera.

2. The method according to claim 1, characterized in that, The generating the target calibration parameters based on the initial calibration parameters includes: Determine the initial calibration parameters as the target calibration parameters; Or, Detect whether the total number of the obtained initial calibration parameters reaches a preset total number; If the total number of the obtained initial calibration parameters reaches the preset total number, determine the average value of the preset total number of initial calibration parameters as the target calibration parameters; If the total number of the obtained initial calibration parameters does not reach the preset total number, perform the operation of obtaining a preset number of lane line data pairs again.

3. The method according to claim 1, characterized in that, The calculating the initial calibration parameters according to the mapping relationship of each lane line data pair includes: For the k-th lane line data pair among the preset number of lane line data pairs, calculate the angle deviation corresponding to the k-th lane line data pair, where k is an integer greater than or equal to 1; If the angle deviation meets the preset convergence condition, accumulate the angle deviation on the basis of the deviation accumulation value, and use the updated deviation accumulation value as the initial calibration parameter. The deviation accumulation value is the deviation accumulation value updated based on the angle deviation corresponding to the (k - 1)-th lane line data pair; If the angle deviation does not meet the preset convergence condition, iteratively adjust the external camera parameters after the (k - 1)-th adjustment using the angle deviation, and calculate the angle deviation corresponding to the (k + 1)-th lane line data pair according to the iteratively adjusted external camera parameters.

4. The method according to claim 3, characterized in that, The calculating the angle deviation corresponding to the k-th lane line data pair includes: Based on the external camera parameters after the (k - 1)-th adjustment, obtain the mapping relationship obtained by back-projecting the first lane line data in the k-th lane line data pair to the second lane line data. The road image corresponding to the first lane line data is the road image after the road image corresponding to the second lane line data travels the preset distance; Calculate the angle deviation between the first lane line data and the second lane line data according to the mapping relationship to obtain the angle deviation.

5. The method according to claim 4, characterized in that, The angle deviation includes a yaw angle deviation and a pitch angle deviation. The calculating the angle deviation between the first lane line data and the second lane line data according to the mapping relationship includes: Calculate the lateral deviation and the longitudinal deviation between the first lane line data and the second lane line data; Calculate the yaw angle deviation based on the lateral deviation, and calculate the pitch angle deviation based on the longitudinal deviation and the lateral deviation.

6. The method according to claim 5, characterized in that, The calculating the lateral deviation and the longitudinal deviation between the first lane line data and the second lane line data includes: The calculating the lateral deviation and the longitudinal deviation between the first lane line data and the second lane line data includes: Calculate the lateral deviation and longitudinal deviation between the first left lane line data and the second left lane line data to obtain a first lateral deviation and a first longitudinal deviation; the first left lane line data is the data of the left lane line in the first lane line data, and the second left lane line data is the data of the left lane line in the second lane line data; Calculate the lateral deviation and longitudinal deviation between the first right lane line data and the second right lane line data to obtain a second lateral deviation and a second longitudinal deviation; the first right lane line data is the data of the right lane line in the first lane line data, and the second right lane line data is the data of the right lane line in the second lane line data; Take the sum of the first lateral deviation and the second lateral deviation as the lateral deviation between the first lane line data and the second lane line data, and take the sum of the first longitudinal deviation and the second longitudinal deviation as the longitudinal deviation between the first lane line data and the second lane line data.

7. The method according to claim 5, characterized in that, The calculating the yaw angle deviation based on the lateral deviation and calculating the pitch angle deviation based on the longitudinal deviation and the lateral deviation includes: Calculate the average value of the lateral deviation within the preset distance, and the average value is the yaw angle deviation; Calculate the pitch angle deviation according to the height of the camera from the ground, and the longitudinal deviation and the lateral deviation.

8. A camera calibration device, characterized in that, Includes: An acquisition module, configured to acquire a preset number of lane line data pairs, and any one lane line data pair is the three-dimensional coordinate data of the reference lane line in two road images spaced apart by a preset distance, and the reference lane line includes the left lane line and the right lane line of the vehicle where the camera is located; A calculation module, configured to calculate initial calibration parameters according to the mapping relationship of each lane line data pair; A generation module, configured to generate target calibration parameters based on the initial calibration parameters, and the target calibration parameters are used to calibrate the camera.

9. An electronic device, comprising a memory, a processor, and a computer program stored on the memory and executable on the processor, wherein, The processor runs the computer program to implement the method according to any one of claims 1-7.

10. A computer-readable storage medium, having a computer program stored thereon, wherein, The program is executed by the processor to implement the method according to any one of claims 1-7.

Images