Lane line equation estimation method based on vehicle camera pitch angle and world coordinate system

Abstract

The present invention discloses a method for estimating the pitch angle of a vehicle-mounted camera and the lane line equation in a world coordinate system, comprising the following steps: step 1, obtaining an image taken by a vehicle-mounted camera; step 2, extracting the pixel points of each lane line using a neural network; step 3, pixel point distortion correction; step 4, straight / curved road judgment; step 5, lane line parallel judgment and camera pitch angle estimation; step 6, calculating the lane line equation in a world coordinate system. The present invention solves the problem of inaccurate lane line detection in a world coordinate system caused by changes in the pitch angle of the camera; at the same time, the present invention makes full use of the information of multiple lane line pixels in the picture, avoids complex transmission transformation, and can obtain accurate and robust estimation of the pitch angle of the vehicle-mounted camera.

Description

Technical Field

[0001] The invention relates to a method for estimating a lane line equation in a vehicle-mounted camera pitch angle and a world coordinate system. Background Art

[0002] The automatic driving system usually needs to detect the lane line information of the current position through the on-board camera placed in front of the vehicle. The lane line information is generally represented by a polynomial in the world coordinate system. With the application of deep neural network technology, the lane line pixels in the on-board camera image can usually be accurately detected. In order to obtain the lane line equation in the world coordinate system, the typical approach is to first calculate the physical coordinate points in the world coordinate system corresponding to the lane line pixels through inverse transmission transformation, and then perform curve fitting on these points. The accuracy of the above inverse transmission transformation is very dependent on the accuracy of the camera's attitude parameters relative to the world coordinate system; these attitude parameters can refer to the calibration parameters of the vehicle before leaving the factory. However, when the vehicle is driving normally on the road, the ups and downs of the road surface and the changes in the vehicle's own throttle, brake, load and other states are likely to cause rapid changes in the camera's pitch angle. If the vehicle body pitch angle is not compensated in real time and accurately, the lane line coordinates in the world coordinate system obtained by the inverse transmission transformation will have obvious deviations, and ultimately lead to lane line information detection errors, affecting subsequent vehicle control functions. By jointly estimating the pitch angle of the vehicle camera and the lane line equation in the world coordinate system, accurate lane line detection can be provided. At the same time, the pitch angle information obtained can also be used for other functions of the vehicle camera, such as obstacle detection.

[0003] The methods currently used by those skilled in the art and their disadvantages are as follows: 1. Using other attitude measurement or estimation methods, such as high-precision gyroscopes, this method is expensive and not convenient for actual mass production deployment; 2. Using an attitude estimation algorithm based on multi-frame image information, and then using the estimated pitch angle to calculate the equation of the lane line in the world coordinate system, this method requires a large amount of calculation and is not convenient for actual mass production deployment; 3. Using lane line points in a single frame image to estimate the pitch angle, the lane line pixel points can be searched for the pitch angle, and the lane line pixel points are subjected to multiple inverse projection transformation calculations, and then the constraint of the parallelism of the lane lines in the world coordinate system is used to find the optimal pitch angle that makes the lane lines parallel. When there are many lane line pixel points, in order to obtain higher accuracy, the camera pitch angle needs to be searched at a small interval. At this time, the amount of calculation of repeated multi-point inverse projection transformation is very large; if only a small number of pixel points are selected for similar processing, since the positions of the detected pixel points themselves may also have errors, the information of all pixel points cannot be fully utilized, and the accuracy and robustness of the pitch angle estimation may be affected. Summary of the invention

[0004] The purpose of the present invention is to provide a method for fast joint estimation of the pitch angle of a vehicle-mounted camera and the lane line equation in a world coordinate system that is easy to deploy, so as to solve the problem of inaccurate lane line detection in a world coordinate system caused by changes in the pitch angle of the camera.

[0005] The technical solution to achieve the purpose of the present invention is: a method for estimating the pitch angle of a vehicle-mounted camera and the lane line equation in a world coordinate system, comprising the following steps:

[0006] Step 1: Obtain an image captured by a vehicle-mounted camera;

[0007] Step 2: Use a neural network to extract the pixel points of each lane line;

[0008] Step 3: pixel distortion correction;

[0009] Step 4: straight road / curve judgment;

[0010] Step 5: Lane line parallelism judgment and camera pitch angle estimation;

[0011] Step 6: Calculate the lane line equation in the world coordinate system.

[0012] Furthermore, the step 2 is specifically as follows: a deep neural network is used to extract semantic information of lane lines and identify the coordinates of pixel points belonging to each lane line in the image.

[0013] Furthermore, the step three is specifically: using the distortion coefficient of the lens itself to correct the coordinates of the pixel points in the picture to remove the influence of the lens distortion.

[0014] Furthermore, the step 4 uses a curve fitting-based method to determine whether the lane line is a straight road or a curve.

[0015] Furthermore, the method based on curve fitting is specifically:

[0016] First, for the image pixel (u k ,v k ), using P-order polynomial fitting, the fitting curve equation is obtained:

[0017]

[0018] Then, several pixel points at the top and bottom of the lane line are examined, and the derivatives of the curve at these pixel points are calculated using the fitted high-order polynomial. If the derivative variance of these points is less than the preset threshold th0, the lane line is considered to be a straight line; otherwise, the lane line is a curve.

[0019] Furthermore, when the lane is two lanes, the step five specifically includes:

[0020] Step 5.1, assume that the lane line l in the real world 1 and l 2 Satisfies the third-order curve equation and the two are parallel, that is, assume that the lane line point (x, y) in the world coordinate system satisfies the following equation:

[0021]

[0022]

[0023] After the transmission change of the camera imaging model, it can be proved that when the camera distortion is not considered, the pixel position in the picture can be expressed by a second-order hyperbolic equation, that is,

[0024]

[0025]

[0026] Among them, v′ 1 =v 1 -c y , v′ 2 =v 2 -c y , u′ 1 =u 1 -c x , u′ 1 =u 1 -c x , c x and c y They are the imaging positions of the objects at infinity in front of the camera in the picture;

[0027] Step 5.2, take the inverse of both sides of the above two equations and calculate p 11 and p 12 , and re-substituting it into the above formula, we can obtain the following relationship:

[0028]

[0029]

[0030]

[0031] Among them, the first-order inverse and It can be directly calculated using the above P-order polynomial fitting;

[0032] Step 5.3, for a given parameter c y , calculate the optimal parameter q=(p 0 ,p 2 ,p3 ), so that the joint fitting error is minimized, and the fitting error is recorded. The specific calculation formula is:

[0033]

[0034] Among them, N 1 and N 2 are the number of pixels belonging to lane lines and respectively, i 1 and i 2 is the pixel index; c y It is a parameter that changes dynamically with the pitch angle of the vehicle during driving;

[0035] Among them, N 1 and N 2 are the number of pixels belonging to lane lines and respectively, i 1 and i 2 is the pixel index, Err(q,c y ) is the fitting error.

[0036] Due to c y is a parameter that changes dynamically with the pitch angle of the vehicle during driving. y When the actual value of the vehicle pitch angle corresponding to the current image is corresponding, the error is minimized. As the optimal estimate, it can be expressed as follows:

[0037]

[0038] Step 5.4: Analyze the minimum fitting error Err obtained using the above method min , if Err min If it is less than the preset threshold th1, the assumption that the lane lines are parallel is considered valid. and is valid, and the camera pitch angle is calculated by the following formula:

[0039]

[0040] Among them, c y0 is the pixel ordinate corresponding to the optical center of the camera, and F is the focal length of the camera.

[0041] Furthermore, in step 5.3, c y The estimation method is: within the preset angle range, according to the preset step, search for the fitting error q opt (c y ) to achieve the minimum c y , as the current moment c y The estimated value is calculated as:

[0042]

[0043] Further, the step six is ​​specifically: according to the step five obtained The lane line pixel coordinates are projected into the world coordinate system using the inverse transmission transformation, and the projected lane line points are fitted to obtain the lane line equation in the world coordinate system.

[0044] By adopting the above technical scheme, the present invention has the following beneficial effects: (1) The present invention uses a neural network to obtain pixels belonging to each lane line, and then judges the parallel relationship of different lane lines in the world coordinate system based on the assumption that the ground lane lines satisfy a straight line (straight road) or a second-order / third-order polynomial (curve); when there are parallel lane lines in the road scene, the vehicle body pitch angle can be output simultaneously, and the lane line equation in the world coordinate system can be further obtained, thereby solving the problem of inaccurate lane line detection in the world coordinate system caused by the change of the camera pitch angle; at the same time, the present invention makes full use of the information of multiple lane line pixels in the picture, avoids complex transmission transformation, and can obtain accurate and robust estimation of the pitch angle of the vehicle-mounted camera.

[0045] (2) The method for determining whether lane lines are parallel and estimating pitch angles in a two-lane situation provided by the present invention can be easily extended to other multi-lane situations. BRIEF DESCRIPTION OF THE DRAWINGS

[0046] In order to make the content of the present invention more clearly understood, the present invention is further described in detail below according to specific embodiments and in conjunction with the accompanying drawings, wherein

[0047] Figure 1 It is a flow chart of the present invention. DETAILED DESCRIPTION

[0048] In order to better understand the above technical solution, the above technical solution will be described in detail below in conjunction with the accompanying drawings and specific implementation methods.

[0049] (Example 1)

[0050] See Figure 1 The method for estimating the lane line equation in the camera pitch angle and world coordinate system of this embodiment includes the following steps:

[0051] Step 1: Get the image captured by the vehicle camera.

[0052] Step 2: Use a neural network to extract the pixels of each lane line. Specifically, a deep neural network is used to extract the semantic information of the lane line and identify the pixel coordinates of each lane line in the image. This step can be implemented by some typical lane line recognition networks, such as LaneNet, SCNN, LaneATT, etc.

[0053] Step three, pixel distortion correction, specifically: use the lens' own distortion coefficient to correct the pixel coordinates in the picture to remove the influence of lens distortion.

[0054] Step 4, straight road / curve judgment, specifically, using a curve fitting method to judge whether the lane line is a straight road or a curve. The curve fitting method is specifically:

[0055] First, for the image pixel (u k ,v k ), perform high-order polynomial fitting, such as using P-order polynomial fitting (usually P ≥ 6), and obtain the fitting curve equation:

[0056]

[0057] Then, several pixel points at the top and bottom of the lane line are examined, and the derivatives of the curve at these pixel points are calculated using the fitted high-order polynomial. If the derivative variance of these points is less than the preset threshold th0 (for example, 0.001), the lane line is considered to be a straight line; otherwise, the lane line is a curve.

[0058] Step 5: Lane line parallelism judgment and camera pitch angle estimation, when the lane is two lanes, specifically includes:

[0059] Step 5.1, assume that the lane line l in the real world 1 and l 2 Satisfies the third-order curve equation and the two are parallel, that is, assume that the lane line point (x, y) in the world coordinate system satisfies the following equation:

[0060]

[0061]

[0062] After the transmission change of the camera imaging model, it can be proved that when the camera distortion is not considered, the pixel position in the picture can be expressed by a second-order hyperbolic equation, that is,

[0063]

[0064]

[0065] Among them, v′ 1 =v 1 -c y , v′ 2 =v 2 -c y , u′ 1 =u1 -c x , u′ 1 =u 1 -c x , c x and c y They are the imaging positions of the objects at infinity in front of the camera in the picture;

[0066] Step 5.2, take the inverse of both sides of the above two equations and calculate p 11 and p 12 , and re-substituting it into the above formula, we can obtain the following relationship:

[0067]

[0068]

[0069]

[0070] Among them, the first-order inverse and It can be directly calculated using the above P-order polynomial fitting;

[0071] Step 5.3, for a given parameter c y , calculate the optimal parameter q=(p 0 ,p 2 ,p 3 ), so that the joint fitting error is minimized, and the fitting error is recorded. The specific calculation formula is:

[0072]

[0073] Among them, N 1 and N 2 are the number of pixels belonging to lane lines and respectively, i 1 and i 2 is the pixel index, Err(q,c y ) is the fitting error.

[0074] Due to c y is a parameter that changes dynamically with the pitch angle of the vehicle during driving. y When the actual value of the vehicle pitch angle corresponding to the current image is corresponding, the error is minimized. As the optimal estimate, it can be expressed as follows:

[0075]

[0076] The above c yThe estimation method is: within the preset angle range (the angle range can be -3 to 3 degrees), according to the preset step (such as 0.1 degree), search for the fitting error q opt (c y ) to achieve the minimum c y , as the current moment c y The estimated value is calculated as:

[0077]

[0078] Step 5.4: Analyze the minimum fitting error Err obtained using the above method min , if Err min If the value is less than the preset threshold th1 (e.g. 20, 50), the assumption that the lane lines are parallel is considered valid. and is valid, and the camera pitch angle is calculated by the following formula:

[0079]

[0080] Among them, c y0 is the pixel ordinate corresponding to the optical center of the camera, and F is the focal length of the camera.

[0081] Step 6: Calculate the lane line equation in the world coordinate system. Specifically, based on the equation obtained in step 5 The lane line pixel coordinates are projected into the world coordinate system using the inverse transmission transformation, and the projected lane line points are fitted to obtain the lane line equation in the world coordinate system.

[0082] When the number of lanes is greater than two, only the left lane and / or right lane closest to the vehicle needs to be selected for pitch angle estimation, and then the estimated pitch angle is used to calculate the equations of all lane lines. If there is only one lane, this method fails.

[0083] The specific embodiments described above further illustrate the objectives, technical solutions and beneficial effects of the present invention in detail. It should be understood that the above description is only a specific embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present invention should be included in the protection scope of the present invention.

Claims

1. Estimation method of lane line equation in vehicle-mounted camera pitch angle and world coordinate system, It is characterized in that The following steps are involved: Step 1: Obtain an image captured by a vehicle-mounted camera; Step 2: Use a neural network to extract the pixel points of each lane line; Step 3: pixel distortion correction; Step 4: straight road / curve judgment: using a curve fitting method to judge whether the lane line is a straight road or a curve; the curve fitting method is specifically as follows: First, for the image pixel (u k ,v k ), using P-order polynomial fitting, we get the fitting curve equation: Then, several pixel points at the top and bottom of the lane line are examined, and the derivatives of the curve at these pixel points are calculated using the fitted high-order polynomial. If the variance of the derivatives of these points is less than the preset threshold th0, the lane line is considered to be a straight line; otherwise, the lane line is a curve. Step 5: Lane line parallelism judgment and camera pitch angle estimation; When the lane is two lanes, this step five specifically includes: Step 5.1, assume that the lane line l in the real world 1 and l 2 Satisfies the third-order curve equation and the two are parallel, that is, assume that the lane line point (x, y) in the world coordinate system satisfies the following equation: After the transmission change of the camera imaging model, it can be proved that when the camera distortion is not considered, the pixel position in the picture can be expressed by a second-order hyperbolic equation, that is, Among them, v′ 1 =v 1 -c y , v′ 2 =v 2 -c y , u′ 1 =u 1 -c x , u′ 1 =u 1 -c x , c x and c y They are the imaging positions of the objects at infinity in front of the camera in the picture; Step 5.2, take the derivatives of both sides of the above two equations and calculate p 11 and p 12 , and re-substituting it into the above formula, we can obtain the following relationship: Among them, the first-order derivative and It can be directly calculated using the above P-order polynomial fitting; Step 5.3, for a given parameter c y , calculate the optimal parameter q=(p 0 , p 2 , p 3 ), so that the joint fitting error is minimized, and the fitting error is recorded. The specific calculation formula is: Among them, N 1 and N 2 are the number of pixels belonging to lane lines and respectively, i 1 and i 2 is the pixel index, Err(q,c y ) is the fitting error; Due to c y is a parameter that changes dynamically with the pitch angle of the vehicle during driving. y When the actual value of the vehicle pitch angle corresponding to the current image is corresponding, the error is minimized. As the optimal estimate, it can be expressed as follows: Step 5.4: Analyze the minimum fitting error Err obtained using the above method min , if Err min If it is less than the preset threshold th1, the assumption that the lane lines are parallel is considered valid. and is valid, and the camera pitch angle is calculated by the following formula: Among them, c y0 is the pixel ordinate corresponding to the optical center of the camera, and F is the focal length of the camera Step 6: Calculate the lane line equation in the world coordinate system.

2. The method for estimating the lane line equation in the vehicle-mounted camera pitch angle and world coordinate system according to claim 1, It is characterized in that The step 2 is specifically as follows: a deep neural network is used to extract the semantic information of the lane lines and identify the coordinates of the pixel points belonging to each lane line in the image.

3. The method for estimating the lane line equation in the vehicle-mounted camera pitch angle and world coordinate system according to claim 1, It is characterized in that The step three is specifically: using the distortion coefficient of the optical lens itself to correct the coordinates of the pixel points in the picture to remove the influence of lens distortion.

4. The method for estimating the lane line equation in the vehicle-mounted camera pitch angle and world coordinate system according to claim 1, It is characterized in that In step 5.3, y The estimation method is: within the preset angle range, according to the preset step, search for the fitting error q opt (c y ) to achieve the minimum c y , as the current moment c y The estimated value is calculated as:

5. The method for estimating the lane line equation in the vehicle-mounted camera pitch angle and world coordinate system according to claim 1, It is characterized in that The step six is ​​specifically: according to the step five obtained The lane line pixel coordinates are projected into the world coordinate system using the inverse transmission transformation, and the projected lane line points are fitted to obtain the lane line equation in the world coordinate system.

Images