A lane line detection inverse perspective transformation method and system

Abstract

The application provides a lane line detection inverse perspective transformation method and system, and belongs to the field of automatic driving. The method comprises the following steps: acquiring laser point cloud information around a vehicle and front view image information of the vehicle; acquiring two-dimensional coordinate information of a lane line from the front view image information, and performing road semantic segmentation on the laser point cloud information to obtain a terrestrial laser point cloud; performing fitting calculation on the terrestrial laser point cloud to obtain a terrestrial curved surface equation; projecting the two-dimensional coordinate information into the terrestrial curved surface equation to perform inverse perspective transformation to obtain map information; and performing road trajectory planning according to the map information. The application solves the problem that the lane line in a two-dimensional image is projected into a three-dimensional Dicke coordinate system under the assumption that the earth is absolutely flat.

Description

Technical Field

[0001] This invention relates to the field of autonomous driving technology, and in particular to an inverse perspective transformation method and system for lane line detection. Background Technology

[0002] In existing technologies, most lane detection algorithms used in the field of autonomous driving are based on image-based lane recognition (including traditional lane feature extraction methods and deep learning end-to-end lane detection algorithm models that have emerged in recent years). Then, based on the lane detection results in the image, the lane point sequence in the two-dimensional image space is projected onto the three-dimensional Cartesian coordinate system, thereby mapping the lane lines into the vehicle's driving space. This allows it to interact with the trajectory planning module of the autonomous vehicle to realize functions such as Adaptive Cruise Control (ACC) and Lane Keep Assistance (LKA).

[0003] Existing lane line point sequence inverse perspective mapping (IPM) technology requires the use of camera intrinsic and extrinsic parameter models, and also relies on an extremely strict assumption: the ground plane assumption. This means that a point in a 2D image projected into a 3D vehicle coordinate system is a ray, assuming the ground is a plane, and then the intersection of the ray and the ground plane is used to determine the specific location of that point in the lane line in 3D space. This assumption cannot be guaranteed in most scenarios. Therefore, the result of lane line inverse perspective mapping is often acceptable in the near distance to the vehicle, but becomes extremely unstable and unreliable when the ground is undulating in the distance.

[0004] Therefore, a new inverse perspective transformation method is needed to address the assumption that the ground is an absolute plane when the lane lines in the two-dimensional image are projected into the three-dimensional Cartesian coordinate system in the existing technology. This would enable a reliable three-dimensional lane line description equation to be obtained regardless of whether the ground is flat (uphill or downhill). Summary of the Invention

[0005] One objective of this invention is to provide an inverse perspective transformation method for lane line detection that does not require the assumption that the ground is absolutely flat. For non-planar ground, such as uphill, downhill, and sloping terrain, the inverse perspective transformation model of the lane line detection algorithm is still applicable.

[0006] A further objective of this invention is to enable lane detection to maintain high performance even at a distance.

[0007] Specifically, the present invention provides an inverse perspective transformation method for lane line detection, comprising the following steps:

[0008] Acquire laser point cloud information around the vehicle and front view image information of the vehicle;

[0009] Two-dimensional coordinate information of lane lines is obtained from the front view image information, and road semantic segmentation is performed on the laser point cloud information to obtain the ground laser point cloud;

[0010] The geodetic laser point cloud is fitted and calculated to obtain the geodetic surface equation;

[0011] The two-dimensional coordinate information is projected into the geodetic surface equation and inverse perspective transformation is performed to obtain map information;

[0012] Road trajectory planning is performed based on the map information.

[0013] Furthermore, the step of fitting the geodetic laser point cloud to obtain the geodetic surface equation includes:

[0014] Establish a vehicle coordinate system with the rear axle center of the vehicle as the origin, the front direction of the vehicle as the positive x-axis, the left side direction of the vehicle as the positive y-axis, and the height direction of the vehicle as the positive z-axis.

[0015] The area surrounding the vehicle is divided into multiple grid areas based on the vehicle coordinate system.

[0016] The ground equation is obtained by fitting the ground laser point cloud within each grid region.

[0017] Furthermore, each of the grid regions is fitted using the least squares method or the random sampling consensus method.

[0018] Furthermore, the step of projecting the two-dimensional coordinate information onto the geodetic surface equation and performing inverse perspective transformation to obtain map information also includes:

[0019] Obtain data from the vehicle-mounted camera to obtain the transformation matrix;

[0020] The two-dimensional coordinate information of the lane is calculated with the transformation matrix to obtain the point coordinates of the lane in the vehicle coordinate system;

[0021] The map information is calculated based on the coordinates of the points.

[0022] Furthermore, the transformation matrix includes the intrinsic parameter matrix I of the vehicle-mounted camera. 3×3 extrinsic parameter matrix, rotation matrix R 3×3 Translation matrix T 3×1 .

[0023] Furthermore, the point coordinates in the two-dimensional coordinate information are P. i = (u, v, 1) T The point corresponding to this point in the camera coordinate system is P. c =(x c y c , z c ) T The position of this point in the vehicle coordinate system is represented as P. w =(x w y w , z w ) T The P i The P c and the P w The relationship between them is as follows:

[0024] P c =R 3×3 P w -T 3×1 Formula 1;

[0025]

[0026] Combining Formula 1 and Formula 2, we can obtain:

[0027]

[0028] Right now:

[0029]

[0030] Obtain the two-dimensional coordinate information in the camera coordinate system z c ;

[0031] According to the z c The position of the two-dimensional coordinate information in the vehicle coordinate system is calculated.

[0032] Furthermore, the step of obtaining the two-dimensional coordinate information in the camera coordinate system (z) c The steps include:

[0033] Discretely obtain the z c ;

[0034] According to the z c Calculate the P w ;

[0035] If the P w If the object falls within the grid area, the inverse perspective transformation is complete.

[0036] If the P wIf it does not fall within the grid area, then the z-value is reacquired. c until the P w It falls within the grid area.

[0037] Furthermore, the statement based on the z c Calculate the P w The steps also include:

[0038] If the P w If the absolute value of the nearest distance α in the equation of the Earth's surface is less than a preset value, then output P. w ;

[0039] If the P w If the absolute value of the nearest distance α in the equation of the Earth's surface is greater than the preset value and α is less than 0, then z c =z c +Δ;

[0040] If the P w If the absolute value of the nearest distance α in the equation of the Earth's surface is greater than the preset value and α is greater than 0, then z c =z c -Δ.

[0041] Furthermore, Δ represents the iteration step size.

[0042] The present invention also discloses a reverse perspective transformation system for lane line detection, including a control device. The control device includes a memory and a processor. The memory stores a control program, which, when executed by the processor, is used to implement the reverse perspective transformation method for lane line detection described above.

[0043] This invention uses laser point clouds to rigorously fit the equation of the Earth's surface, thereby abandoning the assumption of a flat Earth. By projecting two-dimensional coordinate information onto the equation of the Earth's surface and performing inverse perspective transformation, the error of the data during inverse perspective transformation is reduced. This solves the problem that the Earth is an absolute plane when the lane lines in the two-dimensional image are projected into the three-dimensional Cartesian coordinate system during inverse perspective transformation. Thus, a reliable three-dimensional lane line description equation can be obtained regardless of whether the Earth is flat (uphill or downhill).

[0044] The above and other objects, advantages and features of the present invention will become more apparent to those skilled in the art from the following detailed description of specific embodiments of the invention in conjunction with the accompanying drawings. Attached Figure Description

[0045] The following sections will describe some specific embodiments of the invention in detail by way of example and not limitation, with reference to the accompanying drawings. The same reference numerals in the drawings denote the same or similar parts or portions. Those skilled in the art should understand that these drawings are not necessarily drawn to scale. In the drawings:

[0046] Figure 1 This is a flowchart of inverse perspective transformation according to an embodiment of the present invention;

[0047] Figure 2 This is a flowchart of the fitting calculation according to an embodiment of the present invention;

[0048] Figure 3 This is a flowchart of inverse perspective calculation according to an embodiment of the present invention;

[0049] Figure 4 This is a second flowchart of inverse perspective calculation according to an embodiment of the present invention;

[0050] Figure 5 This is a third flowchart of inverse perspective calculation according to an embodiment of the present invention. Detailed Implementation

[0051] Figure 1 This is a flowchart of inverse perspective transformation according to an embodiment of the present invention. Figure 2 This is a flowchart of fitting calculation according to an embodiment of the present invention. Figure 3 This is a flowchart of inverse perspective calculation according to an embodiment of the present invention. In one embodiment, such as Figure 1 , Figure 2 and Figure 3 As shown. The inverse perspective transformation method for lane line detection includes the following steps:

[0052] S1. Obtain laser point cloud information around the vehicle and front view image information of the vehicle;

[0053] S2. Obtain the two-dimensional coordinate information of the lane lines from the front view image information, and perform road semantic segmentation on the laser point cloud information to obtain the ground laser point cloud.

[0054] S3. Fit the geodetic laser point cloud to obtain the geodetic surface equation;

[0055] S4. Project the two-dimensional coordinate information into the geodetic surface equation and perform inverse perspective transformation to obtain map information;

[0056] S5. Plan the road trajectory based on map information.

[0057] Specifically, after acquiring the laser point cloud information around the vehicle, it is necessary to extract the ground laser point cloud from it. Therefore, ground semantic segmentation is required to obtain the ground laser point cloud. Simultaneously, the vehicle's onboard camera acquires the front-view image information to obtain the two-dimensional coordinate information of the lane lines in front of the vehicle. The ground laser point cloud is then fitted and calculated to obtain the ground surface equation. This two-dimensional coordinate information is then used inverse perspective calculation with the ground surface equation to obtain map information. Based on this map information, the vehicle's road trajectory is planned, thus achieving autonomous driving.

[0058] In this embodiment, the equation of the earth surface is rigorously fitted by laser point cloud, thereby abandoning the assumption of an earth plane. By projecting two-dimensional coordinate information onto the equation of the earth surface and performing inverse perspective transformation, the error of the data during inverse perspective transformation is reduced. This solves the problem that the earth is an absolute plane when the lane lines in the two-dimensional image are projected into the three-dimensional Cartesian coordinate system during inverse perspective transformation. Thus, a reliable three-dimensional lane line description equation can be obtained regardless of whether the earth is flat (uphill or downhill).

[0059] In a further embodiment, such as Figure 3 As shown, step S3, fitting the geodetic laser point cloud to obtain the geodetic surface equation, also includes the following steps:

[0060] S30. Establish the vehicle coordinate system;

[0061] S31. Divide the area surrounding the vehicle into multiple grid areas according to the vehicle's coordinate system;

[0062] S32. Perform fitting calculations on the geodetic laser point cloud within each grid region;

[0063] S33. Compile the results of the fitting calculation for each grid into a set.

[0064] Specifically, a vehicle coordinate system is established with the rear axle center of the vehicle as the origin, the front direction of the vehicle as the positive x-axis, the left side direction of the vehicle as the positive y-axis, and the height direction of the vehicle as the positive z-axis. The surrounding area of ​​the vehicle is then divided into several gridded regions. The geodetic laser point cloud within each grid is fitted and calculated to obtain several fitting equations. Finally, each fitting equation is combined to obtain the geodetic surface fitting equation.

[0065] Each grid cell is fitted using the least squares method or RANSAC (RandomSampleConsensus) to obtain its computational equation. These equations are then grouped into a set π = {U0, U1, ..., U...}. n-1}, where U0, U1, ..., U n-1 The fitting equation for a single grid is used to obtain the overall geodetic surface fitting equation.

[0066] In this embodiment, by dividing the area around the vehicle into grids, the geodetic surface equation can be fitted more accurately, and the fitting calculation of each grid can closely approximate the actual road conditions of the vehicle. This eliminates the geodetic plane assumption and improves the accuracy of lane lines in inverse perspective calculation.

[0067] In a further embodiment, such as Figure 4 As shown, S4, projecting the two-dimensional coordinate information onto the geodetic surface equation and performing inverse perspective transformation to obtain map information includes the following steps:

[0068] S40. Obtain data information from the vehicle-mounted camera to obtain the transformation matrix;

[0069] S41. Calculate the two-dimensional coordinate information of the lane and the transformation matrix to obtain the coordinates of the lane's points in the vehicle coordinate system;

[0070] S42. Calculate map information based on point coordinates.

[0071] The transformation matrix includes the intrinsic parameter matrix I of the vehicle-mounted camera. 3×3 extrinsic parameter matrix, rotation matrix R 3×3 Translation matrix T 3×1 .

[0072] Specifically, the transformation matrix is ​​obtained through data from the vehicle-mounted camera. Only after obtaining the transformation matrix can the two-dimensional coordinate information be subjected to inverse perspective transformation to obtain the three-dimensional coordinate point P corresponding to the lane line in the vehicle's coordinate system. w .

[0073] In a specific embodiment, such as Figure 5 As shown, let the coordinates of the point in this two-dimensional coordinate information be P. i = (u, v, 1) T Then the point corresponding to this point in the camera coordinate system is P. c =(x c y c , z c ) T The position of this point in the vehicle coordinate system is represented as P. w =(x w y w , z w ) T Then P i P c and P w The relationship between them is as follows:

[0074] P c =R 3×3 P w -T 3×1 Formula 1

[0075]

[0076] Combining Formula 1 and Formula 2, we can obtain:

[0077]

[0078] Right now:

[0079]

[0080] As can be seen from the above formula, to obtain the three-dimensional coordinates of point P in the vehicle coordinate system... w Besides knowing u, v, and the intrinsic parameter matrix I, 3×3 extrinsic parameter matrix, rotation matrix R 3×3 Translation matrix T 3×1 Besides that, the only thing needed is Z. c Where u and v represent the positions of points in the two-dimensional coordinate information, and Z... c This represents the depth of that point in the camera coordinate system. Therefore, it is only necessary to obtain the Z-axis depth of the point in the camera coordinate system. c Then it can be based on Z c The position of the two-dimensional coordinate information in the vehicle coordinate system is calculated.

[0081] In one embodiment, such as Figure 5 As shown in Table 1, to accurately obtain Z c The value mainly includes the following steps:

[0082] S101. Discretely obtain z c ;

[0083] S102, according to z c Calculate P w ;

[0084] S103, if P w If the absolute value of the nearest distance α in the equation of the Earth's surface is less than the preset value, then it indicates that P w If the value falls within the grid area, then output the value P. w ;

[0085] S104, if P w If the absolute value of the nearest distance α in the equation of the Earth's surface is greater than a preset value and α is less than 0, then P w If it does not fall within the grid area, then re-acquire z. c , where: zc =z c +Δ;

[0086] S105, if P w If the absolute value of the nearest distance α in the equation of the Earth's surface is greater than a preset value and α is greater than 0, then P w If it does not fall within the grid area, then re-acquire z. c , where: z c =z c -Δ.

[0087] S106, If P w If the image falls within the grid area, the inverse perspective transformation is complete.

[0088] S107, If P w If it does not fall within the grid area, then re-acquire z. c until P w It falls within the grid area.

[0089] Where Δ is the iteration step size.

[0090] Specifically, according to Formula 4, by discretely taking different Z values... c Then, the three-dimensional point P in the vehicle coordinate system is calculated. w Because the discrete Z takes different values c It cannot guarantee the P after inverse perspective transformation w It lands exactly on the road surface, so a judgment needs to be made every time. If point P... w If the projection falls precisely on the surface of a specific region (e.g., with a preset vertical error of 0.05 meters), the inverse projection process is complete; otherwise, according to P... w The Z-axis is adjusted based on whether it falls above or below the surface of a specific region for the next discrete sampling. c That's it. The entire inverse projection process is shown in the table below.

[0091]

[0092]

[0093] Table 1

[0094] In this embodiment, by adjusting Z c The value is repeatedly verified, which can improve the accuracy of the calculation on the one hand, and ensure that the value of each region is not necessarily the same on the other hand, thus making the entire inverse perspective transformation process more accurate than existing technologies.

[0095] Furthermore, the present invention also discloses a reverse perspective transformation system for lane line detection, including a control device. The control device includes a memory and a processor. The memory stores a control program, which is executed by the processor to implement the above-mentioned reverse perspective transformation method for lane line detection.

[0096] Therefore, those skilled in the art should recognize that although numerous exemplary embodiments of the present invention have been shown and described in detail herein, many other variations or modifications conforming to the principles of the present invention can be directly determined or derived from the disclosure of the present invention without departing from the spirit and scope of the invention. Thus, the scope of the present invention should be understood and construed as covering all such other variations or modifications.

Claims

1. A method for inverse perspective transformation in lane line detection, characterized in that, Includes the following steps: Acquire laser point cloud information around the vehicle and front view image information of the vehicle; Two-dimensional coordinate information of lane lines is obtained from the front view image information, and road semantic segmentation is performed on the laser point cloud information to obtain the ground laser point cloud; The geodetic laser point cloud is fitted and calculated to obtain the geodetic surface equation; The two-dimensional coordinate information is projected into the geodetic surface equation and inverse perspective transformation is performed to obtain map information; The step of projecting the two-dimensional coordinate information onto the geodetic surface equation and performing inverse perspective transformation to obtain map information also includes: Obtain data from the vehicle-mounted camera to obtain the transformation matrix; The two-dimensional coordinate information of the lane is calculated with the transformation matrix to obtain the point coordinates of the lane in the vehicle coordinate system; The map information is calculated based on the point coordinates; The transformation matrix includes the intrinsic parameter matrix of the vehicle-mounted camera. extrinsic matrix, rotation matrix Translation matrix ; The point coordinates in the two-dimensional coordinate information are: The point corresponding to this point in the camera coordinate system is The position of this point in the vehicle coordinate system is represented as The The and stated The relationship between them is as follows: Formula 1; Formula 2; Combining Formula 1 and Formula 2, we can obtain: Formula 3; Right now: Formula 4; Obtain the two-dimensional coordinate information in the camera coordinate system ; According to the above The position of the two-dimensional coordinate information in the vehicle coordinate system is calculated; Road trajectory planning is performed based on the map information.

2. The inverse perspective transformation method according to claim 1, characterized in that, The step of fitting the geodetic laser point cloud to obtain the geodetic surface equation includes: Establish a vehicle coordinate system with the rear axle center of the vehicle as the origin, the front direction of the vehicle as the positive x-axis, the left side direction of the vehicle as the positive y-axis, and the height direction of the vehicle as the positive z-axis. The area surrounding the vehicle is divided into multiple grid areas based on the vehicle coordinate system. The ground equation is obtained by fitting the ground laser point cloud within each grid region.

3. The inverse perspective transformation method according to claim 2, characterized in that, Each grid region is fitted using either the least squares method or the random sampling consensus method.

4. The inverse perspective transformation method according to claim 1, characterized in that, The acquisition of the two-dimensional coordinate information in the camera coordinate system The steps include: Discretely obtain the ; According to the above Calculate the ; If the above If the image falls within the grid area, the inverse perspective transformation is complete. If the above If it does not fall within the grid area, then reacquire the data. until the stated It falls within the grid area.

5. The inverse perspective transformation method according to claim 4, characterized in that, According to the Calculate the The steps also include: If the above If the absolute value of the nearest distance α in the equation of the Earth's surface is less than a preset value, then the following is output: ; If the above If the absolute value of the nearest distance α in the equation of the Earth's surface is greater than the preset value and α is less than 0, then... ; If the above If the absolute value of the nearest distance α in the equation of the Earth's surface is greater than the preset value and α is greater than 0, then... ; The This is the iteration step size.

6. A lane line detection inverse perspective transformation system, characterized in that, The system includes a control device comprising a memory and a processor. The memory stores a control program, which, when executed by the processor, is used to implement the inverse perspective transformation method for lane line detection according to any one of claims 1-5.

Images