A post-processing method for lane tracking

Abstract

The present invention relates to the field of autonomous driving, and specifically discloses a post-processing method for lane line tracking, including: predicting the lane line model Model1 at time t according to the lane line model Model0 and the vehicle body postures O1 and O2 at time t-1; obtaining the lane line features at time t, recorded as Lane1; performing Hungarian matching according to the N lane line models Model1 of the current frame and the M lane line features Lane1 of the current frame to achieve a one-to-one matching relationship between the lane line model and the lane line features; obtaining the internal and external parameters CamParams of the coordinate system where the camera and the lane line model are located; calculating the vanishing point VanishPoint of the camera, calculating the pitch\roll angle of the camera according to the vanishing point, and correcting it to CamParams; correcting the lane line model Model1 with the matched lane line features and the updated camera external parameters and the vehicle body posture O2, and recording the updated lane line model as Model2. The present invention improves the accuracy of lane line tracking and increases the robustness to lane interruption, occlusion, and uphill and downhill scenes.

Description

Technical Field

[0001] The present invention relates to the field of autonomous driving, and in particular to a post-processing method for lane line tracking. Background Art

[0002] Lane tracking is typically located between the perception module and the control module. It processes the perceived lane lines into specific lane information that can be used by the control module. This plays a crucial role in intelligent driving functional modules such as lane departure warning and lane keeping assist.

[0003] Currently, common lane line tracking mainly tracks discrete feature points and performs lane line fitting. When lane line images are interrupted, blocked, or dirty, its robustness is poor, making it difficult to continuously complete good lane line tracking. In actual road conditions, uphill and downhill turns are also relatively common and pose great challenges to lane line tracking. This paper proposes a lane line tracking method that can improve the accuracy of lane line tracking and increase robustness to lane interruptions, occlusions, and uphill and downhill scenarios. Summary of the Invention

[0004] In order to solve the problems in lane line tracking caused by lane line occlusion, car driving up and downhill, and lane line detection not being 100% stable in actual road conditions, the present invention proposes a more robust lane line tracking method.

[0005] The present invention adopts the following technical solutions:

[0006] A post-processing method for lane line tracking includes the following steps:

[0007] S1: Get the vehicle body pose at time t-1 and time t, denoted as O1 and O2 respectively;

[0008] S2: Based on the lane line model Model0 and vehicle body postures O1 and O2 at time t-1, predict the lane line model Model1 at time t;

[0009] S3: Get the lane line feature at time t, recorded as Lane1;

[0010] S4: Perform Hungarian matching based on the N lane line models Model1 of the current frame and the M lane line features Lane1 of the current frame to achieve a one-to-one matching relationship between the lane line models and the lane line features;

[0011] S5: Get the internal and external parameters CamParams of the coordinate system where the camera and lane line model are located;

[0012] S6: Calculate the vanishing point of the camera, calculate the pitch and roll angles of the camera based on the vanishing point, and correct them to CamParams;

[0013] S7: Use the matched lane line features, the updated camera extrinsics, and the vehicle body posture O2 to correct the lane line model Model1, and record the updated lane line model as Model2. Output the updated lane line model Model2, which is the lane line model at time t.

[0014] As a further technical solution of the present invention, step S1 includes the following steps:

[0015] S1.1: Methods for obtaining the vehicle body pose of the previous and next frames include but are not limited to using various combinations of inertial measurement units, wheel speed meters, encoders, and GPS to obtain absolute or relative pose estimates of the vehicle body.

[0016] As a further technical solution of the present invention, step S2 includes the following steps:

[0017] S2.1: Obtain lane line model Model0. Generally, the lane line is represented as a model by fitting. The lane line model used in the present invention is:

[0018]

[0019] Where c0 represents the intercept of the lane line with the x-axis, c1 represents the slope of the lane line, c2 represents the curvature of the lane line, and c3 represents the rate of change of curvature.

[0020] S2.2 The lane parameters c0-C3 at the initial moment are calculated using the two-point method to calculate the lane feature parameters c0 and c1. The objective function is then optimized using the least squares method to obtain the lane model at the initial moment. The objective equation for constructing the least squares problem is:

[0021] f(x)=y i -f(x i );

[0022] where X i ,y i Represents the x,y coordinates of the i-th point in the bev coordinate system, and constructs the target equation f(x) to represent the error between the feature point and the curve equation in the y direction;

[0023] S2.3 For the predicted lane line model at time t, combine the lane line model Model0 at time t-1 and the vehicle body postures O1 and O2 to calculate the state transfer matrix, and then obtain Model1. The following is the state transfer equation of the lane line:

[0024]

[0025] As a further technical solution of the present invention, step S3 includes the following steps:

[0026] S3.1: Lane line features on the image extracted based on existing traditional or CNN-based methods. Lane line features are lane contour coordinate point features, and also include contour coordinate points in other coordinate systems.

[0027] As a further technical solution of the present invention, step S4 includes the following steps:

[0028] S4.1: As described in step S3, M sets of lane features are obtained. Lane parallelism theory is not pre-set here. Initially detected lane features generate corresponding lane model Model 0. Non-initial detected lane features enter the matching phase. This phase should include screening of lane features to ensure that the input lane features are correct and reasonable. Screening methods include, but are not limited to:

[0029] The lane line feature length is screened. Lane lines with too few feature points or whose actual length is not up to standard will be deleted.

[0030] Lane line attributes should be eliminated. Non-lane input should be excluded based on attributes.

[0031] S4.2: Match the M sets of filtered lane features with the N sets of lane models obtained in step S2. In this example, the Hungarian matching method is selected. The weights in the Hungarian matching method are calculated by averaging the Euclidean distances from all points in the lane features to the lane lines.

[0032] S4.3: Obtain a one-to-one correspondence between the lane line model and the lane line features based on the Hungarian optimal matching relationship.

[0033] As a further technical solution of the present invention, step S5 includes the following steps:

[0034] The internal and external parameters of the camera and lane line coordinate systems are obtained through the calibration algorithm. Based on the existing internal and external parameters, the coordinate system transformation of the lane feature points is completed. For example, during the matching process in step S4, the lane feature points need to be projected into the lane line coordinate system to calculate the weight.

[0035] As a further technical solution of the present invention, step S6 includes the following steps:

[0036] S6.1: Determine the camera model for the lane feature. If it is a fisheye camera model, convert it to a linear camera model to obtain new lane features. Fit each lane feature with a straight line equation: y = kx + b. Calculate the intersection of multiple straight line equations to obtain the camera's vanishing point (denoted as VanishPoint).

[0037] S6.2: Given the camera intrinsic parameter K and the vanishing point V calculated in step S6.1, calculate the corrected roll and pitch angles. The calculation process is as follows:

[0038]

[0039] zV=Kr3;

[0040] zV represents the coordinates of the vanishing point on the image, K is the memory, r1 represents the first column element of the rotation matrix from the world coordinate system to the camera coordinate system, and the coordinates of the vanishing point in the world coordinate system are Where R = Rroll*Rpitch*Ryaw, r3 can be expressed as follows:

[0041]

[0042] pitch=-sin -1 (r31);

[0043] roll=tan -1 (r32 / r33);

[0044] S6.3 updates the corrected yaw and roll angles to the camera extrinsics.

[0045] Compared with the prior art, the present invention has the following beneficial effects:

[0046] 1. Currently, lane feature tracking is typically based on discrete feature points, upon which lane fitting is performed. This invention, by tracking the lane model, can better maintain the consistency of the lane equation and prevent discontinuities caused by misdetection of lane feature points when the time series changes.

[0047] 2. Existing lane tracking methods cannot simultaneously address real-world road conditions, such as non-parallel lane lines, lane line occlusion, dirt, and lane tracking errors caused by uphill and downhill slopes. This invention circumvents the requirement for lane line parallelism by continuously tracking the lane line model. It also exhibits high robustness to short-term or partial occlusion and dirtiness of lane lines. Furthermore, by correcting camera extrinsics, it can better address large tracking errors caused by uneven road surfaces.

[0048] 3. The solution of the present invention is relatively complete, can solve various problems in real road conditions, and has high practicability. BRIEF DESCRIPTION OF THE DRAWINGS

[0049] Figure 1 4 is a flowchart of a post-processing method for lane line tracking in an embodiment of the present invention.

[0050] Figure 2 This is a derivation diagram of the pic2 lane line model in an embodiment of the present invention. DETAILED DESCRIPTION

[0051] The following is a clear and complete description of the technical solutions in the embodiments of the present invention in conjunction with specific embodiments. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of the present invention.

[0052] The present invention provides a post-processing method for lane line tracking. The method predicts the lane line model at time t based on the lane line model at time t-1 and the vehicle body posture at time t-1 and t, and performs a one-to-one matching with the lane line features acquired at time t to obtain a pairing relationship. The vanishing point is calculated based on the acquired lane line features at time t, and the camera's extrinsic parameters yaw and roll are then updated. Based on the paired lane line model and lane line features at time t, combined with the latest camera extrinsic parameters and the vehicle body posture at time t, the lane line model tracking result at time t is corrected.

[0053] See also Figures 1 to 2 , an embodiment of the present invention provides a post-processing method for lane line tracking, comprising the following steps:

[0054] S1: Get the vehicle body pose at time t-1 and time t, denoted as O1 and O2 respectively;

[0055] S2: Based on the lane line model Model0 and vehicle body postures O1 and O2 at time t-1, predict the lane line model Model1 at time t;

[0056] S3: Get the lane line feature at time t, recorded as Lane1;

[0057] S4: Perform Hungarian matching based on the N lane line models Model1 of the current frame and the M lane line features Lane1 of the current frame to achieve a one-to-one matching relationship between the lane line models and the lane line features;

[0058] S5: Get the internal and external parameters CamParams of the coordinate system where the camera and lane line model are located;

[0059] S6: Calculate the vanishing point of the camera, calculate the pitch and roll angles of the camera based on the vanishing point, and correct them to CamParams;

[0060] S7: Use the matched lane line features, the updated camera extrinsics, and the vehicle body posture O2 to correct the lane line model Model1, and record the updated lane line model as Model2. Output the updated lane line model Model2, which is the lane line model at time t.

[0061] In this embodiment of the present invention, step S1 includes the following steps:

[0062] S1.1: Methods for obtaining the vehicle body pose of the previous and next frames include but are not limited to using various combinations of inertial measurement units, wheel speed meters, encoders, and GPS to obtain absolute or relative pose estimates of the vehicle body.

[0063] The present invention uses the inertial measurement unit (IMU) and wheel speedometer information, combined with the ESKF algorithm to perform trajectory extrapolation, and calculates the vehicle's external parameter information: posture, rotation angle, and vehicle speed. It then integrates the linear acceleration and angular acceleration information of the IMU and uses them as input for vehicle posture information. This method of calculating vehicle posture information can rely on fewer sensors, reducing engineering costs.

[0064] The trajectory calculation formula is as follows:

[0065] or

[0066]

[0067]

[0069]

[0070] Sampling time Δt, vehicle angular velocity and acceleration at sampling time b g , b a It is the zero bias part. Through integration over a period of time, we can obtain the integration results R (rotational external parameters of the vehicle at time t), p (position of the vehicle at time t), and v (speed of the vehicle at time t) in continuous time.

[0071] In this embodiment, step S2 includes the following steps:

[0072] S2.1: Obtain lane line model Model0. Generally, the lane line is represented as a model by fitting. The lane line model used in the present invention is:

[0073]

[0074] Where c0 represents the intercept of the lane line with the x-axis, c1 represents the slope of the lane line, c2 represents the curvature of the lane line, and c3 represents the rate of change of curvature.

[0075] S2.2 The lane parameters c0-C3 at the initial moment are calculated using the two-point method to calculate the lane feature parameters c0 and c1. The objective function is then optimized using the least squares method to obtain the lane model at the initial moment. The objective equation for constructing the least squares problem is:

[0076] f(x)=y i -f(X i );

[0077] where X i ,y i Represents the x,y coordinates of the i-th point in the bev coordinate system, and constructs the target equation f(x) to represent the error between the feature point and the curve equation in the y direction;

[0078] S2.3 For the predicted lane line model at time t, combine the lane line model Model0 at time t-1 and the vehicle body postures O1 and O2 to calculate the state transfer matrix, and then obtain Model1. The following is the state transfer equation of the lane line:

[0079]

[0080] The derivation process can be referred to Figure pic2. Taking parameter c0 as an example, the state transition equation is explained as follows:

[0081] At time t-1, c0=f(0). At time t, the coordinate system moves with the vehicle to the intermediate change coordinate system xy. In the new coordinate system, c0 changes to c′0=dx+dx1. At time t, the final coordinate system at time t is obtained by rotating the direction angle of the intermediate change coordinate system xy. In the coordinate system at time t, c′0 is expressed as c″0, plus rotation compensation. Here, we set the motion compensation to dy*θ, so that the change equation from the previous frame coordinate system to the current frame coordinate system c0 can be obtained as follows: c″0=dx+dx1+dy*θ. Expanding the formula yields:

[0082]

[0083] Where yawrate is the angle change rate, and d*sinθ is added to c0 separately as motion compensation;

[0084] The other three parameters can be derived similarly to obtain the lane line transfer equation from time t-1 to time t.

[0085] In this embodiment of the present invention, step S3 includes the following steps:

[0086] S3.1: Lane line features on the image extracted based on existing traditional or CNN-based methods. Lane line features are lane contour coordinate point features, and also include contour coordinate points in other coordinate systems.

[0087] In this embodiment of the present invention, step S4 includes the following steps:

[0088] S4.1: As described in step S3, M sets of lane features are obtained. Lane parallelism theory is not pre-set here. Initially detected lane features generate corresponding lane model Model 0. Non-initial detected lane features enter the matching phase. This phase should include screening of lane features to ensure that the input lane features are correct and reasonable. Screening methods include, but are not limited to:

[0089] The lane line feature length is screened. Lane lines with too few feature points or whose actual length is not up to standard will be deleted.

[0090] Lane line attributes should be eliminated. Non-lane input should be excluded based on attributes.

[0091] S4.2: Match the M sets of filtered lane features with the N sets of lane models obtained in step 2. In this example, the Hungarian matching method is selected. The weights in the Hungarian matching method are calculated by averaging the Euclidean distances from all points in the lane features to the lane lines.

[0092] S4.3: Obtain a one-to-one correspondence between the lane line model and the lane line features based on the Hungarian optimal matching relationship.

[0093] In this embodiment of the present invention, step S5 includes the following steps:

[0094] The internal and external parameters of the camera and lane coordinate systems are obtained through a calibration algorithm. Based on the existing internal and external parameters, the coordinate system of the lane feature points is transformed. For example, during the matching process in step S4, the lane feature points need to be projected into the lane coordinate system to calculate the weights.

[0095] Generally, more accurate camera internal and external parameters can be obtained through factory or off-line calibration.

[0096] In this embodiment of the present invention, step S6 includes the following steps:

[0097] S6.1: Determine the camera model for the lane feature. If it is a fisheye camera model, convert it to a linear camera model to obtain new lane features. Fit each lane feature with a straight line equation: y = kx + b. Calculate the intersection of multiple straight line equations to obtain the camera's vanishing point (denoted as VanishPoint).

[0098] S6.2: Given the camera intrinsic parameter K and the vanishing point V calculated in S6.1, calculate the corrected roll and pitch angles. The calculation process is as follows:

[0099]

[0100] zV=Kr3;

[0101] zV represents the coordinates of the vanishing point on the image, K is the memory, r1 represents the first column element of the rotation matrix from the world coordinate system to the camera coordinate system, and the coordinates of the vanishing point in the world coordinate system are Where R = Rroll*Rpitch*Ryaw, r3 can be expressed as follows:

[0102]

[0103] pitch=-sin -1 (r31);

[0104] roll=tan -1 (r32 / r33);

[0105] S6.3 updates the corrected yaw and roll angles to the camera extrinsics.

[0106] It will be apparent to those skilled in the art that the present invention is not limited to the details of the exemplary embodiments described above and that the invention can be embodied in other specific forms without departing from the spirit or essential characteristics of the invention. Therefore, the embodiments should be considered in all respects as illustrative and non-restrictive, and the scope of the invention is defined by the appended claims rather than the foregoing description, and it is intended that all variations that come within the meaning and range of equivalents of the claims be embraced therein.

[0107] In addition, it should be understood that although this specification is described in terms of implementation methods, not every implementation method contains only one independent technical solution. This narrative method of the specification is only for the sake of clarity. Those skilled in the art should regard the specification as a whole. The technical solutions in each embodiment can also be appropriately combined to form other implementation methods that can be understood by those skilled in the art.

Claims

1. A post-processing method for lane line tracking, characterized in that: The following steps are involved: S1: Get the vehicle posture at time t-1 and time t, recorded as O1 and O2 respectively; S2: Based on the lane line model Model0 at time t-1 and the vehicle body postures O1 and O2, the lane line model Model1 at time t is predicted; S3: Get the lane line feature at time t, recorded as Lane1; S4: Perform Hungarian matching based on the N lane line models Model1 at time t-1 and the M lane line features Lane1 at time t to achieve a one-to-one matching relationship between the lane line model and the lane line features; S5: Get the internal and external parameters CamParams of the coordinate system where the camera and lane line model are located; S6: Calculate the vanishing point of the camera, calculate the pitch\roll angle of the camera according to the vanishing point, and correct it to CamParams; S7: Use the matched lane line features, the updated camera external parameters and the vehicle body posture O2 to correct the lane line model Model1, and record the updated lane line model as Model2, and output the updated lane line model Model2, which is the lane line model at time t.

2. A lane line tracking post-processing method according to claim 1, characterized in that: Step S1 includes the following steps: S1.1: The method of obtaining the vehicle posture of the previous and next two frames includes but is not limited to using a combination of an inertial measurement unit, a wheel speed meter, an encoder and a GPS to obtain an absolute or relative posture estimate of the vehicle body.

3. A lane line tracking post-processing method according to claim 2, characterized in that: Step S2 includes the following steps: S2.1: Get the lane line model Model0, and express the lane line in the form of a model by fitting. The lane line model is: Where c0 represents the intercept of the lane line with the x-axis, c1 represents the slope of the lane line, c2 represents the curvature of the lane line, and c3 represents the rate of change of the curvature; S2.2 The calculation method for the lane line parameters c0-c3 at the initial moment is to use the two-point method to calculate the parameters c0 and c1 of the lane line characteristics, and then optimize the objective function through the least squares method to obtain the lane line model at the initial moment; the objective equation for constructing the least squares problem is: f(x)=y i -f(x i ); where x i ,y i represents the x,y coordinates of the ith point in the bev coordinate system, and constructs the target equation f(x) to represent the error between the feature point and the curve equation in the y direction; S2.3 For the predicted lane line model at time t, the state transfer matrix is ​​calculated by combining the lane line model Model0 at time t-1 and the vehicle body posture O1, O2, and then Model1 is obtained. The following is the state transfer equation of the lane line:

4. The post-processing method for lane line tracking according to claim 1, characterized in that: Step S3 includes the following steps: S3.1: Lane line features on the image extracted by the CNN-based method. Lane line features are lane contour coordinate point features, which also include contour coordinate points in other coordinate systems. One lane line corresponds to a set of lane line contour coordinate points, and multiple lane line information output by the model is obtained.

5. The post-processing method for lane line tracking according to claim 1, characterized in that: Step S4 includes the following steps: S4.1: As described in step S3, M groups of lane line features are obtained. The lane line parallel theory is not preset here. The lane line features detected for the first time generate the corresponding lane line model Model0. The lane line features not detected for the first time enter the matching stage. This should include the screening of lane line features to ensure that the input of lane line features is correct and reasonable. The screening methods include but are not limited to: The lane line feature length screening, if there are too few feature points or the actual length of the lane line is not up to standard, it will be deleted; Lane line attribute selection: non-lane input should be excluded according to the attribute; S4.2: Matching the filtered M groups of lane line features with the N groups of lane line models obtained in step 2. Taking the present invention as an example, the Hungarian matching method is selected. The weight calculation method in the Hungarian matching is: averaging the Euclidean distances from all points in the lane line features to the lane line; S4.3: According to the Hungarian optimal matching relationship, the one-to-one correspondence between the lane line model and the lane line features is obtained.

6. A lane line tracking post-processing method according to claim 1, characterized in that: Step S5 includes the following steps: The internal and external parameters of the camera and lane line coordinate systems are obtained through the calibration algorithm; Based on the internal and external parameters, the coordinate system transformation of the lane feature points is completed, and the lane feature points are projected into the lane line coordinate system to calculate the weights.

7. The post-processing method for lane line tracking according to claim 1, characterized in that: Step S6 includes the following steps: S6.1: Determine the camera model where the lane feature is located. If it is a fisheye camera model, convert it to a linear camera model to obtain a new lane feature. Fit the straight line equation of each lane feature: y=kx+b, calculate the intersection of multiple straight line equations, and obtain the vanishing point V of the camera; S6.2: Given the camera intrinsic parameter K and the vanishing point V calculated in step S6.1, calculate the corrected roll and pitch angles. The calculation process is as follows: zV=Kr3; zV represents the coordinates of the vanishing point on the image, K is the memory, r1 represents the first column element of the rotation matrix from the world coordinate system to the camera coordinate system, and the coordinates of the vanishing point in the world coordinate system are Where R = Rroll*Rpitch*Ryaw, r3 can be expressed as follows: pitch=-sin -1 (r31); <h2 style=";text-align:left;direction:ltr">roll=tan<h2 style=";text-align:left;direction:ltr"> -1 <h2 style=";text-align:left;direction:ltr"> (r32 / r33); S6.3 updates the corrected yaw and pitch angles to the camera external parameters.

Images