A lane line equation fitting and curvature calculation method based on EKF multi-model fusion

Abstract

The application discloses a lane line equation fitting and curvature calculation method based on EKF multi-model fusion, and comprises the following steps: A, image information acquisition; B, image preprocessing; C, inverse perspective transformation lane line detection; D, lane line detection based on Ultra-Fast-Lane-Net; E, lane line coordinate point fusion by improving an extended Kalman filtering algorithm; F, lane line cubic equation construction; and G, curvature calculation.In step E, when constructing a system state equation, a change amount of a deep learning model at t time relative to t-1 time is added to the system state equation, and the extended Kalman filtering algorithm is improved.The application can well fuse two detection results, solves the disadvantage that a traditional algorithm lacks prior visual objects, and compared with a deep learning method, the fusion model has smaller system error.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of image processing, and in particular to a lane line equation fitting and curvature calculation method based on EKF multi-model fusion BACKGROUND

[0002] At present, in the field of automatic driving and auxiliary driving, the method for detecting lane lines includes a traditional image inverse perspective transformation detection method and a lane line detection method through a deep learning detection model. The above two algorithms have their own advantages and disadvantages. The traditional image detection method can only calculate through the lane line marks in the collected image, lacks prior visual objects, and in the case that the lane line is blocked by a vehicle due to traffic congestion, there is no lane line or a large number of missing in the image. In this case, there is no visual information (color and shape of the lane) to guide the recognition of the lane line, which causes great difficulty to the traditional detection method. The latter method through deep learning can make a guess according to the position of the vehicle and the semantic information of the environment, but the actual use effect is affected by the difference between the training sample set and the actual route, and the system error is uncontrollable. When there is a large difference between the actual driving environment and the training sample, the deep learning system error has great uncertainty, and the effect is even worse than that of the traditional detection method. However, the detection results obtained by the above two methods are difficult to directly fuse, and the prior art fails to realize the fusion and complementation of the two methods under the premise of ensuring high accuracy and stability. SUMMARY

[0003] The purpose of the present application is to provide a lane line equation fitting method based on EKF multi-model fusion, which is used to solve the technical problem that the traditional inverse perspective transformation method and the deep learning detection method for lane line detection have defects and are difficult to fuse, resulting in insufficient stability and accuracy in the prior art.

[0004] The lane line equation fitting method based on EKF multi-model fusion comprises the following steps:

[0005] A: image information acquisition;

[0006] B: image preprocessing;

[0007] C: inverse perspective transformation lane line detection;

[0008] D: lane line detection based on Ultra-Fast-Lane-Net;

[0009] E: lane line coordinate point fusion through an improved extended Kalman filter algorithm;

[0010] F: construction of a cubic equation of the lane line;

[0011] In step E, the system state equation and the system observation equation are constructed in combination with the vehicle driving equation, the variation of the deep learning model at time t relative to time t-1 is added to the system state equation when constructing the system state equation, and an improved extended Kalman filter fusion algorithm is obtained, the two sets of lane line marker point coordinates detected in steps C and D are fused by the improved extended Kalman filter fusion algorithm, and step F utilizes the least square method to perform cubic curve fitting on the fused lane line marker point coordinates, and the detection of the lane line is completed.

[0012] Preferably, in step E, since the inverse perspective transformation and the deep learning detection model do not have a fixed model formula, the system state equation is improved as follows:

[0013] F t =F t-1 +(N dn，t -N dn，t-1 )+Q t-1

[0014] Wherein, F t is the output of the fusion system model at time t, N dn，t -N dn，t-1 is the variation of the deep learning model (Ultra-Fast-Lane-Net) at time t relative to time t-1, N dn，t is the output of the deep learning model at time t, and Q t-1 is a system state error matrix.

[0015] The system observation equation is:

[0016] Z t =HF t +R t =N dm，t

[0017] Wherein, R t is an observation value error matrix, the measurement value Z t and the state value N dm，t are the same variable, H=1, the Jacobian matrix and the error covariance matrix are calculated according to the existing formula.

[0018] Preferably, the specific steps of step E are as follows:

[0019] (1) obtaining the lane line marker point coordinates of step C and the lane line marker point coordinates of step D as input;

[0020] (2) constructing the system state equation and the system observation equation;

[0021] (3) calculating the Jacobian matrix of the system state equation and the system observation equation;

[0022] (4) Calculate the system error covariance matrix;

[0023] (5) Predict lane line marker point coordinates based on the Jacobian matrix;

[0024] (6) Combine the predicted lane line marker point coordinates with the system error covariance matrix to calculate the Kalman gain;

[0025] (7) Update the lane line marker point coordinates and the system error covariance matrix according to the calculation results of the previous step, and complete the fusion.

[0026] Preferably, the step C specifically comprises:

[0027] (1) First, collect a single lane picture in the experiment, and determine the inverse perspective transformation matrix coefficient and the perspective transformation matrix coefficient according to the relationship between the target point in the original picture and the target point in the perspective view;

[0028] (2) In step C, inverse perspective transformation is performed on the processed image information, and the inverse perspective transformation matrix coefficient determined in the previous experiment is used to obtain an inverse perspective transformation image;

[0029] (3) The lane line marker points are obtained by sliding window, that is, the inverse perspective transformation image is subjected to marker point recognition;

[0030] (4) The inverse perspective transformation image with lane line marker points is subjected to perspective transformation again to obtain the original lane line marker point coordinates.

[0031] Preferably, the step C specifically comprises:

[0032] (1) Obtain the lane line dataset before use, train the Ultra-Fast-Lane-Net model with the lane line dataset, and obtain the trained Ultra-Fast-Lane-Net model;

[0033] (2) In step D, the processed image information is input into the trained Ultra-Fast-Lane-Net model, and the lane line marker point coordinates are output.

[0034] Preferably, the step B corresponds to steps C and D and includes two parts of image preprocessing:

[0035] (1) The inverse perspective transformation image is preprocessed to correct image distortion.

[0036] (2) The image corresponding to the deep learning model is preprocessed, including image size change and image enhancement.

[0037] Preferably, the image preprocessing in step B further includes image graying, Gaussian filter processing, image binarization, and image denoising.

[0038] Based on the above technical scheme, the application further provides a curvature calculation method of a lane line based on EKF multi-model fusion, wherein the detection of the lane line is completed according to the lane line equation fitting method based on EKF multi-model fusion as described above, and a cubic equation formula of the lane line is fitted as follows:

[0039] y = a3x 3 + a2x 2 + a1x + a0

[0040] The method further comprises the following steps:

[0041] G: curvature calculation;

[0042] According to the curvature calculation formula of the cubic equation of the lane line:

[0043]

[0044] The curvature of the lane line is calculated.

[0045] The application has the following advantages: the method combines the inverse perspective transformation lane line detection method and the lane line detection method based on Ultra-Fast-Lane-Net, which are originally independent lane line detection methods, and through the improved extended Kalman filtering algorithm, the change amount of the output of the deep learning model between adjacent time is considered in the system state equation, so that the two detection results can be well fused, and the detection result after fusion has better accuracy and stability than the independent detection method before fusion. The method uses the fusion model to solve the problem that the traditional algorithm lacks prior visual objects, and compared with the deep learning method, the fusion model has smaller system error. BRIEF DESCRIPTION OF DRAWINGS

[0046] Figure 1 FIG. 1 is a flowchart of the lane line equation fitting and curvature calculation method based on EKF multi-model fusion of the application.

[0047] Figure 2 FIG. 2 is a flowchart of the inverse perspective transformation lane line detection method.

[0048] Figure 3 FIG. 3 is a flowchart of the lane line detection method based on Ultra-Fast-Lane-Net.

[0049] Figure 4 FIG. 4 is a model structure diagram of Ultra-Fast-Lane-Net.

[0050] Figure 5 FIG. 5 is a model structure diagram of ResNet34 shown in FIG. 4. Figure 4 FIG. 5 is a model structure diagram of ResNet34 shown in FIG. 4.

[0051] Figure 6 Flow chart of the improved extended Kalman filter model fusion algorithm of step E in the application. DETAILED DESCRIPTION

[0052] The specific embodiments of the application will be further described in detail below with reference to the accompanying drawings, and the description of the embodiments will help the skilled in the art to have a more complete, accurate and in-depth understanding of the inventive concept and technical solutions of the application.

[0053] As shown in the drawings, the application provides a lane line equation fitting and curvature calculation method based on EKF multi-model fusion, including the following steps: Figure 1 A: image information acquisition.

[0054] The lane line image data is obtained through software and hardware means, such as various vehicle-mounted cameras.

[0055] B: image preprocessing.

[0056] After obtaining the image data, the image needs to be preprocessed, including two parts.

[0057] (1) The inverse perspective transformation image is preprocessed, and the image distortion correction is performed.

[0058] (2) The image corresponding to the deep learning model is preprocessed, including image size change and image enhancement.

[0059] C: inverse perspective transformation lane line detection, as shown in the drawings, the specific steps of this step are as follows.

[0060] Figure 2 First, a single lane picture is collected in the experiment, and the inverse perspective transformation matrix coefficient and the perspective transformation matrix coefficient are determined according to the relationship between the target points in the original image and the target points in the perspective image.

[0061] In actual application, after step A and step B of the foregoing total steps, image information acquisition and image preprocessing are sequentially performed, wherein the preprocessing methods include but are not limited to image graying, Gaussian filter processing, image binarization and image denoising.

[0062] Then in step C, the inverse perspective transformation is performed on the processed image information, and the inverse perspective transformation matrix coefficient determined in the previous experiment is adopted to obtain the inverse perspective transformation image.

[0063] The lane line marker points are obtained through the sliding window, that is, the inverse perspective transformation image is subjected to marker point recognition.

[0064] The lane line marker points are obtained through the sliding window, that is, the inverse perspective transformation image is subjected to marker point recognition.

[0065] ​The inverse perspective transformation image with lane line mark points is subjected to perspective transformation again to obtain original lane line mark points and lane line mark point coordinates.

[0066] D: Lane line detection based on Ultra-Fast-Lane-Net.

[0067] Lane line detection is performed by using the Ultra-Fast-Lane-Net model. The principle is to introduce row anchors to convert the segmentation problem into a classification problem, greatly improving the speed and solving the problem of inferring the lane when there is no lane on the graph. As shown in Figures 3-5 , the specific steps are as follows.

[0068] Before use, the lane line dataset is obtained to train the Ultra-Fast-Lane-Net model, and the trained Ultra-Fast-Lane-Net model is obtained. The input of the Ultra-Fast-Lane-Net model is the collected image information, and the output is the lane line mark point coordinates.

[0069] In actual application, the image information is collected and the collected data is preprocessed by the steps A and B of the foregoing total steps.

[0070] Then in step D, the processed image information is input into the trained Ultra-Fast-Lane-Net model, and the lane line mark point coordinates are output.

[0071] E: Lane line coordinate point fusion is performed by improving the extended Kalman filter (EKF) algorithm.

[0072] Two equations constructed in combination with the vehicle driving equation are used to improve the conventional EKF algorithm (extended Kalman filter algorithm) to form an improved extended Kalman filter fusion algorithm. The two sets of lane line mark point coordinates detected by the two model detection steps C and D are fused by the improved algorithm.

[0073] As shown in Figure 6 , the specific steps are as follows:

[0074] (1) Obtain the lane line mark point coordinates obtained by the inverse perspective transformation detection method in step C and the lane line mark point coordinates obtained by the deep learning model in step D as inputs.

[0075] (2) Construct system state equations and system observation equations.

[0076] Since the inverse perspective transformation and the deep learning detection model have no fixed model formula, the system state equation is improved as follows:

[0077] F t = Ft-1 +(N dn，t -N dn，t-1 )+Q t-1

[0078] Among them, F t For the output of the fusion system model at time t, N dn，t -N dn，t-1 N represents the change in the deep learning model (Ultra-Fast-Lane-Net) at time t relative to time t-1. dn，t Let Q be the output of the deep learning model at time t. t-1 This is the system state error matrix.

[0079] The system observation equation is:

[0080] Z t =HF t +R t =N dm，t

[0081] Among them, R t Let Z be the observation error matrix and the measured value Z be the measured error matrix. t With state value N dm，t For the same variable, H = 1. The Jacobian matrix and error covariance matrix are calculated using existing formulas.

[0082] (3) Calculate the Jacobian matrix of the system state equation and the system observation equation.

[0083] (4) Calculate the system error covariance matrix.

[0084] (5) Predict the coordinates of lane line markers based on the Jacobian matrix.

[0085] (6) Calculate the Kalman gain by combining the predicted lane line marker coordinates with the system error covariance matrix.

[0086] (7) Update the lane line marker coordinates and system error covariance matrix based on the calculation results of the previous step to complete the fusion.

[0087] By applying the improved extended Kalman filter fusion algorithm described above to the two sets of lane line marker coordinates detected by the two models in steps C and D, the fused lane line marker coordinates can be obtained.

[0088] F: Constructing the cubic equation of the lane line.

[0089] The least squares method is used to fit the coordinates of the merged lane line markers. The coordinates of the lane line markers are obtained by fitting the cubic equation model of the lane lines, thus completing the detection of lane lines, which include left and right lane lines.

[0090] Fitting lane line cubic equation formula:

[0091] y=a3x+a2x+a1x+a0 3 2

[0092] The specific method process of least square curve fitting is prior art, and will not be described here.

[0093] G: curvature calculation.

[0094] According to the lane line cubic equation, the curvature of the lane line is calculated, and the curvature calculation formula is:

[0095]

[0096] The curvature of the lane line is used to control the vehicle driving.

[0097] In the above method, the lane line coordinate points obtained by the original inverse perspective transformation lane line detection and the lane line detection based on Ultra-Fast-Lane-Net are independently acquired, and it cannot be determined which side has better effect. By improving and extending the Kalman filtering algorithm, the change amount of the output of the deep learning model between adjacent time is considered in the system state equation, so that the output results are the good fusion of the two detection results of the lane line marker point coordinates, and the detection results obtained after fusion are more accurate and stable due to the independent detection method before fusion, and the advantages of both are considered.

[0098] The above describes the present application in conjunction with the drawings, and it is obvious that the specific implementation of the present application is not limited by the above method, as long as various non-essential improvements are made by using the inventive concept and technical solution of the present application, or the inventive concept and technical solution of the present application is directly applied to other occasions without improvement, all of which are within the protection scope of the present application.​​

Claims

1. A lane line equation fitting method based on EKF multi-model fusion, characterized in that: Comprise the following steps: A: image information acquisition; B: image preprocessing; C: inverse perspective transformation lane line detection; D: lane line detection based on Ultra-Fast-Lane-Net; E: lane line coordinate point fusion by improved extended Kalman filtering algorithm; F: lane line cubic equation construction; In step E, the system state equation and the system observation equation are constructed by combining the vehicle driving equation, and the change of the deep learning model at time t relative to time t-1 is added to the system state equation, obtaining an improved extended Kalman filtering fusion algorithm. The two sets of lane line marker point coordinates detected in steps C and D are fused by the improved extended Kalman filtering fusion algorithm. In step F, the least squares method is used to fit the fused lane line marker point coordinates to a cubic curve, completing the detection of the lane line. In step E, since there is no fixed model formula for inverse perspective transformation and deep learning detection model, the improved system state equation is as follows: wherein, is the output of the fusion system model at time t, is the change of the deep learning model at time t relative to time t-1, is the output of the deep learning model at time t, is a system state error matrix; the deep learning model adopts an Ultra-Fast-Lane-Net model; The system observation equation is: wherein is the observation error matrix, the measurement and the state value are the same variable, is the identity matrix, the Jacobian matrix, the error covariance matrix is calculated according to the existing formula; Step C specifically includes: (1) First, collect a single lane picture in the experiment, and determine the inverse perspective transformation matrix coefficient and the perspective transformation matrix coefficient according to the relationship between the target point in the original picture and the target point in the perspective picture; (2) In step C, the inverse perspective transformation is performed on the processed image information, and the inverse perspective transformation matrix coefficient determined in the previous experiment is used to obtain the inverse perspective transformation image; (3) Lane line marker points are obtained by sliding window, that is, marker point recognition is performed on the inverse perspective transformation image; (4) The inverse perspective transformation image with lane line marker points is further perspective transformed to obtain the original lane line marker point coordinates; Step D specifically includes: (1) Before use, first obtain the lane line dataset to train the Ultra-Fast-Lane-Net model, and obtain the trained Ultra-Fast-Lane-Net model; (2) In step D, the processed image information is input into the trained Ultra-Fast-Lane-Net model, and the lane line marker point coordinates are output.

2. The lane line equation fitting method based on EKF multi-model fusion according to claim 1, characterized in that: The specific steps of step E are as follows: (1) Obtain the lane line marker point coordinates of step C and the lane line marker point coordinates of step D as input; (2) Construct the system state equation and the system observation equation; (3) Calculate the Jacobian matrix of the system state equation and the system observation equation; (4) Calculate the system error covariance matrix; (5) Predict the lane line marker point coordinates based on the Jacobian matrix; (6) Calculate the Kalman gain by combining the predicted lane line marker point coordinates with the system error covariance matrix; (7) Update the lane line marker point coordinates and the system error covariance matrix based on the calculation result of the previous step to complete the fusion.

3. The lane line equation fitting method based on EKF multi-model fusion of claim 1, wherein: Step B corresponding to step C and step D includes two parts of image preprocessing: (1) Preprocess the inverse perspective transformation image to correct image distortion; (2) Preprocess the image corresponding to the deep learning model, including image size change and image enhancement.

4. The lane line equation fitting method based on EKF multi-model fusion of claim 3, wherein: The image preprocessing in step B further includes image graying, Gaussian filtering processing, image binarization, and image denoising.

5. A lane line curvature calculation method based on EKF multi-model fusion, characterized in that: A lane line equation fitting method based on EKF multi-model fusion according to any one of claims 1-4 completes lane line detection, and a cubic equation formula of the lane line is fitted as: The method further comprises the following steps: G: curvature calculation; According to the curvature calculation formula of the cubic equation of the lane line: The curvature of the lane line is calculated.

Images