Lane line optimization method and electronic device

Abstract

The application relates to the technical field of intelligent auxiliary driving, and provides a lane line optimization method and electronic equipment, a classical Kalman filtering-based algorithm framework model is set, vehicle motion information, IMU information, HDmap information, INS information and sensing lane line information are used to optimize lane line precision, optimize lane line detection stability, robustness and simultaneously detect abnormal lane line information; a main model is based on adaptive Kalman filtering, vehicle motion information is combined to predict lane line information, high-precision map lane lines are combined, and sensing lane line information is used to realize observation updating compensation; wherein observation and prediction residual information are used to realize abnormal detection, and can provide stable and reliable guarantee for an automatic driving vehicle running on a public road.

Description

Technical Field

[0001] This invention relates to the field of intelligent assisted driving technology, and in particular to a lane line optimization method and electronic device. Background Technology

[0002] As a key component of advanced driver assistance systems (ADAS), lane markings are being studied in increasingly diverse scenarios. For example, they are used in lane keeping assist, lane departure warning, and adaptive cruise control. However, in real-world traffic environments, objective factors such as weather changes, lighting conditions, road damage, or camera malfunctions can easily cause lane marking generation failures, disrupting vehicle operation and potentially degrading or disabling the ADAS.

[0003] The key parameters (c0, c1, c2, c3) of the classic lane line equation represent "the lateral distance of the vehicle from the lane line, the lane line heading angle, the lane line curvature, and the rate of change of curvature," respectively. This traditional lane line perception scheme suffers from problems such as false detections, missed detections, unstable detection results, and poor / even incorrect lane line curvature estimation accuracy. For example:

[0004] 1) Lane line accuracy depends on the results of the sensing end. Lane line accuracy is limited by distance and restricted by the sensing range. The farther away from the vehicle, the lower the accuracy.

[0005] 2) Abnormal lane lines at intersections cannot be distinguished, and intersection scenarios cannot be effectively identified. Lane line perception will be abnormal in situations such as lane merging during lane changes, thus failing to provide effective and accurate lane line recognition services. Summary of the Invention

[0006] This invention provides a lane line optimization method and electronic device, which solves the technical problem that the perception accuracy of existing lane line perception schemes is limited by the perception range and cannot effectively identify abnormal lane lines, resulting in poor lane line recognition accuracy.

[0007] To address the above technical problems, this invention provides a lane line optimization method and electronic device, comprising the following steps:

[0008] Data collection is performed to obtain vehicle motion information, IMU information, HDmap information, INS information, and perceived lane line information, and time synchronization processing is performed on the vehicle motion information, IMU information, INS information, and perceived lane line information.

[0009] First observation information of lane lines is constructed using INS information and HDmap information. Second observation information of lane line fitting is obtained by extracting lane lines by image recognition of the perceived lane line information.

[0010] The perceived lane line information is input into a Kalman filter, and lane line tracking and prediction are performed based on the IMU information. The lane line parameters are then optimized and smoothed by combining the vehicle motion information to obtain prediction information.

[0011] During the lane line tracking and prediction process, the prediction information is verified based on the first observation information and the second observation information, and then fed back to the Kalman filter for iterative updates.

[0012] This basic scheme sets up an algorithm framework model based on the classic Kalman filter. It utilizes vehicle motion information, IMU information, HDmap information, INS information, and perceived lane line information to specifically optimize lane line accuracy, lane line detection stability, and robustness, while also detecting abnormal lane line information. The main model is based on adaptive Kalman filtering combined with vehicle motion information to predict lane line information, combined with high-precision map lane lines, and perceived lane line information to achieve observation update compensation. Among them, the use of observation and prediction residual information to achieve anomaly detection can provide a stable and reliable guarantee for autonomous vehicles traveling on public roads.

[0013] In a further implementation, the construction of the first lane line observation information using INS information and HDmap information includes:

[0014] The INS information is transformed to obtain the vehicle's current location coordinates;

[0015] The lane information of the area where the vehicle is located is determined based on the HDmap information;

[0016] The lane information is matched with the positioning coordinates, and then the current driving lane of the vehicle is determined based on the binary search method, which is used as the target lane.

[0017] The lane data of the target lane is obtained, and it is determined whether the lane data is valid. If so, the lane line parameters are fitted to construct a vehicle driving road model, and the key parameters of the corresponding lane line equation are obtained as the first observation information.

[0018] This solution performs coordinate transformation on the INS information to obtain the vehicle's current location coordinates, and then determines the lane information of the area where the vehicle is located based on HDmap information. This combination of high-precision map and vehicle GPS positioning technology provides lane-level navigation information for driving, and the positioning of autonomous vehicles is highly accurate. This allows for the acquisition of correct observation data as the basis for filter correction, thereby further optimizing the prediction accuracy of the filter.

[0019] In a further implementation, the step of performing coordinate system transformation on the INS information to obtain the vehicle's current positioning coordinates includes:

[0020] Obtain the user's GPS location coordinates based on INS information;

[0021] Based on the general coordinate transformation formula, the GPS positioning coordinates are converted into the first coordinates in the geocentric coordinate system;

[0022] Based on the universal coordinate transformation formula, the first coordinate is transformed from the geocentric coordinate system to the projected coordinate system, and the UTM coordinate is obtained as the current positioning coordinate of the vehicle.

[0023] In a further embodiment, lane line extraction is performed using image recognition of the perceived lane line information to obtain second observation information for lane line fitting, including:

[0024] Collect raw images of different roads, generate an image dataset, and divide it into training and test sets;

[0025] Construct a deep learning network model, input the training set and test set into the deep learning network model for optimization training, and obtain an optimized deep network learning model;

[0026] The perceived lane line information is input into the deep learning network model to obtain the feature map of the road image to be identified. The feature map is then fitted to obtain the lane line recognition result map, a vehicle driving road model is constructed, and the confidence score is output.

[0027] If the confidence level is greater than a preset threshold, then based on the corresponding vehicle driving road model, the key parameters of the corresponding lane line equation will be obtained as the second observation information.

[0028] This solution uses a deep learning model for image recognition to obtain second observation information. Since deep learning models can extract useful features from a large amount of data through learning, these features can help the model better adapt to different road environments and lighting conditions, improving the accuracy and robustness of detection. Thus, through its powerful learning and adaptability as well as its efficient processing speed, it has shown significant advantages in the field of lane detection, providing more reliable and real-time lane detection for autonomous driving and assisted driving systems.

[0029] In a further implementation, the step of inputting the perceived lane line information into a Kalman filter, performing lane line tracking and prediction based on the IMU information, and optimizing and smoothing the lane line parameters by combining the vehicle motion information to obtain prediction information includes:

[0030] Based on the universal coordinate system transformation formula, the perceived lane line information is converted into lane line data in the local coordinate system;

[0031] The lane line data is input into a Kalman filter, and the key parameters of the lane line equation for constructing a vehicle driving road model are output as the first prediction parameters.

[0032] A vehicle kinematic model is constructed based on the vehicle motion information and the IMU information;

[0033] The state transition matrix and control matrix of the Kalman filter are updated based on the vehicle kinematics model.

[0034] Based on the state transition matrix and control matrix, the lane line parameters are optimized and smoothed according to the first prediction parameter, and the current lane line information is predicted to obtain the prediction information.

[0035] In a further implementation, the system model of the Kalman filter is as follows:

[0036]

[0037] In the formula, , Let be the state variables at time t and time t-1, respectively. Let A be the control quantity at time t-1, and B be the state transition matrix and the control matrix.

[0038] The state transition matrix is ​​as follows:

[0039]

[0040] The control matrix is ​​as follows:

[0041]

[0042] In the formula, The distance the vehicle traveled. Let t be the yaw rate of the vehicle, and t be the time.

[0043] This solution utilizes perceived lane line information, IMU information, and vehicle motion information to construct the system equations of the filter. Based on the current motion trend of the vehicle, it achieves short-term prediction of lane lines, thereby effectively identifying abnormal lane lines and improving driving safety.

[0044] In a further embodiment, the observation verification includes:

[0045] The observation noise is estimated based on an adaptive filtering model, which is as follows:

[0046]

[0047]

[0048]

[0049] In the formula, k is the current frame, and k - 1 is the previous frame. is the observation matrix. is the covariance prediction matrix of the corresponding parameters. is the predicted value of the lane line parameters. is the residual between the predicted value and the observed value. is the confidence level, R

[0052] ,

[0051] , , , ,

[0057] ,

[0054] ,

[0056] , ,

[0055] , , , , , , ,

[0053] , , , , The covariance matrix of the observation noise. is the matrix R k The i-th value in is the matrix The i-th value in is the observed value at time k;

[0050] , where 0 < b < 1, is called the fading factor, and according to experience, b = 0.9 - 0.999.

[0051] In a further embodiment, in the observation verification, the observation model is as follows;

[0052]

[0053]

[0054] In the formula, represents the perceived lane line data. represents the lane line data of the high-precision map; Observation matrix. is the variance matrix of the observation state. <000015​​​​​​​The present invention also provides an electronic device, comprising: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the steps of the lane optimization method described above. Attached Figure Description

[0058] Figure 1 This is a flowchart of a lane line optimization method provided in an embodiment of the present invention;

[0059] Figure 2 This is a flowchart of a lane line optimization method provided in an embodiment of the present invention. Detailed Implementation

[0060] The embodiments of the present invention are described in detail below with reference to the accompanying drawings. The embodiments are given for illustrative purposes only and should not be construed as limiting the present invention. The accompanying drawings are for reference and illustration only and do not constitute a limitation on the scope of patent protection of the present invention, because many changes can be made to the present invention without departing from the spirit and scope of the present invention.

[0061] This invention provides a lane line optimization method, such as... Figure 1 , Figure 2 As shown, in this embodiment, steps S1 to S4 are included:

[0062] S1. Perform data collection, acquire vehicle motion information, IMU information, HDmap information, INS information and perceived lane line information, and perform time synchronization processing on the vehicle motion information, IMU information, INS information and perceived lane line information.

[0063] The vehicle motion information includes the distance the vehicle traveled and the vehicle's yaw rate.

[0064] In this embodiment, the time synchronization process specifically involves: reconstructing the vehicle motion information, IMU information, INS information, and perceived lane line information based on the Newton interpolation algorithm, and aligning the time of the data in the vehicle motion information, IMU information, INS information, and perceived lane line information to a reference.

[0065] In this embodiment, the road model for vehicle travel is as follows:

[0066]

[0067] The key parameters of the classic lane line equation are (c0, c1, c2, c3), where c0 represents the lateral distance between the vehicle and the lane line, c1 represents the lane line heading angle, c2 represents the lane line curvature, and c3 represents the rate of change of curvature.

[0068] S2. Construct the first observation information of the lane line using INS information and HDmap information, and extract the lane line by image recognition of the perceived lane line information to obtain the second observation information of the lane line fitting.

[0069] In this embodiment, the first observation information of the lane line is constructed using INS information and HDmap information, including:

[0070] S21A. Perform coordinate system transformation on the INS information to obtain the current positioning coordinates of the vehicle, including:

[0071] S211A. Obtain the user's GPS location coordinates based on the INS information;

[0072] S212A. Based on the general coordinate transformation formula, the GPS positioning coordinates are converted into the first coordinates in the geocentric coordinate system, for example, the GCJ02 National Bureau of Surveying and Mapping coordinate system.

[0073] S212A. Based on the universal coordinate transformation formula, the first coordinate is transformed from the geocentric coordinate system to the projected coordinate system to obtain the UTM coordinate as the current positioning coordinate of the vehicle.

[0074] S22A. Determine the lane information of the area where the vehicle is located based on the HDmap information;

[0075] S23A. Match the lane information according to the positioning coordinates, and then determine the current driving lane of the vehicle based on the binary search method, and use it as the target lane.

[0076] S24A. Obtain the lane data of the target lane, determine whether the lane data is valid, and if so, perform lane line parameter fitting to construct a vehicle driving road model, and obtain the corresponding lane line equation key parameters as the first observation information.

[0077] The above-mentioned general coordinate transformation formula is common knowledge in this field, and will not be repeated in this embodiment.

[0078] Specifically, the method for obtaining the lane data of the target lane is to provide least squares fitting data points to obtain the lane line data of the corresponding target lane as lane data. The lane line data includes the lane width and the lane line UTM coordinate points.

[0079] This embodiment performs coordinate system transformation on the INS information to obtain the vehicle's current location coordinates, and then determines the lane information of the area where the vehicle is located based on the HDmap information. In this way, the combination of high-precision map and vehicle GPS positioning technology provides lane-level navigation information for driving, and the positioning of autonomous vehicles is highly accurate. Thus, the correct observation data is obtained as the basis for filter correction, thereby further optimizing the prediction accuracy of the filter.

[0080] In this embodiment, lane line extraction is performed by recognizing the perceived lane line information through image recognition to obtain second observation information for lane line fitting, including:

[0081] S21B: Collect original images of different roads, generate an image dataset, and divide it into training and test sets;

[0082] S22B: Construct a deep learning network model, input the training set and test set into the deep learning network model for optimization training, and obtain an optimized deep network learning model.

[0083] S23B. Input the perceived lane line information into the deep learning network model to obtain the feature map of the road image to be identified, and fit the feature map to obtain the lane line recognition result map. Construct a vehicle driving road model and output the confidence score.

[0084] S24B. Determine whether the confidence level is greater than a preset threshold. If so, obtain the key parameters of the corresponding lane line equation as the second observation information based on the corresponding vehicle driving road model.

[0085] This embodiment uses a deep learning model for image recognition to obtain second observation information. Since deep learning models can extract useful features from a large amount of data through learning, these features can help the model better adapt to different road environments and lighting conditions, improve the accuracy and robustness of detection, and thus show significant advantages in the field of lane detection through its powerful learning and adaptability and efficient processing speed, providing more reliable and real-time lane detection for autonomous driving and assisted driving systems.

[0086] S3. Input the perceived lane line information into a Kalman filter, perform lane line tracking and prediction based on the IMU information, and optimize and smooth the lane line parameters by combining the vehicle motion information to obtain prediction information, including:

[0087] S31. Based on the universal coordinate system transformation formula, the perceived lane line information is converted into lane line data in the local coordinate system (e.g., Northeast-Sky Cartesian coordinate system, ENU coordinate system).

[0088] S32. Input the lane line data into a Kalman filter and output the key parameters of the lane line equation for constructing a vehicle driving road model as the first prediction parameter.

[0089] S33. Construct a vehicle kinematics model based on the vehicle motion information and the IMU information;

[0090] S34. Update the state transition matrix and control matrix of the Kalman filter based on the vehicle kinematics model;

[0091] In this embodiment, the classic Kalman filter model is as follows:

[0092]

[0093]

[0094]

[0095]

[0096]

[0097] In the formula, This represents the previous state estimate. Represents the system transition matrix. Represents the predicted value. The covariance matrix of the corresponding parameters, Represents Kalman gain, Let I be the observation matrix, and R be the identity matrix. k The covariance matrix of the observation noise.

[0098] The system model of the Kalman filter is as follows:

[0099]

[0100] In the formula, , Let be the state variables at time t and time t-1, respectively. Let A be the control quantity at time t-1, and B be the state transition matrix and the control matrix.

[0101] The state variable X is as follows:

[0102]

[0103]

[0104] The state transition matrix is ​​as follows:

[0105]

[0106] The control matrix is ​​as follows:

[0107]

[0108]

[0109] In the formula, For the vehicle's speed, The distance the vehicle traveled. Let t be the yaw rate of the vehicle, and t be the time.

[0110] S35. Based on the state transition matrix and control matrix, update the lane line parameters according to the first prediction parameters, optimize and smooth them, and predict the current lane line information to obtain prediction information.

[0111] This embodiment utilizes perceived lane line information, IMU information, and vehicle motion information to construct the system equation of the filter. Based on the current motion trend of the vehicle, it achieves short-term prediction of lane lines, thereby effectively identifying abnormal lane lines and improving driving safety.

[0112] S4. During the lane line tracking and prediction process, the prediction information is verified based on the first observation information and the second observation information, and then fed back to the Kalman filter for iterative updates.

[0113] In this embodiment, the observation model for observation verification is as follows;

[0114]

[0115]

[0116] In the formula, Represents perceived lane line data, Lane line data representing high-precision maps; Observation matrix Let be the variance matrix of the observed state.

[0117] In this embodiment, the observation verification includes:

[0118] The observation noise is estimated based on an adaptive filtering model, which is as follows:

[0119]

[0120]

[0121]

[0122] Where \(k\) is the current frame and \(k - 1\) is the previous frame. is the observation matrix. is the covariance prediction matrix of the corresponding parameters. is the predicted value of the lane line parameters. is the residual between the predicted value and the observed value (i.e., the difference between the predicted lane line parameters and the lane line parameters obtained by fitting, which is the perceived lane line parameters). is the confidence level, \(R\) k is the covariance matrix of the observation noise. is the \(i\)-th value in the matrix \(R\) k and is the \(i\)-th value in the matrix . is the observed value at time \(k\).

[0123] In the formula, where \(0 < b < 1\), called the fading factor, and according to experience \(b = 0.9\sim0.999\).

[0124] where the process noise \(w\) k and the observation noise \(v\) k covariance matrices \(Q\) k and \(R\) k respectively represent the degree of trust in the model and the observation, and their values indirectly affect the Kalman gain \(K\) k , so they have a greater impact on the accuracy of the filtering result; \(R\) k can be obtained from the accuracy parameters of the sensor, while \(Q\) k is usually set according to experience.

[0125] In this embodiment, the perception lane line information, the lane line information of the HDmap, and the additional constrained lane line width information are introduced into the observation model of the filter to achieve observation update compensation, perform lane line parameter optimization and smoothing processing, and combine the actual observation data for prediction to break away from the distance limitation, give full play to the over-the-horizon perception ability of the HDmap to improve the lane line accuracy, and the autonomous driving system can obtain higher positioning accuracy and stronger environmental perception ability, and further improve the lane line prediction ability and the lane line information anomaly detection performance.

[0126] In this embodiment, the steps of measurement update are as follows:

[0127] Based on the covariance matrices of the measurement noise and the process noise, calculate a weight factor, that is, the Kalman gain.

[0128] Use the calculated Kalman gain and the new measurement value to correct the previous state estimate to obtain the updated state estimate value.

[0129] ​​​​​​​​​Based on the updated state estimate and new measurement information, the covariance matrix of the state estimate is adjusted to reflect the new level of uncertainty.

[0130] This embodiment proposes a robust adaptive Kalman filter algorithm for lane line anomaly detection. The algorithm performs a one-step prediction of the lane line state based on prior lane line information and vehicle motion information, and then compares the predicted information with the current observation information for current observation verification. For scenarios such as turning intersections and lane line disappearance, the algorithm uses vehicle motion information to predict lane line information.

[0131] This invention establishes an algorithm framework model based on classic Kalman filtering. It utilizes vehicle motion information, IMU information, HDmap information, INS information, and perceived lane line information to specifically optimize lane line accuracy, lane line detection stability, and robustness, while also detecting abnormal lane line information. The main model is based on adaptive Kalman filtering combined with vehicle motion information to predict lane line information, combined with high-precision map lane lines, and perceived lane line information to achieve observation update compensation. Anomaly detection is achieved using observation and prediction residual information, providing a stable and reliable guarantee for the operation of autonomous vehicles traveling on public roads.

[0132] Example 2

[0133] This invention also provides an electronic device, including: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the steps of a lane line optimization method as described in Embodiment 1 above.

[0134] The above embodiments are preferred embodiments of the present invention, but the embodiments of the present invention are not limited to the above embodiments. Any changes, modifications, substitutions, combinations, or simplifications made without departing from the spirit and principle of the present invention shall be considered equivalent substitutions and shall be included within the protection scope of the present invention.

Claims

1. A lane line optimization method, characterized by, Including the following steps: Data collection is performed to obtain vehicle motion information, IMU information, HDmap information, INS information, and perceived lane line information, and time synchronization processing is performed on the vehicle motion information, IMU information, INS information, and perceived lane line information. First observation information of lane lines is constructed using INS information and HDmap information, and second observation information of lane line fitting is obtained by extracting lane lines through image recognition of the perceived lane line information. The perceived lane line information is input into a Kalman filter, and lane line tracking and prediction are performed based on the IMU information. The lane line parameters are then optimized and smoothed by combining the vehicle motion information to obtain prediction information. During the lane line tracking and prediction process, the prediction information is verified based on the first observation information and the second observation information, and then fed back to the Kalman filter for iterative updates. The perceived lane line information is input into a Kalman filter, and lane line tracking and prediction are performed based on the IMU information. The lane line parameters are then optimized and smoothed by incorporating the vehicle motion information to obtain prediction information, including: Based on the universal coordinate system transformation formula, the perceived lane line information is converted into lane line data in the local coordinate system; The lane line data is input into a Kalman filter, and the key parameters of the lane line equation for constructing a vehicle driving road model are output as the first prediction parameters. A vehicle kinematic model is constructed based on the vehicle motion information and the IMU information; The state transition matrix and control matrix of the Kalman filter are updated based on the vehicle kinematics model. The first prediction parameters are updated based on the state transition matrix and the control matrix, the lane line parameters are optimized and smoothed, and the current lane line information is predicted to obtain prediction information.

2. The lane line optimization method of claim 1, wherein, The construction of the first observation information for lane lines using INS information and HDmap information includes: The INS information is transformed to obtain the vehicle's current location coordinates; The lane information of the area where the vehicle is located is determined based on the HDmap information; The lane information is matched with the positioning coordinates, and then the current driving lane of the vehicle is determined based on the binary search method, which is used as the target lane. The lane data of the target lane is obtained, and it is determined whether the lane data is valid. If so, the lane line parameters are fitted to construct a vehicle driving road model, and the key parameters of the corresponding lane line equation are obtained as the first observation information.

3. The lane line optimization method of claim 2, wherein, The step of performing coordinate system transformation on the INS information to obtain the vehicle's current location coordinates includes: The user's GPS location coordinates are obtained based on the INS information; Based on the general coordinate transformation formula, the GPS positioning coordinates are converted into the first coordinates in the geocentric coordinate system; Based on the universal coordinate transformation formula, the first coordinate is transformed from the geocentric coordinate system to the projected coordinate system, and the UTM coordinate is obtained as the current positioning coordinate of the vehicle.

4. The lane line optimization method of claim 1, wherein, Lane line extraction is performed using image recognition of the perceived lane line information to obtain second observation information for lane line fitting, including: Collect raw images of different roads, generate an image dataset, and divide it into training and test sets; Construct a deep learning network model, input the training set and test set into the deep learning network model for optimization training, and obtain an optimized deep network learning model; The perceived lane line information is input into the deep learning network model to obtain the feature map of the road image to be identified. The feature map is then fitted to obtain the lane line recognition result map. A vehicle driving road model is constructed, and the confidence score is output. If the confidence level is greater than a preset threshold, then based on the corresponding vehicle driving road model, the key parameters of the corresponding lane line equation will be obtained as the second observation information.

5. The lane line optimization method as described in claim 4, characterized in that: The system model of the Kalman filter is as follows: In the formula, , are state variables at time t and time t-1, respectively, is a control variable at time t-1, A is a state transition matrix, and B is a control matrix. The state transition matrix is ​​as follows: The control matrix is ​​as follows: wherein is the vehicle motion distance, is the vehicle yaw rate, t is time.

6. The lane line optimization method of claim 1, wherein, The observation verification includes: The observation noise is estimated based on an adaptive filtering model, which is as follows: In the formula, k is the current frame, and k-1 is the previous frame. For the observation matrix, This is the covariance prediction matrix for the corresponding parameters. These are the predicted values ​​for lane line parameters. The residual between the predicted and observed values, For confidence level, R k The covariance matrix of the observation noise, Let matrix R k The i-th value in For matrix The i-th value in Let be the observation value at time k; wherein wherein 0 < b < 1, b is called the evanescent factor.

7. The lane line optimization method of claim 6, wherein: In the observation verification, the following applies; In the formula, Represents perceived lane line data, Lane line data representing high-precision maps; Observation matrix Let be the variance matrix of the observed state.

8. The lane line optimization method of claim 1, wherein, The time synchronization process specifically involves: reconstructing vehicle motion information, IMU information, INS information, and perceived lane line information based on the Newton interpolation algorithm, and aligning the time of the data in the vehicle motion information, IMU information, INS information, and perceived lane line information to a reference.

9. An electronic device, comprising: include: The memory, the processor, and the computer program stored in the memory and executable on the processor, wherein the processor, when executing the computer program, implements the steps of a lane optimization method as described in any one of claims 1-8.

Images