Lane tracking method, electronic device, storage medium and vehicle

Abstract

The present application discloses a lane line tracking method, electronic device, storage medium and vehicle, and belongs to the field of lane line detection technology. The method includes: preprocessing the input image, detecting and screening the lane line hotspot area in the input image through probabilistic Hough transform; inputting the lane line hotspot area and the input image into the lane line detection convolutional neural network to obtain the lane line coordinate points; using the quadratic curve to perform the least squares fitting on the lane line coordinate points to obtain the fitted lane line; inputting the coefficients in the fitted lane line as the system state variables into the Kalman network model for tracking prediction, outputting the final tracked lane line structure and drawing it in the input image. The present application reduces the amount of algorithm model calculation, improves the accuracy of lane line detection and tracking, and can still achieve effective detection and tracking of lane lines in complex scenarios, thereby improving the robustness of lane line detection and tracking.

Description

Technical Field

[0001] The present application belongs to the technical field of lane line detection, and in particular, relates to a lane line tracking method, electronic equipment, storage medium and vehicle. Background Art

[0002] The existing lane line recognition algorithms are mainly divided into two methods: traditional image processing and deep learning. Traditional image processing methods are difficult to meet the detection requirements in complex working conditions (night, rainy days, crowded, etc.), with high false detection and missed detection rates, and the robustness of the algorithm needs to be improved; while the anchor-based deep learning method has higher robustness than the traditional image processing method, but a large number of candidate frames will be generated before the network training begins, which will lead to an increase in the overall algorithm calculation amount, making the algorithm's computing power usage increase, and unable to meet the needs of real-time detection. Summary of the invention

[0003] The present application aims to solve at least one of the technical problems existing in the prior art. To this end, the present application proposes a lane line tracking method, an electronic device, a storage medium and a vehicle, which preprocesses the input image and locates the lane line hotspot area through probabilistic Hough transform, uses the lane line hotspot area and the input image as the input of the lane line detection convolutional neural network, and obtains the lane line through quadratic curve fitting. The lane line fitting coefficient is input into the Kalman network model to predict and track the lane line, and finally the tracking result is visualized in the input image, which reduces the amount of calculation of the algorithm model while improving the accuracy of lane line detection and tracking. In complex scenarios, it can still achieve effective detection and tracking of the lane line, and improve the robustness of lane line detection and tracking.

[0004] In a first aspect, the present application provides a lane line tracking method, the method comprising:

[0005] Preprocessing the input image, detecting and screening out lane line hot spots in the input image by using a probabilistic Hough transform method;

[0006] Using the lane line hotspot area and the input image as inputs of a trained lane line detection convolutional neural network to obtain detected lane line coordinate points;

[0007] Using a quadratic curve to perform least squares fitting on the detected lane line coordinate points to obtain a fitted lane line;

[0008] The coefficients in the fitted lane line are input into the trained Kalman network model as system state variables for tracking prediction, and the final tracking lane line structure is output and drawn in the input image.

[0009] In the above technical solution, the input image is preprocessed to enhance the lane line features, the hot spots of the lane lines are located by probabilistic Hough transform, the hot spots of the lane lines and the input image are used as the input of the lane line detection convolutional neural network to detect the lane line coordinates, and the lane lines are obtained by quadratic curve fitting. The lane line fitting coefficients are input into the Kalman network model to predict and track the lane lines, and the tracking results are finally visualized in the input image, which reduces the amount of calculation of the algorithm model while improving the accuracy of lane line detection and tracking. In complex scenarios, effective detection and tracking of lane lines can still be achieved, which improves the robustness of lane line detection and tracking.

[0010] According to one embodiment of the present application, the lane line detection convolutional neural network is trained by the following steps:

[0011] Obtain images containing lane lines and corresponding lane line labels, and divide them into training data sets and validation data sets according to a preset ratio;

[0012] Preprocessing each image in the training data set by grayscale transformation, color space feature enhancement and edge detection, and detecting and screening lane line hot spots in each image in the training data set by probabilistic Hough transform method;

[0013] Taking the lane line hotspot area in each image in the training data set as a candidate box, and inputting it into the lane line detection convolutional neural network together with the corresponding image, to obtain the prediction output of the lane line detection convolutional neural network;

[0014] A loss is calculated based on the predicted output and the lane line label, and parameters of the lane line detection convolutional neural network are optimized based on the loss.

[0015] In the above technical solution, an image containing lane lines and corresponding lane line labels are obtained and divided into a training set and a validation set. The training set images are preprocessed by grayscale transformation, color space feature enhancement and edge detection, and the hot spots of lane lines are screened out by probabilistic Hough transformation. The hot spots are input into the convolutional neural network together with the image as candidate boxes for training, and the prediction results of lane lines are output. Then, the loss is calculated based on the prediction results and the true labels, and the lane line detection convolutional neural network is optimized. The hot spots of the lane line positions are obtained in advance through the traditional lane line detection algorithm, and then the lane detection convolutional neural network model based on the candidate boxes is trained accordingly, so that the lane detection convolutional neural network can reduce the excessive calculation amount generated by the candidate boxes during the training and detection process, thereby improving the accuracy of lane line detection and tracking.

[0016] According to an embodiment of the present application, the lane line hotspot area in each image in the training data set is used as a candidate box, and is input into the lane line detection convolutional neural network together with the corresponding image to obtain the prediction output of the lane line detection convolutional neural network, including:

[0017] Loading the corresponding image into the backbone network, and obtaining an image feature map through convolution downsampling operation;

[0018] Generate a candidate box based on the lane line hotspot area in each image in the training data set;

[0019] Input the candidate box into a feature pooling layer based on Hough anchor points, and perform a feature layer reduction operation based on the candidate box;

[0020] The output of the feature pooling layer based on the Hough anchor point is subjected to attention calculation through the fully connected layer and the global image layer to obtain the attention feature;

[0021] Input the attention features into two fully connected layers to obtain classification prediction values ​​and regression prediction values;

[0022] The candidate boxes of predicted values ​​are screened by the non-maximum suppression method to obtain the optimal predicted value.

[0023] In the above technical solution, the lane line hotspot area in each image is used as a candidate box and input into the lane line detection convolutional neural network together with the corresponding image. The image features are extracted through the backbone network, and the candidate box features are reduced by using the feature pooling layer based on Hough anchor points. The global information is obtained through the attention mechanism, and the classification and regression prediction are obtained through the fully connected layer. Finally, the optimal prediction value is screened through non-maximum suppression, which reduces the excessive calculation amount generated by the candidate box and improves the accuracy of lane line detection and tracking.

[0024] According to an embodiment of the present application, the calculating the loss based on the predicted output and the lane line label, and optimizing the parameters of the lane line detection convolutional neural network based on the loss, includes:

[0025] The optimal prediction value and the lane line label are input into a loss function to calculate the loss, and the parameters of the lane line detection convolutional neural network are optimized based on the loss and the optimizer, wherein the loss function is:

[0026]

[0027] L is the loss, is the classification loss function, is the regression loss function, is a balance parameter.

[0028] In the above technical solution, the loss is calculated by inputting the optimal prediction value of the lane line detection convolutional neural network and the lane line label into the loss function, and the parameters of the lane line detection convolutional neural network are optimized based on the loss and the optimizer to improve the performance of the lane line detection convolutional neural network, thereby improving the accuracy of lane line detection and tracking.

[0029] According to one embodiment of the present application, the Kalman network model is used to predict and calculate the Kalman gain in the Kalman filter through the GRU gated recurrent unit in the recurrent neural network.

[0030] In the above technical solution, the Kalman gain in the Kalman filter is predicted and calculated through the GRU gated recurrent unit in the recurrent neural network, so as to enhance the matching degree between the Kalman filter and the lane line model, reduce the tracking disturbance caused by the model mismatch in lane line detection and tracking, and thus improve the accuracy of lane line detection and tracking.

[0031] According to one embodiment of the present application, the Kalman network model is trained by the following steps:

[0032] Establish the lane line system state equation;

[0033] By inputting random time into the lane line system state equation, data parameters in the system evolution process are obtained, thereby obtaining lane line tracking data;

[0034] The lane line tracking data and the lane line system state equation are loaded into a convolutional recurrent neural network for training. After the training is completed, the Kalman network model is obtained.

[0035] In the above technical solution, a lane line system state equation is established, and lane line tracking data is obtained by inputting random time quantities into the lane line system state equation, and then the lane line system state equation and the lane line tracking data are loaded into a convolutional recurrent neural network for training to obtain a Kalman network model, and then tracking prediction is performed through the Kalman network model to reduce tracking disturbances caused by model mismatch in lane line detection and tracking, so that lane line detection and tracking can still be achieved in complex scenarios, thereby improving the robustness of the lane line tracking method.

[0036] According to an embodiment of the present application, the coefficients in the fitted lane line are input as system state variables into a trained Kalman network model for tracking prediction, and the final tracking lane line structure is output, including:

[0037] The a, b, c parameters of the fitted lane line quadratic curve are input as system state variables into the trained Kalman network model for tracking prediction, and the optimal a, b, c parameters of the tracking prediction and the lane line tracking result curve equation are obtained.

[0038] In the above technical solution, the a, b, c parameters of the fitted lane line quadratic curve are input as system state variables into the trained Kalman network model, and the Kalman network model is used to track and predict the fitted lane line quadratic curve, and the optimal a, b, c parameters and the lane line tracking result curve equation are output, so that the lane line can still be detected and tracked in complex scenarios, thereby improving the robustness of the lane line tracking method.

[0039] In a second aspect, the present application provides an electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the lane tracking method as described in the first aspect above when executing the computer program.

[0040] In a third aspect, the present application provides a non-transitory computer-readable storage medium having a computer program stored thereon, and when the computer program is executed by a processor, the lane line tracking method as described in the first aspect above is implemented.

[0041] In a fourth aspect, the present application provides a vehicle, comprising: the electronic device as described in the second aspect.

[0042] Additional aspects and advantages of the present application will be given in part in the description below, and in part will become apparent from the description below, or will be learned through the practice of the present application. BRIEF DESCRIPTION OF THE DRAWINGS

[0043] The above and / or additional aspects and advantages of the present application will become apparent and easily understood from the description of the embodiments in conjunction with the following drawings, in which:

[0044] Figure 1 is a flow chart of a lane line tracking method provided in an embodiment of the present application;

[0045] Figure 2 It is a schematic diagram of the structure of the lane detection convolutional neural network provided in an embodiment of the present application;

[0046] Figure 3 A schematic diagram of the structure of a GRU gated recurrent unit in a Kalman network provided in an embodiment of the present application;

[0047] Figure 4 is a schematic diagram of the Kalman network model structure provided in an embodiment of the present application;

[0048] Figure 5 is a schematic diagram of the structure of a lane tracking device provided in an embodiment of the present application;

[0049] Figure 6 is a schematic diagram of the structure of an electronic device provided in an embodiment of the present application;

[0050] Reference numerals:

[0051] 500: Lane tracking device;

[0052] 501: pre-processing unit; 502: lane detection unit;

[0053] 503: Lane line fitting unit; 504: Lane line tracking unit;

[0054] 600: Electronic equipment;

[0055] 601: processor; 602: memory. DETAILED DESCRIPTION

[0056] The following will be combined with the drawings in the embodiments of the present application to clearly describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, rather than all the embodiments. All other embodiments obtained by ordinary technicians in this field based on the embodiments in the present application belong to the scope of protection of this application.

[0057] The terms "first", "second", etc. in the specification and claims of the present application are used to distinguish similar objects, and are not used to describe a specific order or sequence. It should be understood that the data used in this way can be interchangeable under appropriate circumstances, so that the embodiments of the present application can be implemented in an order other than those illustrated or described here, and the objects distinguished by "first", "second", etc. are generally of one type, and the number of objects is not limited. For example, the first object can be one or more. In addition, "and / or" in the specification and claims represents at least one of the connected objects, and the character " / " generally indicates that the objects associated with each other are in an "or" relationship.

[0058] The lane tracking method, electronic device, storage medium and vehicle provided in the embodiments of the present application are described in detail below with reference to the accompanying drawings through specific embodiments and their application scenarios.

[0059] The lane line tracking method provided in the embodiment of the present application may be executed by an electronic device or a functional module or functional entity in the electronic device that can implement the lane line tracking method. The electronic devices mentioned in the embodiment of the present application include but are not limited to controllers in vehicles, processors in vehicles, cloud servers, and background servers. The lane line tracking method provided in the embodiment of the present application is described below using the electronic device as an example of the execution subject.

[0060] Figure 1 is a flow chart of the lane line tracking method provided in the embodiment of the present application. Figure 1As shown, the lane line tracking method includes: step 110, step 120, step 130 and step 140.

[0061] Step 110 : pre-process the input image, and detect and filter out lane hotspot areas in the input image by using a probabilistic Hough transform method.

[0062] Preprocessing of the input image includes: grayscale transformation, color space feature enhancement and edge detection.

[0063] Grayscale transformation is a method in image processing that changes the grayscale value of each pixel in the source image point by point according to a certain target condition or transformation relationship. The purpose of this process is to improve the image quality, make the image display clearer, and enhance the image contrast and detail performance.

[0064] Color space enhancement refers to the process of converting an image from one color space to another, adjusting and optimizing its characteristics (such as hue, saturation, brightness, etc.) in the new color space, and finally converting the image back to the original color space or the desired target color space.

[0065] The purpose of edge detection algorithms is to identify places in an image where there are significant changes in brightness. These changes usually correspond to the boundaries of objects, such as lane lines. By detecting edges, the amount of data that needs to be processed in subsequent processing can be greatly reduced, and the focus can be placed on key structures in the image. For lane line detection, preprocessing with edge detection algorithms is more important because lane lines usually appear as straight or curved boundaries in an image.

[0066] The input image is preprocessed by grayscale transformation, color space feature enhancement and edge detection, with the aim of improving image quality, highlighting key features such as lane lines, and providing data for subsequent lane line detection. Then, the probabilistic Hough transform method is used to detect line segments in the preprocessed input image using the edge detection results. These line segments are considered candidates for lane lines and are screened to form lane line hotspots. The screening process may involve operations such as removing noise segments and merging adjacent line segments to improve the accuracy of lane line detection, detect and screen lane line hotspots from the input image, and provide input for subsequent lane line tracking, fitting or other processing steps.

[0067] Step 120: Use the lane line hotspot area and the input image as inputs to a trained lane line detection convolutional neural network to obtain detected lane line coordinate points.

[0068] It can be understood that the lane hotspot area is the area screened out from the input image through the preprocessing step and the probabilistic Hough transform. This area is considered to be the most likely location for the lane line and is therefore used as a hotspot for further processing. A trained lane detection convolutional neural network refers to a convolutional neural network model that has been trained with a large amount of labeled data and has the ability to accurately identify lane lines in an image.

[0069] Optionally, the lane line hotspot area is represented by at least one of the following information: lane line starting point, end point and horizontal angle information.

[0070] Step 130: Use a quadratic curve to perform least squares fitting on the detected lane line coordinate points to obtain a fitted lane line.

[0071] It should be noted that the lane detection convolutional neural network outputs a set of coordinate points representing the location of the lane lines. These coordinate points may contain noise or be inaccurate, so they need to be fitted to generate a smooth curve to represent the location and shape of the lane lines. A quadratic curve (such as a parabola, hyperbola, or part of an ellipse, but in lane detection it usually refers to a parabola or a similar single curve) is a suitable model that can better simulate the shape of the lane lines in the image. The least squares method is a mathematical optimization technique that obtains the best function match for the data by minimizing the sum of squared errors.

[0072] Step 140: Input the coefficients of the fitted lane line as system state variables into the trained Kalman network model for tracking prediction, output the final tracking lane line structure and draw the final tracking lane line structure in the input image.

[0073] It should be noted that the lane line coefficients obtained through fitting (such as the a, b, c parameters of the quadratic curve) represent the position and shape of the lane line in the current image frame. However, due to factors such as vehicle movement, road changes or image noise, the position and shape of the lane line may change between different image frames. Tracking and prediction through the Kalman network model can reduce the tracking disturbance caused by model mismatch in lane line detection and tracking, so that lane line detection and tracking can still be achieved in complex scenes. Therefore, the coefficients in the fitted lane line are input as system state variables into the trained Kalman network model for tracking prediction.

[0074] In the above technical solution, the input image is preprocessed to enhance the lane line features, the hot spots of the lane lines are located by probabilistic Hough transform, the hot spots of the lane lines and the input image are used as the input of the lane line detection convolutional neural network to detect the lane line coordinates, and the lane lines are obtained by quadratic curve fitting. The lane line fitting coefficients are input into the Kalman network model to predict and track the lane lines, and the tracking results are finally visualized in the input image, which reduces the amount of calculation of the algorithm model while improving the accuracy of lane line detection and tracking. In complex scenarios, effective detection and tracking of lane lines can still be achieved, which improves the robustness of lane line detection and tracking.

[0075] In one embodiment of the present application, the lane line detection convolutional neural network is trained by the following steps:

[0076] Obtain images containing lane lines and corresponding lane line labels, and divide them into training data sets and validation data sets according to a preset ratio;

[0077] Preprocessing each image in the training data set by grayscale transformation, color space feature enhancement and edge detection, and detecting and screening lane line hot spots in each image in the training data set by probabilistic Hough transform method;

[0078] Taking the lane line hotspot area in each image in the training data set as a candidate box, and inputting it into the lane line detection convolutional neural network together with the corresponding image, to obtain the prediction output of the lane line detection convolutional neural network;

[0079] A loss is calculated based on the predicted output and the lane line label, and parameters of the lane line detection convolutional neural network are optimized based on the loss.

[0080] It can be understood that lane line labels are manually annotated for each image to mark the precise position of the lane line, which usually involves drawing the bounding box of the lane line or using pixel-level annotation to define the precise outline of the lane line. The annotated dataset is divided into a training dataset and a validation dataset according to a preset ratio, which can be used to evaluate the performance of the model during training and prevent overfitting.

[0081] It can be understood that the lane hotspot area is the area screened out from the input image through the preprocessing steps and the probabilistic Hough transform. This area is considered to be the most likely location for the lane line to appear. Using this area as a candidate box can reduce the amount of calculation of the algorithm model while achieving effective detection and tracking of the lane line. Inputting the lane hotspot area as a candidate box together with the original image into the lane detection convolutional neural network helps the lane detection convolutional neural network focus on the area that is more likely to contain the lane line.

[0082] The lane detection convolutional neural network processes the input image and outputs the prediction result of the lane line, which may include information such as the location, shape or category of the lane line. The prediction output of the lane detection convolutional neural network is compared with the actual lane line label to calculate the loss value. The loss value measures the difference between the network prediction and the actual situation. Based on the calculated loss value, the parameters of the lane detection convolutional neural network are adjusted using an optimization algorithm (such as gradient descent) to reduce the loss value and improve the performance of the network.

[0083] In some embodiments, the training and validation data sets are divided according to a preset ratio of 7:3, and the lane line labels of the training data set are generated using Labelme software.

[0084] In some embodiments, filtering out lane hotspots in each image in the training data set includes:

[0085] Batch input image data, the batch size parameter Batchsize is 8, and the candidate boxes are generated in advance using traditional image processing methods;

[0086] Convert the RGB image into a YUV color space image, calculate the average value of the brightness Y, and determine the color space enhancement value gamma as 0.5. Multiply each RGB color space by gamma to perform color space enhancement. Perform a ROI mask operation on the enhanced image to extract the 2 / 5 effective lane line information area in the height direction of the image.

[0087] Use the Canny edge detection method to calculate the edge information of the captured image;

[0088] Finally, the lane lines are detected by the probabilistic Hough transform method using the cv2.HoughLinesP function. The Vote parameter is set to 30, the line segment length range is selected as [10, 200], and the horizontal angle of the detected straight line is compared with the threshold (the angle threshold is 22°). The horizontal angle of the left and right lane lines and the coordinates of the starting point and the end point are stored. The storage list length is set to 5, and each storage will be overwritten.

[0089] In the above technical solution, an image containing lane lines and corresponding lane line labels are obtained and divided into a training set and a validation set. The training set images are preprocessed by grayscale transformation, color space feature enhancement and edge detection, and the hot spots of lane lines are screened out by probabilistic Hough transformation. The hot spots are input into the convolutional neural network together with the image as candidate boxes for training, and the prediction results of lane lines are output. Then, the loss is calculated based on the prediction results and the true labels, and the lane line detection convolutional neural network is optimized. The hot spots of the lane line positions are obtained in advance through the traditional lane line detection algorithm, and then the lane detection convolutional neural network model based on the candidate boxes is trained accordingly, so that the lane detection convolutional neural network can reduce the excessive calculation amount generated by the candidate boxes during the training and detection process, thereby improving the accuracy of lane line detection and tracking.

[0090] In one embodiment of the present application, the lane line hotspot area in each image in the training data set is used as a candidate box, and is input into the lane line detection convolutional neural network together with the corresponding image to obtain the prediction output of the lane line detection convolutional neural network, including:

[0091] Loading the corresponding image into the backbone network, and obtaining an image feature map through convolution downsampling operation;

[0092] Generate a candidate box based on the lane line hotspot area in each image in the training data set;

[0093] Input the candidate box into a feature pooling layer based on Hough anchor points, and perform a feature layer reduction operation based on the candidate box;

[0094] The output of the feature pooling layer based on the Hough anchor point is subjected to attention calculation through the fully connected layer and the global image layer to obtain the attention feature;

[0095] Input the attention features into two fully connected layers to obtain classification prediction values ​​and regression prediction values;

[0096] The candidate boxes of predicted values ​​are screened by the non-maximum suppression method to obtain the optimal predicted value.

[0097] Figure 2 Schematic diagram of the structure of the lane detection convolutional neural network provided in the embodiment of the present application. Figure 2 As shown, the corresponding image is loaded into the backbone network (Backbone). Through the convolution layer and the downsampling layer, the backbone network can gradually reduce the spatial dimensions (i.e., width and height) of the image, and increase the depth (i.e., the number of channels) of the feature map, thereby extracting high-level features in the image and obtaining an image feature map.

[0098] Convolution downsampling is an image processing method. In a convolutional neural network, the convolution layer slides on the input feature map (image or output of the previous layer) through the convolution kernel (a small matrix or set of weights), performs local weighted summation (convolution operation), and may apply an activation function. By setting an appropriate stride (that is, the number of pixels that the convolution kernel slides on the feature map each time), the size of the output feature map can be controlled to achieve downsampling.

[0099] It is understandable that the generated candidate box is input into the Hough-Anchor-based Feature Pooling layer, which performs a pooling operation on the image feature map according to the position and size of the candidate box to extract feature information related to the candidate box. Since the candidate box is usually much smaller than the original image, this step can be regarded as a local feature layer reduction operation on the feature map, which helps to reduce the amount of calculation for subsequent processing and improve detection efficiency.

[0100] The output of the feature pooling layer based on the Hough anchor point is passed through the fully connected layer to obtain the current feature layer With the global image layer Perform attention calculation , where W is the weight matrix of the current feature layer, The purpose is to combine the features of the candidate box with the features of the global image to improve the accuracy of judging whether the candidate box contains lane lines. It is a commonly used function in machine learning and deep learning. When dealing with classification problems, it is usually used in the output layer of a multi-classification model. Through the attention mechanism, the lane detection convolutional neural network can distinguish the importance of different regions or features for lane detection, and thus give different weights when predicting. The attention features are input into two fully connected layers for classification prediction (Classification Loss) and regression prediction (Regression Loss), respectively. Classification prediction is used to determine whether the candidate box contains a lane line (i.e., a binary classification problem), while regression prediction is used to predict the exact position of the lane line (such as the coordinates of the bounding box). The predicted value candidate boxes are screened by the non-maximum suppression (NMS) method. Since there may be multiple candidate boxes in an image and some candidate boxes may overlap, non-maximum suppression NMS suppresses elements that are not maximum values ​​and searches for local maximum values. Therefore, the NMS method is needed to remove redundant candidate boxes and retain the best prediction results.

[0101] In some embodiments, the candidate box is 39-dimensional vector data, including lane line classification, confidence, length, starting point and 36 intermediate points. The lane line candidate box is generated based on the storage list. The number of candidate boxes is 10, and the number of left and right lane line candidate boxes is 5 respectively.

[0102] In the above technical solution, the lane line hotspot area in each image is used as a candidate box and input into the lane line detection convolutional neural network together with the corresponding image. The image features are extracted through the backbone network, and the candidate box features are reduced by using the feature pooling layer based on Hough anchor points. The global information is obtained through the attention mechanism, and the classification and regression prediction are obtained through the fully connected layer. Finally, the optimal prediction value is screened through non-maximum suppression, which reduces the excessive calculation amount generated by the candidate box and improves the accuracy of lane line detection and tracking.

[0103] In one embodiment of the present application, the calculating the loss based on the predicted output and the lane line label, and optimizing the parameters of the lane line detection convolutional neural network based on the loss, includes:

[0104] The optimal prediction value and the lane line label are input into a loss function to calculate the loss, and the parameters of the lane line detection convolutional neural network are optimized based on the loss and the optimizer, wherein the loss function is:

[0105]

[0106] L is the loss, is the classification loss function, is the regression loss function, is a balance parameter.

[0107] It can be understood that the optimal prediction value is the best estimate of the lane line position and existence in the input image by the lane line detection convolutional neural network. It includes the classification prediction value (determining whether the candidate box contains the lane line) and the regression prediction value (the exact position of the lane line, such as the coordinates of the bounding box).

[0108] The loss function is used to quantify the difference between the predicted value and the true value of the lane line label. For the calculated loss L, an optimizer (such as stochastic gradient descent SGD, adaptive momentum estimation Adam, etc.) can be used for optimization iteration, that is, to update the parameters of the lane line detection convolutional neural network. The optimizer guides the adjustment direction of the parameters according to the gradient of the loss function (that is, the derivative of the loss with respect to the network parameters) to minimize the loss.

[0109] In the above technical solution, the loss is calculated by inputting the optimal prediction value of the lane line detection convolutional neural network and the lane line label into the loss function, and the parameters of the lane line detection convolutional neural network are optimized based on the loss and the optimizer to improve the performance of the lane line detection convolutional neural network, thereby improving the accuracy of lane line detection and tracking.

[0110] In one embodiment of the present application, the Kalman network model is used to predict and calculate the Kalman gain in the Kalman filter through the GRU gated recurrent unit in the recurrent neural network.

[0111] GRU (Gated Recurrent Unit) is a recurrent neural network architecture that effectively manages the flow of information in the sequence processing process by introducing two gating mechanisms, update gate and reset gate, so that it can capture long-distance dependencies while maintaining high computational efficiency and low model complexity. It is suitable for tasks that need to process sequence data, such as natural language processing and time series analysis.

[0112] It should be noted that the traditional Kalman filter relies on accurate model parameters when estimating the system state, such as the system noise covariance Q and the observation noise covariance R, as well as the initial estimation error covariance P. However, in practical applications, these parameters are often difficult to obtain accurately, resulting in limited filtering effects.

[0113] The Kalman network model combines the Kalman filter with the GRU gated recurrent unit, and uses the GRU gated recurrent unit to dynamically predict and adjust the Kalman gain K in the Kalman filter, thereby improving the tracking effect of the lane line position. This combination takes advantage of the RNN's advantage in processing sequence data, that is, it can capture historical information and predict future states based on it.

[0114] Figure 3 Schematic diagram of the structure of the GRU gated recurrent unit in the Kalman network provided in the embodiment of the present application. Figure 3 As shown, in some embodiments, there are three GRU gated recurrent unit structures in the RNN network structure, which respectively calculate and estimate the system noise covariance Q and the observation noise covariance R, as well as the initial estimation error covariance P, so as to obtain the optimal Kalman gain, and the loss function in the Kalman network is set to the square error loss:

[0115]

[0116] in, represents the true value (or target value) of the state at time t, Represents the predicted value of the posterior state estimate at time t (also called the optimal estimate), Loss represents the loss value, which is the output of the squared error loss function, indicating the degree of difference between the predicted value and the true value.

[0117] The squared error loss is one of the commonly used loss functions in regression tasks. The squared error loss calculates the average of the square of the difference between the predicted value and the true value. It measures the accuracy of the predicted value. By minimizing the squared error loss, the Kalman network can improve its prediction accuracy for the Kalman gain in the Kalman filter, thereby improving the accuracy of lane tracking.

[0118] It should be noted that the fully convolutional network (FCN) is a special type of convolutional neural network (CNN). In traditional CNN, the fully connected layer is usually located at the end of the network to convert the feature map into the final classification result. In FCN, the traditional fully connected layer is replaced by the transposed convolutional layer. The transposed convolutional layer can be regarded as the inverse process of the convolutional layer, which can upsample the dimension of the feature map back to the size of the original image, so that FCN can process input images of any size and output segmentation maps of the same size or close to the input size.

[0119] In the above technical solution, the Kalman gain in the Kalman filter is predicted and calculated through the GRU gated recurrent unit in the recurrent neural network, so as to enhance the matching degree between the Kalman filter and the lane line model, reduce the tracking disturbance caused by the model mismatch in lane line detection and tracking, and thus improve the accuracy of lane line detection and tracking.

[0120] In one embodiment of the present application, the Kalman network model is trained by the following steps:

[0121] Establish the lane line system state equation;

[0122] By inputting random time into the lane line system state equation, data parameters in the system evolution process are obtained, thereby obtaining lane line tracking data;

[0123] The lane line tracking data and the lane line system state equation are loaded into a convolutional recurrent neural network for training. After the training is completed, the Kalman network model is obtained.

[0124] It should be noted that the lane line system state equation is a mathematical model used to describe the law of changes in the position, shape or other related characteristics of the lane line over time. This equation is usually based on physical principles or empirical models, and links the current state of the lane line (such as position, slope, curvature, etc.) with the state at the previous moment and possible external factors (such as vehicle movement, road changes, etc.).

[0125] It should be noted that random time is an uncertainty factor introduced when simulating or predicting the state change of the lane system. In actual situations, the change of lanes is often affected by many random factors that are difficult to accurately quantify, such as the driver's driving habits, the interaction between vehicles, the roughness of the road surface, etc. In order to more realistically reflect these randomness, random time can be introduced into the lane system state equation.

[0126] In the process of loading lane tracking data and lane system state equations into the convolutional recurrent neural network for training, the introduction of random time can help the convolutional recurrent neural network learn the complexity and uncertainty of lane changes, thereby improving its robustness in practical applications. Through training, the convolutional recurrent neural network will learn how to extract features from noisy input data, improve the accuracy of lane state prediction, and obtain a Kalman network model.

[0127] In some embodiments, training the Kalman network model includes:

[0128] The lane line system state equation is established, and the tracking object is the lane line quadratic function fitting curve:

[0129]

[0130] Among them, y represents the vertical position of a point on the lane line in the image or actual space, x represents the horizontal position of a point on the lane line in the image or actual space, a is the quadratic term coefficient, which controls the opening direction and opening size of the quadratic function, b is the linear term coefficient, which together with x controls the tilt direction and degree of the quadratic function, and c is the constant term, which represents the y coordinate of the intersection of the quadratic function and the y axis.

[0131] Since the lane line changes between adjacent frames are small, the system state equation and parameter settings are as follows:

[0132] Prediction stage:

[0133] System prediction state equation:

[0134] System covariance: ;

[0135] System observation equation:

[0136] The state variable is ; is the prior estimator at time k; is the k-1 moment posterior estimator; the state transformation matrix: ; is the transposed matrix of the state transformation matrix; B is the control input matrix; is the control vector at time k;

[0137] Observation matrix: ; is the transposed matrix of the observation matrix; the system has no control output, so the control vector u=0; is the prior covariance matrix at time k; is the k-1 moment posterior covariance matrix; is the prior estimator of the observed data at time k; is the process noise; is the observation noise; is the prediction error covariance matrix, which is used to calculate the Kalman gain.

[0138] The state variable x Represent the parameters a, b, c of the quadratic function fitting curve of the left lane line; Represents the parameters a, b, c of the quadratic function fitting curve of the right lane line.

[0139] Update phase:

[0140] Update the posterior estimate of the system state:

[0141] Covariance update:

[0142] Kalman gain of traditional Kalman filter:

[0143] Predicted observations and observed differences:

[0144] is the k-time posterior estimator; is the prior estimator at time k; is the posterior covariance matrix at time k; is the prior covariance matrix at time k; is the Kalman gain matrix at time k; is the inverse matrix of the Kalman gain matrix at time k; is the transposed matrix of the observation matrix, is the prediction error covariance matrix; is the inverse matrix of the forecast error covariance matrix;

[0145] To observe the difference; is the observed value at time k; is the observed predicted value at time k.

[0146] By inputting the lane line system state equation with random time quantity, the Kalman gain in the evolution process is obtained , observation difference Difference with updated state variables - And other data parameters are used to generate and save lane line tracking data.

[0147] Figure 4 Schematic diagram of the Kalman network model structure provided by the embodiment of the present application. Figure 4 As shown, the Kalman gain in the Kalman filter is predicted and calculated by the GRU gated recurrent unit in the convolutional recurrent neural network, replacing the original Kalman gain calculation process to obtain the Kalman network model.

[0148] In the above technical solution, a lane line system state equation is established, and lane line tracking data is obtained by inputting random time quantities into the lane line system state equation, and then the lane line system state equation and the lane line tracking data are loaded into a convolutional recurrent neural network for training to obtain a Kalman network model, and then tracking prediction is performed through the Kalman network model to reduce tracking disturbances caused by model mismatch in lane line detection and tracking, so that lane line detection and tracking can still be achieved in complex scenarios, thereby improving the robustness of lane line detection and tracking.

[0149] In one embodiment of the present application, the coefficients in the fitted lane line are input as system state variables into a trained Kalman network model for tracking prediction, and the final tracking lane line structure is output, including:

[0150] The a, b, c parameters of the fitted lane line quadratic curve are input as system state variables into the trained Kalman network model for tracking prediction, and the optimal a, b, c parameters of the tracking prediction and the lane line tracking result curve equation are obtained.

[0151] It should be noted that the a, b, c parameters of the lane line quadratic curve obtained through fitting represent the position and shape of the lane line in the current image frame. However, due to factors such as vehicle movement, road changes or image noise, the position and shape of the lane line may change between different image frames, and the Kalman network can reduce the tracking disturbance caused by model mismatch in lane line detection and tracking, so that the lane line can still be detected and tracked in complex scenes. Therefore, the a, b, c parameters of the fitted lane line quadratic curve are input into the Kalman network to achieve continuous tracking and prediction of the lane line.

[0152] In the above technical solution, the a, b, c parameters of the fitted lane line quadratic curve are input into the trained Kalman network model as system state variables, and the Kalman network model is used to track and predict the fitted lane line quadratic curve, and the optimal a, b, c parameters and the lane line tracking result curve equation are output, so that the lane line can still be detected and tracked in complex scenarios, thereby improving the robustness of the lane line detection and tracking.

[0153] The lane line tracking method provided in the embodiment of the present application can be executed by a lane line tracking device. In the embodiment of the present application, the lane line tracking device executing the lane line tracking method is taken as an example to illustrate the lane line tracking device provided in the embodiment of the present application.

[0154] An embodiment of the present application provides a lane tracking device. Figure 5 is a schematic diagram of the structure of the lane tracking device provided in the embodiment of the present application, such as Figure 5 As shown, the lane line tracking device 500 includes:

[0155] A preprocessing unit 501 is used to preprocess the input image, and detect and filter out lane hotspot areas in the input image by using a probabilistic Hough transform method;

[0156] A lane line detection unit 502, used to use the lane line hotspot area and the input image as inputs of a trained convolutional neural network to obtain detected lane line coordinate points;

[0157] A lane line fitting unit 503 is used to perform least squares fitting on the detected lane line coordinate points using a quadratic curve to obtain a fitted lane line;

[0158] The lane line tracking unit 504 is used to input the coefficients of the fitted lane line as system state variables into the trained Kalman network model for tracking prediction, output the final tracked lane line structure and draw the final tracked lane line structure in the input image.

[0159] Optionally, the lane detection convolutional neural network is trained by the following steps:

[0160] Obtain images containing lane lines and corresponding lane line labels, and divide them into training data sets and validation data sets according to a preset ratio;

[0161] Preprocessing each image in the training data set by grayscale transformation, color space feature enhancement and edge detection, and detecting and screening lane line hot spots in each image in the training data set by probabilistic Hough transform method;

[0162] Taking the lane line hotspot area in each image in the training data set as a candidate box, and inputting it into the lane line detection convolutional neural network together with the corresponding image, to obtain the prediction output of the lane line detection convolutional neural network;

[0163] A loss is calculated based on the predicted output and the lane line label, and parameters of the lane line detection convolutional neural network are optimized based on the loss.

[0164] Optionally, taking the lane line hotspot area in each image in the training data set as a candidate box and inputting it into the lane line detection convolutional neural network together with the corresponding image to obtain the prediction output of the lane line detection convolutional neural network includes:

[0165] Loading the corresponding image into the backbone network, and obtaining an image feature map through convolution downsampling operation;

[0166] Generate a candidate box based on the lane line hotspot area in each image in the training data set;

[0167] Input the candidate box into a feature pooling layer based on Hough anchor points, and perform a feature layer reduction operation based on the candidate box;

[0168] The output of the feature pooling layer based on the Hough anchor point is subjected to attention calculation through the fully connected layer and the global image layer to obtain the attention feature;

[0169] Input the attention features into two fully connected layers to obtain classification prediction values ​​and regression prediction values;

[0170] The candidate boxes of predicted values ​​are screened by the non-maximum suppression method to obtain the optimal predicted value.

[0171] Optionally, the calculating a loss based on the predicted output and the lane line label, and optimizing parameters of the lane line detection convolutional neural network based on the loss, includes:

[0172] The optimal prediction value and the lane line label are input into a loss function to calculate the loss, and the parameters of the lane line detection convolutional neural network are optimized based on the loss and the optimizer, wherein the loss function is:

[0173]

[0174] L is the loss, is the classification loss function, is the regression loss function, is a balance parameter.

[0175] Optionally, the Kalman network model is used to predict and calculate the Kalman gain in the Kalman filter through a GRU gated recurrent unit in a recurrent neural network.

[0176] Optionally, the Kalman network model is trained by the following steps:

[0177] Establish the lane line system state equation;

[0178] By inputting random time into the lane line system state equation, data parameters in the system evolution process are obtained, thereby obtaining lane line tracking data;

[0179] The lane line tracking data and the lane line system state equation are loaded into a convolutional recurrent neural network for training. After the training is completed, the Kalman network model is obtained.

[0180] Optionally, the lane tracking unit 504 may be used to:

[0181] The a, b, c parameters of the fitted lane line quadratic curve are input as system state variables into the trained Kalman network model for tracking prediction, and the optimal a, b, c parameters of the tracking prediction and the lane line tracking result curve equation are obtained.

[0182] In the above technical solution, the lane line tracking device preprocesses the input image to enhance the lane line features, locates the lane line hotspot area through probabilistic Hough transform, uses the lane line hotspot area and the input image as the input of the lane line detection convolutional neural network to detect the lane line coordinates, and obtains the lane line through quadratic curve fitting. The lane line fitting coefficient is input into the Kalman network model to predict and track the lane line, and the tracking result is finally visualized in the input image, which reduces the amount of calculation of the algorithm model while improving the accuracy of lane line detection and tracking. In complex scenarios, effective detection and tracking of lane lines can still be achieved, which improves the robustness of lane line detection and tracking.

[0183] The lane tracking device in the embodiment of the present application can be an electronic device or a component in an electronic device, such as an integrated circuit or a chip. The electronic device can be a terminal or other devices other than a terminal. For example, the electronic device can be a controller in a vehicle, a processor in a vehicle, a cloud server, a backend server, etc., which is not specifically limited in the embodiment of the present application.

[0184] The lane tracking device in the embodiment of the present application may be a device having an operating system. The operating system may be a Microsoft (Windows) operating system, an Android (Android) operating system, an IOS operating system, or other possible operating systems, which are not specifically limited in the embodiment of the present application.

[0185] The lane line tracking device provided in the embodiment of the present application can implement each process implemented in the lane line tracking method embodiment, and can achieve the same technical effect. To avoid repetition, it will not be repeated here.

[0186] In some embodiments, Figure 6 As shown, an embodiment of the present application further provides an electronic device 600, including a processor 601, a memory 602, and a computer program stored in the memory 602 and executable on the processor 601. When the program is executed by the processor 601, each process of the above-mentioned lane line tracking method embodiment is implemented, and the same technical effect can be achieved. To avoid repetition, it will not be described here.

[0187] It should be noted that the electronic devices in the embodiments of the present application include the mobile electronic devices and non-mobile electronic devices mentioned above.

[0188] The embodiment of the present application also provides a non-transitory computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, the various processes of the above-mentioned lane line tracking method embodiment are implemented, and the same technical effect can be achieved. To avoid repetition, it will not be repeated here.

[0189] The processor is the processor in the electronic device described in the above embodiment. The readable storage medium includes a computer readable storage medium, such as a computer read-only memory ROM, a random access memory RAM, a magnetic disk or an optical disk.

[0190] The present application also provides a vehicle, which may include: Figure 5 The lane line tracking device shown may also include Figure 6 For the understanding of the electronic device shown in the figure, the embodiments of the present application can refer to the description in the foregoing embodiments, which will not be repeated here.

[0191] It should be noted that, in this article, the terms "comprise", "include" or any other variant thereof are intended to cover non-exclusive inclusion, so that the process, method, article or device including a series of elements includes not only those elements, but also includes other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the sentence "comprise one..." do not exclude the presence of other identical elements in the process, method, article or device including the element. In addition, it should be pointed out that the scope of the methods and devices in the embodiments of the present application is not limited to performing functions in the order shown or discussed, and may also include performing functions in a substantially simultaneous manner or in reverse order according to the functions involved, for example, the described method may be performed in an order different from that described, and various steps may also be added, omitted, or combined. In addition, the features described with reference to certain examples may be combined in other examples.

[0192] Through the description of the above implementation methods, those skilled in the art can clearly understand that the above-mentioned embodiment methods can be implemented by means of software plus a necessary general hardware platform, and of course by hardware, but in many cases the former is a better implementation method. Based on such an understanding, the technical solution of the present application, or the part that contributes to the prior art, can be embodied in the form of a computer software product, which is stored in a storage medium (such as ROM / RAM, a magnetic disk, or an optical disk), and includes a number of instructions for enabling a terminal (which can be a mobile phone, a computer, a server, or a network device, etc.) to execute the methods described in each embodiment of the present application.

[0193] The embodiments of the present application are described above in conjunction with the accompanying drawings, but the present application is not limited to the above-mentioned specific implementation methods. The above-mentioned specific implementation methods are merely illustrative and not restrictive. Under the guidance of the present application, ordinary technicians in this field can also make many forms without departing from the purpose of the present application and the scope of protection of the claims, all of which are within the protection of the present application.

[0194] In the description of this specification, the description with reference to the terms "one embodiment", "some embodiments", "illustrative embodiments", "examples", "specific examples", or "some examples" means that the specific features, structures, materials, or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the present application. In this specification, the schematic representation of the above terms does not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials, or characteristics described may be combined in any one or more embodiments or examples in a suitable manner.

[0195] Although the embodiments of the present application have been shown and described, those skilled in the art will appreciate that various changes, modifications, substitutions and variations may be made to the embodiments without departing from the principles and spirit of the present application, and that the scope of the present application is defined by the claims and their equivalents.

Claims

1. A lane line tracking method, characterized in that: include: Preprocessing the input image, detecting and screening out lane line hot spots in the input image by using a probabilistic Hough transform method; Using the lane line hotspot area and the input image as inputs of a trained lane line detection convolutional neural network to obtain detected lane line coordinate points; Using a quadratic curve to perform least squares fitting on the detected lane line coordinate points to obtain a fitted lane line; Input the coefficients of the fitted lane line as system state variables into the trained Kalman network model for tracking prediction, output the final tracking lane line structure and draw the final tracking lane line structure in the input image; The lane detection convolutional neural network is trained by the following steps: Obtain images containing lane lines and corresponding lane line labels, and divide them into training data sets and validation data sets according to a preset ratio; Preprocessing each image in the training data set by grayscale transformation, color space feature enhancement and edge detection, and detecting and screening lane line hot spots in each image in the training data set by probabilistic Hough transform method; Taking the lane line hotspot area in each image in the training data set as a candidate box, and inputting it into the lane line detection convolutional neural network together with the corresponding image, to obtain the prediction output of the lane line detection convolutional neural network; Calculating a loss based on the predicted output and the lane line label, and optimizing parameters of the lane line detection convolutional neural network based on the loss; The lane line hotspot area in each image in the training data set is used as a candidate frame, and is input into the lane line detection convolutional neural network together with the corresponding image to obtain the prediction output of the lane line detection convolutional neural network, including: Loading the corresponding image into the backbone network, and obtaining an image feature map through convolution downsampling operation; Generate a candidate box based on the lane line hotspot area in each image in the training data set; Input the candidate box into a feature pooling layer based on Hough anchor points, and perform a feature layer reduction operation based on the candidate box; The output of the feature pooling layer based on the Hough anchor point is subjected to attention calculation through the fully connected layer and the global image layer to obtain the attention feature; Input the attention features into two fully connected layers to obtain classification prediction values ​​and regression prediction values; The candidate boxes of predicted values ​​are screened by the non-maximum suppression method to obtain the optimal predicted value.

2. The lane line tracking method according to claim 1, characterized in that: The calculating the loss based on the predicted output and the lane line label, and optimizing the parameters of the lane line detection convolutional neural network based on the loss, comprises: The optimal prediction value and the lane line label are input into a loss function to calculate the loss, and the parameters of the lane line detection convolutional neural network are optimized based on the loss and the optimizer, wherein the loss function is: L is the loss, is the classification loss function, is the regression loss function, is a balance parameter.

3. The lane line tracking method according to claim 1, characterized in that: The Kalman network model is used to predict and calculate the Kalman gain in the Kalman filter through the GRU gated recurrent unit in the recurrent neural network.

4. The lane line tracking method according to claim 3, characterized in that: The Kalman network model is trained by the following steps: Establish the lane line system state equation; By inputting random time into the lane line system state equation, data parameters in the system evolution process are obtained, thereby obtaining lane line tracking data; The lane line tracking data and the lane line system state equation are loaded into a convolutional recurrent neural network for training. After the training is completed, the Kalman network model is obtained.

5. The lane line tracking method according to claim 1, characterized in that: The coefficients of the fitted lane lines are input into the trained Kalman network model as system state variables for tracking prediction, and the final tracking lane line structure is output, including: The a, b, c parameters of the fitted lane line quadratic curve are input as system state variables into the trained Kalman network model for tracking prediction, and the optimal a, b, c parameters of the tracking prediction and the lane line tracking result curve equation are obtained.

6. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the program, the lane line tracking method as described in any one of claims 1-5 is implemented.

7. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the lane line tracking method as described in any one of claims 1 to 5 is implemented.

8. A vehicle, characterized in that: Comprising the electronic device as claimed in claim 6.

Images