Real-time accurate lane detection and classification methods, systems and vehicles

Abstract

This invention discloses a real-time accurate lane line detection and classification method, system, and vehicle, including: Step 1, collecting lane line data; Step 2, cleaning the data; Step 3, labeling the data; Step 4, using LaneATT as the initial model architecture, employing a neural network architecture search to find two backbone networks, one a heavy backbone network and the other a lightweight backbone network; Step 5, using the heavy backbone network as the lane line detection teacher model and the lightweight network as the lane line detection student model, performing knowledge distillation, and then using self-attention distillation to enhance the lane line features learned by the lightweight network; Step 6, simultaneously performing position regression and classification of lane lines to achieve simultaneous prediction of lane line position and category. This invention can simultaneously predict lane line position and category, and has high real-time performance and low resource consumption.

Description

Technical Field

[0001] This invention belongs to the field of autonomous driving technology, specifically relating to a real-time accurate lane line detection and classification method, system, and vehicle. Background Technology

[0002] Autonomous driving systems are one of the core systems of current intelligent vehicles, mainly composed of three major modules: perception fusion module, decision planning module, and control module. Among them, the perception fusion module, as the precursor to the other two modules, directly determines the performance of the entire autonomous driving system through the accuracy of its perception. Lane detection, as a crucial sub-module within the perception fusion module, has a significant impact on the overall system performance due to its accuracy and real-time performance.

[0003] Lane detection currently faces seven major challenges:

[0004] (1) Lane lines are open and continuous geometric curves that extend into space, making it difficult to define them using a detection box;

[0005] (2) Lane lines can be obscured, especially in congested traffic conditions, which places considerable demands on the robustness of the detection algorithm;

[0006] (3) Lane markings may be worn;

[0007] (4) Lane markings may be partially in shadow or difficult to see in low light conditions at night;

[0008] (5) Lane lines on curves are more difficult to locate accurately than on straight sections;

[0009] (6) Lane markings themselves come in many different categories, and the categories are uneven (see...). Figure 1 and Figure 2 );

[0010] (7) Lane lines can be divided and merged by road lines, which makes the definition and detection of lane lines ambiguous.

[0011] Current vision-based lane detection algorithms can be mainly divided into the following four schools of thought:

[0012] (1) Based on traditional computer vision algorithms: This approach (such as CN201310450258.7 Lane line detection method, system, lane departure warning method and system) first calibrates the camera, then uses inverse perspective mapping (IPM) to transform the original image into a top view, and then uses the Canny edge detection algorithm and Hough transform to find possible lane lines in the top view. Lane line detection systems based on the above traditional vision algorithms have low robustness and are difficult to apply to current autonomous driving lane line detection systems.

[0013] (2) Semantic Segmentation Based on Deep Learning: This approach transforms lane detection into semantic segmentation, which is further divided into conventional semantic segmentation (e.g., CN202011612255.5 A Lane Detection Method and Device Based on Deep Neural Network in Low-Light Environments) and lane-specific semantic segmentation (e.g., CN202011550418.1 A Lane Detection and Segmentation Method Based on Attention Space Convolutional Neural Network). The conventional approach applies general-purpose semantic segmentation networks to lane segmentation, which is easily affected by the imbalance of sparse lane pixel samples and the lane lines themselves. Dedicated lane-specific semantic segmentation addresses the continuous and slender structural features of lane lines and the specific characteristics of detection, improving the detection model to more effectively handle issues such as lane occlusion, lane line damage, and uneven lane pixel distribution. Lane detection systems based on semantic segmentation consume significant amounts of embedded chip memory, becoming a performance bottleneck. Furthermore, post-processing may require pixel sampling and clustering, which also consumes CPU resources, making it difficult for the algorithm to achieve very high frame rates.

[0014] (3) Line detection based on deep learning and anchor lines: This approach first assumes a series of prior lane line positions, called anchor lines, and then uses a model to regress the deviation values ​​between the real lane lines in the image and these prior lane lines. Some representative models include: (1) Line-CNN designed by Nanjing University (Li, X., et al. (2019). "Line-CNN: End-to-End Traffic line detection with line proposal unit." 21(1): 248-258.); (2) CurveLane-NAS designed by Huawei (Li, Z. (2020). "CurveLane-NAS: Unifying Lane-Sensitive Architecture Search and Adaptive Point Blending."); (3) LaneATT designed by Lucas Tabelini et al. (Tabelini, L., et al. (2020). "Keep your Eyes on the Lane: Attention-guided Lane Detection."). Compared to semantic segmentation, this type of model has the advantages of: (a) detecting any lane lines without being limited by the maximum number of lane line detections; and (b) directly outputting the detected lane lines, eliminating pixel sampling and clustering, thus reducing intermediate processes. Its limitation lies in the need to assume a series of anchor lines. If the actual lane lines and the anchor line positions differ significantly, it is easy to miss detections. Therefore, the accuracy of the algorithm is closely related to the prior location selection of the anchor lines.

[0015] (4) End-to-end lane detection based on deep learning: This approach can be further divided into two categories. The first category directly outputs the points of each lane line (e.g., CN201910628238.1 An end-to-end lane line detection method and its system), while the second category (e.g., CN202110154312.8 Lane line detection method, system and electronic equipment) does not output the points at all, but directly outputs the four coefficients (C0, C1, C2, C3) of the cubic polynomial curve corresponding to the lane line. Compared with (2) and (3), this approach has a stronger global abstraction ability and requires less post-processing, and can achieve better results for simple scenarios. However, the models of this approach are usually less stable and have weaker generalization ability. At the same time, due to the end-to-end nature of the model, many intermediate processes are hidden, making its interpretability very poor. After multiple practical demonstrations, this type of model performs well on public datasets, but performs very poorly in real scenarios, and therefore is difficult to apply to actual autonomous driving lane line detection systems.

[0016] Lane classification: Most current literature focuses only on lane position regression, with very little attention paid to lane classification. The few papers that do focus on classification (Pizzati, F., et al. (2019). Lane detection and classification using cascaded CNNs. International Conference on Computer Aided Systems Theory, Springer) almost all employ a two-stage model. First, the lane position is regressed; then, the ROI region is extracted using the positional information; and finally, a small classification network is used to further subdivide the lane type. This two-stage model decouples regression and classification, improving accuracy to some extent, but it struggles to achieve real-time performance on computationally scarce autonomous driving chips.

[0017] In summary, current mainstream lane detection algorithms are based on deep learning, including semantic segmentation, line detection, and end-to-end approaches. Semantic segmentation offers the strongest interpretability but has the highest memory requirements and involves complex post-processing. End-to-end methods consume fewer resources but have the worst interpretability and robustness. Line detection represents a compromise between the two, offering some interpretability without involving complex post-processing logic. However, some current line detection-based models still struggle to achieve the real-time performance required for autonomous driving (generally requiring an FPS > 30) on resource-constrained embedded systems, and almost none of them offer multi-task networks that directly output lane position and category simultaneously.

[0018] Therefore, there is a need to develop a new real-time accurate lane detection and classification method, system, and vehicle. Summary of the Invention

[0019] The purpose of this invention is to provide a real-time and accurate lane line detection and classification method, system, and vehicle that can simultaneously predict lane line position and category, and has high real-time performance and low resource consumption.

[0020] In a first aspect, the real-time accurate lane line detection and classification method of the present invention includes the following steps:

[0021] Step 1: Collect lane line data;

[0022] Step 2: Clean the collected data;

[0023] Step 3: Label the cleaned data;

[0024] Step 4: Using LaneATT as the initial model architecture, a neural network architecture search was used to find two backbone networks, one of which is a heavy backbone network and the other is a lightweight backbone network.

[0025] Step 5: Use the heavy backbone network as the teacher model for lane line detection and the lightweight network as the student model for lane line detection. Perform knowledge distillation and then use self-attention distillation to enhance the lane line features learned by the lightweight network.

[0026] Step 6: Perform position regression and classification on the lane lines simultaneously to achieve simultaneous prediction of the position and category of the lane lines.

[0027] Optionally, step 1 specifically includes:

[0028] The camera is mounted on the windshield of the data acquisition vehicle, ensuring that lane lines on the road are visible within the field of view, and data is collected under different operating conditions.

[0029] Optionally, step 2 specifically includes:

[0030] Each video segment from each camera is extracted at preset intervals, and then the extracted images are filtered to cover as many scenes as possible.

[0031] Optionally, step 3 specifically includes:

[0032] Each lane line is marked with a segmented broken line, along with the lane line type.

[0033] Optionally, step 4 specifically includes:

[0034] Using the Resnet122-LaneATT network as a prototype for a large model, training data was extracted, and the DARTS search algorithm was used to search for a large model for detecting lane lines with the lane line detection accuracy of the model as the optimization metric. Then, the large model was trained and tested to obtain the heavy backbone network.

[0035] Using the ResNet18-LaneATT network as a small model prototype, training data was extracted, and the DARTS search algorithm was used to search for a small model for detecting lane lines, with the lane line detection accuracy of the model as the optimization metric. Then, the small model was trained and tested to obtain a lightweight backbone network.

[0036] Optionally, step 5 specifically includes:

[0037] The trained large model is used as the teacher model for lane line detection, and the trained small model is used as the student model for lane line detection. Knowledge distillation is used to transfer the feature knowledge of the large model to the small model.

[0038] After knowledge distillation is completed, self-attention distillation is performed, which makes the slender structural features of lane lines easier for the model to learn.

[0039] Optionally, step 6 specifically includes:

[0040] Using the student model for lane line detection found in the search as the basic architecture, a classification branch is added, and multi-task training is performed simultaneously, including regression and classification. Then, images containing various types of lane lines are selected from the test set and labeled for category. During training, a portion of the images are taken as the training set and a portion of the images are taken as the validation set for training and testing the model.

[0041] Optionally, it also includes step 7: model deployment, specifically:

[0042] The model obtained in step 6 is exported in ONNX format, and all parameters are FP16 quantized using the tensorrt interface of ONNX runtime. At the same time, some operators are fixed-point quantized, resulting in a hybrid quantization model.

[0043] Secondly, the real-time accurate lane line detection and classification system of the present invention includes a memory and a controller. The memory stores a computer-readable program, which, when invoked by the controller, can execute the steps of the real-time accurate lane line detection and classification method of the present invention.

[0044] Thirdly, the vehicle described in this invention employs the real-time accurate lane line detection and classification system as described in this invention.

[0045] This invention has the following advantages: Addressing the high real-time requirements and low resource consumption of autonomous driving, this invention specifically designs a lane detection algorithm based on the line detection approach, suitable for deployment on embedded boards with limited computing resources. Compared with all existing mainstream algorithms, this invention employs a model architecture search method and a model compression method to simultaneously improve the accuracy and real-time performance of the model on embedded boards. Attached Figure Description

[0046] To more clearly illustrate the technical solutions of the embodiments of the present invention, the accompanying drawings used in the embodiments will be briefly introduced below. It should be understood that the following drawings only show some embodiments of the present invention and should not be regarded as a limitation on the scope. For those skilled in the art, other related drawings can be obtained based on these drawings without creative effort.

[0047] Figure 1 This is a flowchart of this embodiment;

[0048] Figure 2 A schematic diagram defining lane line categories;

[0049] Figure 3 The architecture is based on the ResNet122 / 18-LaneATT line detection model.

[0050] Figure 4 Designed for searching cells in neural network architectures;

[0051] Figure 5 The network architecture of the searched model;

[0052] Figure 6 For teacher-student distillation architecture;

[0053] Figure 7 For the model's self-attention distillation architecture;

[0054] Figure 8 A multi-task model (final model) for lane line location and category;

[0055] Figure 9 This is one of the flowcharts for an online real-time inference algorithm;

[0056] Figure 10 This is the second flowchart of the online real-time inference algorithm;

[0057] Figure 11 Annotate the sample images for the data;

[0058] Figure 12 This is a diagram demonstrating the algorithm's performance. Detailed Implementation

[0059] The present invention will now be described in detail with reference to the accompanying drawings.

[0060] like Figure 1 As shown in this embodiment, a real-time accurate lane line detection and classification method includes the following steps:

[0061] Step 1: Collect lane line data;

[0062] Step 2: Clean the collected data;

[0063] Step 3: Label the cleaned data;

[0064] Step 4: Using LaneATT as the initial model architecture, a neural network architecture search was adopted to find two backbone networks. One is a heavy backbone network with high accuracy but poor real-time performance; the other is a lightweight backbone network, which ensures the highest accuracy architecture while maintaining real-time performance.

[0065] Step 5: Use the heavy backbone network as the teacher model for lane line detection and the lightweight network as the student model for lane line detection. Perform knowledge distillation and then use self-attention distillation to enhance the lane line features learned by the lightweight network.

[0066] Step 6: Perform position regression and classification on the lane lines simultaneously to achieve simultaneous prediction of the position and category of the lane lines.

[0067] The following provides a detailed explanation of each step:

[0068] 1. Collect lane line data:

[0069] The requirements for the algorithm in lane detection determine the requirements for data acquisition:

[0070] (1) The algorithm is required to cover the extreme scenarios as much as possible, and at the same time, the algorithm is required to adapt to cameras with different resolutions, different types of cameras, and different installation positions of cameras.

[0071] (2) It is required to collect as many different types of lane lines as possible in order to reduce the imbalance of lane line types.

[0072] Therefore, during data collection, ten 2-megapixel cameras were randomly installed behind the windshields of ten data collection vehicles, ensuring that lane lines on the road were visible within the field of view. The arbitrary camera placement, as long as lane lines were present in the field of view, ensured a wider data distribution and enhanced the model's generalization ability. Data was collected under various operating conditions, covering lane lines in extreme scenarios as much as possible. Detailed data collection scenarios are shown in Table 1.

[0073] Table 1: Lane line data acquisition conditions;

[0074]

[0075]

[0076] 2. Clean the collected data:

[0077] For each video segment from each camera, frames are extracted at certain intervals. The extracted images are then filtered to ensure coverage of as many different scenes as possible. In particular, scenes with extreme operating conditions and lane markings with complex patterns are selected.

[0078] In one instance, each vehicle was captured for a week, with 2 hours of video collected each day. Each video segment was at 15 FPS, resulting in a total of 108,000 images. Ten vehicles collected a total of 7.56 million images in a week. These images were then filtered, and 250,000 representative images were selected, covering most scenarios. Of these, 200,000 were used for training, 20,000 for validation, and 30,000 for testing.

[0079] 3. Label the cleaned data:

[0080] For each lane marking, the sample is marked with a segmented polyline, along with the lane marking type (see [link]). Figure 2 For an example of lane line markings, see [link / example]. Figure 11 For lane lines that are obscured, only those obscured by no more than 1 / 3 should be marked.

[0081] In this embodiment, step 4 specifically includes:

[0082] Using the Resnet122-LaneATT network as a prototype for a large model, training data was extracted, and the DARTS search algorithm was used to search for a large model for detecting lane lines with the lane line detection accuracy of the model as the optimization metric. Then, the large model was trained and tested to obtain the heavy backbone network.

[0083] Using the ResNet18-LaneATT network as a small model prototype, training data was extracted, and the DARTS search algorithm was used to search for a small model for detecting lane lines, with the lane line detection accuracy of the model as the optimization metric. Then, the small model was trained and tested to obtain a lightweight backbone network.

[0084] 4.1 Design of Neural Network Architecture for Large and Small Models:

[0085] Using the Resnet122-LaneATT network as an example (participants) Figure 3 Large-scale model design prototype, using ResNet18-LaneATT network (see...) Figure 3 This paper designs a prototype differential neural network search architecture suitable for lane detection for small-scale models. Taking a large-scale model as an example, in... Figure 3 In this example, a Cell module was redesigned to replace the ResNet residual module (see [link]). Figure 4This cell has a two-layer residual topology. This cell structure has a total of 4 operation nodes (OP Nodes), each corresponding to 5 possible operations: (1) identity operation (short-circuit operation); (2) zero operation (open-circuit operation); (3) 1x1 convolution + BN + ReLU operation; (4) 3x3 separable convolution + BN + ReLU operation; (5) 5x5 separable convolution + BN + ReLU operation. This design ensures that each cell unit can learn the fine and detailed structure of the lane lines as much as possible (through two levels of residuals). For residual networks, there are two basic residual modules (see...). Figure 3 The first type maintains the same input and output dimensions (width and height), while the second type downsamples the last layer with a stride of 2. In this example, a cell structure is used to replace the main body of both types of modules. For the second type, Residual Block, an op is added to the end of the corresponding cell. This op node has two options: conv3x3 and conv5x5. Both ops have a stride of 2, which ensures that they can be downsampled.

[0086] 4.2 Search for Neural Network Architectures of Different Sizes:

[0087] Taking large-scale network architecture search as an example, the DARTS search algorithm was used. On a training set of 100,000 and a validation set of 20,000 (100,000 were extracted from the 200,000 training set as the training set for architecture search), with the objective function of minimizing the lane line loss on the validation set, an optimized lane line detection teacher model was found. Specifically, for each OPNode, the probability vector corresponding to its possible operations is denoted as (taking an op node with 5 operations as an example):

[0088] α={α1,α2,α3,α4,α5} (1)

[0089] Its input and output satisfy the following relationship:

[0090]

[0091] op i (x) means that op i This operation is applied to x (x is called the feature map) and the output is obtained.

[0092] These 5 operations are defined as follows:

[0093] op1 = identity (3)

[0094] op2 = zero (4)

[0095] op3=1×1conv+bn+relu / stride=1 (5)

[0096] op4=3×3conv+bn+relu / stride=1 / separable (6)

[0097] op5=5×5conv+bn+relu / stride=1 / separable (7)

[0098] Therefore, for each node, there is a corresponding vector α as shown in equation (1). This parameter can adjust the weights of the OPs in the network. During training, we consider equation (2) as the mapping formula from input to output for each node, and then perform gradient descent. After optimization, the final operation of each node is:

[0099]

[0100] Specifically, for the gradient descent optimization algorithm described above, the optimization objective of this method is as follows:

[0101]

[0102] w * (α)=argmin w Loss train (w,α) (10)

[0103] Loss = Loss regression +w1×Loss conf (11)

[0104] The loss includes the regression lane line position deviation loss and the lane line confidence loss.

[0105] First, keep α constant, then use gradient descent on a training set of 100,000 images to optimize the network weights until they locally converge to the weights w. * (α). Then, with the weights fixed, the OP weights α are optimized using gradient descent on the validation set until they converge locally. This process is repeated iteratively to train w. * (α) and α, and finally the algorithm is considered to have converged when α does not change much. At this time, the final OP operation corresponding to each node is obtained by using equation (8).

[0106] Thanks to this differential neural network search architecture, on a training set of 100,000 images and a validation set of 20,000 images, using a single TESLA V100 GPU, only about 60 GPUs were needed for a large model and 20 GPUs for a small model to find a specific structure suitable for the lane detection task (see search results). Figure 5 After the architecture search was completed, this method further trained the network structures (large and small models) found above on all 200,000 images and tested them on a test set with 30,000 images. The results show that the networks searched using the DARTS algorithm have improved accuracy metrics compared to the original networks (see Table 2). Specifically, for the large model, the accuracy reached 89.19%, with a pass rate of 2.2% and a false negative rate of 17.15%. Given that the dataset collected by this method contains many challenging scenarios, such metrics are sufficient for use as a teacher network for distillation. For the small model, due to the limited expressive power of its backbone network, the accuracy was 84.24%, with a pass rate of 6.45% and a false negative rate of 30.25%, indicating a significant false negative rate. Therefore, model distillation will be used to further improve the network's metrics.

[0107] Table 2: Comparison of Model Indicators

[0108] Model Accuracy Pass rate False negative rate Classification accuracy Resnet122-LaneATT 86.83％ 4.68％ 20.56％ / Resnet18-LaneATT 80.23％ 8.95％ 35.57％ / Darts-teacher-model(big) 89.19％ 2.20％ 17.15％ / Darts-student-model(small) 84.24％ 6.45％ 30.25％ / Small + distillation 86.75％ 5.37％ 22.67％ / Small + Distillation + Self-Distillation 88.05％ 4.84％ 19.21％ / small + distillation + autodistillation + classification and branching 87.72％ 6.64％ 21.21％ 76.205％ +F16 / INT8 Hybrid Quantization 87.14％ 6.82％ 22.03％ 75.810％

[0109] 5. Model distillation:

[0110] The trained large model is used as the teacher model for lane line detection, and the trained small model is used as the student model for lane line detection. Knowledge distillation is used to transfer the feature knowledge of the large model to the small model. After knowledge distillation, self-attention distillation is performed to make the slender structural features of lane lines easier for the model to learn.

[0111] In one instance, a large model is used as the teacher detection model for lane lines, and a small model is used as the student detection model for lane lines. Knowledge distillation is employed to transfer the feature knowledge from the large model to the small model. See [link to relevant documentation]. Figure 6 During distillation, the weights of the small model trained in the previous step are used as the initial weights, and weights are applied from three different locations (…). Figure 6 The feature maps of the lane line detection teacher model and the lane line detection student model are derived, and then the difference value between the two feature maps is calculated.

[0112] This method performed 50 epochs of knowledge distillation. After distillation, the model's metrics improved further: accuracy increased to 86.75%, the over-detection rate was 5.37%, and the false negative rate decreased significantly to 22.67%. This indicates that the smaller model successfully learned more detailed lane line features from the larger model, further improving its ability to detect lane lines at some edges. To further improve the lane line model's metrics, self-attention distillation was performed, and its distillation architecture is as follows: Figure 7Self-attention distillation only involves the small model and does not involve the teacher-student lane detection model. This method extracts feature maps from the four downsampled feature layers of the backbone network and then performs self-attention distillation. Unlike teacher-student distillation, self-attention distillation requires upsampling the smaller feature maps, then calculating the softmax, and finally taking the L2 difference between the two feature maps as an additional loss term during training. After self-distillation, the accuracy of the small model is further improved to 88.05%, the over-detection rate is reduced to 4.84%, and the false negative rate is 19.21%.

[0113] 6. Training of lane line classification branches:

[0114] Using the student model for lane line detection found in the search as the basic architecture, a classification branch is added, and multi-task training is performed simultaneously, including regression and classification. Then, images containing various types of lane lines are selected from the test set and labeled for category. During training, a portion of the images are taken as the training set and a portion of the images are taken as the validation set for training and testing the model.

[0115] In one instance, this method uses the searched lane line detection student model as its basic architecture, further adding a classification branch head, and simultaneously performing multi-task training (regression + classification). Specifically (see...) Figure 8 This method incorporates an MLP layer as a classification branch (using only a single-layer neural network to avoid additional inference time), and then selects 12,000 images with various lane line types from the test set for category labeling. During training, 10,000 images are used as the training set and 2,000 as the validation set. The initial learning rate for the classification branch head is set to 0.001, the same as the regression model, while the learning rate for the original student architecture parameters is set to 0.00001 (only for fine-tuning weights during multi-task operations, avoiding significant changes to previously learned weights). After training for 50 epochs on the 10,000 images, on the 2,000 validation set, the accuracy is 92.21%, the over-detection rate is 5.68%, the under-detection rate is 23.21%, and the classification accuracy for 17 categories is 76.205%. However, when only the regression index was tested on the original test set of 30,000 images, it was found that the detection accuracy decreased slightly to 87.72%, the pass rate increased to 6.64%, and the false negative rate increased to 21.21%.

[0116] 7. Model deployment and C++ API encapsulation:

[0117] The model obtained in step 6 is exported in ONNX format, and all parameters are FP16 quantized using the tensorrt interface of ONNX runtime. At the same time, some operators are fixed-point quantized, resulting in a hybrid quantization model.

[0118] In one instance, the original model was exported to ONNX format, and all parameters were FP16 quantized using the ONNX runtime's TensorRT interface. Simultaneously, some operators were quantized using fixed-point quantization (INT8), resulting in a hybrid quantized model with a size of 22MB and MACs of 9.3GB. Testing on an NVIDIA XAVIER NX showed an average forward single-batch inference speed of 16ms, achieving 62 FPS. After quantization, testing on the same 30,000 images yielded an accuracy of 87.14%, a pass rate of 6.82%, and a false negative rate of 22.03%. On 2000 images with classification labels, the classification accuracy was 75.81%.

[0119] The output of the original network inference is a series of candidate anchor lines with confidence scores, which require post-processing. Similar to object detection, a threshold needs to be set first to remove lines with confidence scores below the threshold. Then, lines with high overlap are removed using Non-Maximum Suppression (NMS), leaving the lines with the highest local confidence scores. Finally, the category with the highest probability in the classification branch is taken as the lane line category corresponding to this anchor line.

[0120] This method also designs a complete set of lane detection algorithm flowcharts, which are encapsulated into an interface. The interface includes initialization, preprocessing, network inference, postprocessing, 2D to 3D conversion, visualization, etc. See [link to relevant documentation]. Figure 9 and Figure 10 After testing, the average inference time of the entire interface is 28ms. As mentioned above, the average network inference time is 16ms, preprocessing time is 4ms (image resizing and normalization), postprocessing time is 2ms, and communication and transmission time between CUDA and CPU is 6ms. The algorithm achieves a real-time frame rate of 35FPS, which fully meets the requirements for real-time lane detection in vehicles, while consuming relatively few resources. See [link to algorithm performance demonstration] for more information. Figure 12 This algorithm has been successfully tested as a lane detection module on the Changan S202-ICA internal test vehicle equipped with the ADAS system. It can control the vehicle in real time to maintain lane and change lanes under high-speed conditions.

[0121] In this embodiment, a real-time accurate lane line detection and classification system includes a memory and a controller. The memory stores a computer-readable program, which, when invoked by the controller, can execute the steps of the real-time accurate lane line detection and classification method described in this embodiment.

[0122] In this embodiment, a vehicle employs a real-time accurate lane line detection and classification system as described in this embodiment.

[0123] The above embodiments are preferred embodiments of the present invention, but the embodiments of the present invention are not limited to the above embodiments. Any changes, modifications, substitutions, combinations, or simplifications made without departing from the spirit and principle of the present invention shall be considered equivalent substitutions and shall be included within the protection scope of the present invention.

Claims

1. A real-time accurate lane line detection and classification method, characterized in that, Includes the following steps: Step 1: Collect lane line data; Step 2: Clean the collected data; Step 3: Label the cleaned data; Step 4: Using LaneATT as the initial model architecture, a neural network architecture search was employed to identify two backbone networks: one heavy backbone network and the other lightweight backbone network. Specifically: Using the Resnet122-LaneATT network as a prototype for a large model, training data was extracted, and the DARTS search algorithm was used to search for a large model for detecting lane lines with the lane line detection accuracy of the model as the optimization metric. Then, the large model was trained and tested to obtain the heavy backbone network. Using the Resnet18-LaneATT network as a small model prototype, training data was extracted, and the DARTS search algorithm was used to search for a small model for detecting lane lines with the lane line detection accuracy as the optimization metric. Then, the small model was trained and tested to obtain a lightweight backbone network. Step 5: Use the heavy backbone network as the teacher model for lane line detection and the lightweight backbone network as the student model for lane line detection. Perform knowledge distillation and then use self-attention distillation to enhance the lane line features learned by the lightweight backbone network. Step 6: Perform position regression and classification on the lane lines simultaneously to predict both the position and category of the lane lines. Specifically: Using the student model for lane line detection found in the search as the basic architecture, a classification branch is added, and multi-task training is performed simultaneously, including regression and classification. Then, images containing various types of lane lines are selected from the test set and labeled for category. During training, a portion of the images are taken as the training set and a portion of the images are taken as the validation set for training and testing the model.

2. The real-time accurate lane line detection and classification method according to claim 1, characterized in that: Step 1 specifically involves: The camera is mounted on the windshield of the data acquisition vehicle, ensuring that lane lines on the road are visible within the field of view, and data is collected under different operating conditions.

3. The real-time accurate lane line detection and classification method according to claim 2, characterized in that: Step 2 specifically involves: Each video segment from each camera is extracted at preset intervals, and then the extracted images are filtered to cover various scenes.

4. The real-time accurate lane line detection and classification method according to claim 3, characterized in that: Step 3 specifically involves: Each lane line is marked with a segmented broken line, along with the lane line type.

5. The real-time accurate lane line detection and classification method according to claim 4, characterized in that: Step 5 specifically involves: The trained large model is used as the teacher model for lane line detection, and the trained small model is used as the student model for lane line detection. Knowledge distillation is used to transfer the feature knowledge of the large model to the small model. After knowledge distillation is completed, self-attention distillation is performed, which makes the slender structural features of lane lines easier for the model to learn.

6. The real-time accurate lane line detection and classification method according to claim 1, characterized in that, It also includes step 7: model deployment, which is as follows: The model obtained in step 6 is exported in ONNX format, and all parameters are FP16 quantized using the tensorrt interface of ONNX runtime. At the same time, some operators are fixed-point quantized, resulting in a hybrid quantization model.

7. A real-time accurate lane line detection and classification system, characterized in that: It includes a memory and a controller, wherein the memory stores a computer-readable program that, when invoked by the controller, can perform the steps of the real-time accurate lane line detection and classification method as described in any one of claims 1 to 6.

8. A vehicle, characterized in that: The real-time accurate lane line detection and classification system as described in claim 7 is adopted.

Images