Semantic segmentation-based drivable area and lane line detection model construction method
By improving the YOLOv5s model for semantic segmentation, the real-time performance and accuracy issues of drivable areas and lane line detection in complex scenes were resolved, achieving efficient detection results.
Patent Information
- Application Number
- CN202210938073.X
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2022-08-05
- Publication Date
- 2025-11-04
- Estimated Expiration
- 2042-08-05
AI Technical Summary
Existing technologies struggle to achieve real-time, accurate, and efficient detection of drivable areas and lane lines in complex scenarios, especially when obstructed by obstacles such as motor vehicles, non-motor vehicles, pedestrians, rain, snow, or road damage. Traditional methods are ineffective in these situations, while deep learning-based methods suffer from high computational complexity.
An improved YOLOv5s model was constructed, removing the object detection-related branch and adding branches for drivable area and lane line segmentation. Semantic segmentation was performed using the improved YOLOv5s network to improve detection accuracy and speed.
It achieves real-time, accurate, and efficient detection of drivable areas and lane lines, enhancing detection accuracy and speed, and adapting to the detection needs of complex scenarios.
Smart Images
Figure CN115294551B_ABST
Abstract
Description
Technical Field
[0001] This invention belongs to the field of automotive intelligence and autonomous driving, and in particular relates to a method for constructing a drivable area and lane line detection model based on semantic segmentation. Background Technology
[0002] Accurate and reliable drivable area detection is a key perception challenge for autonomous vehicles, and it is a prerequisite for subsequent trajectory planning and control. Lane detection provides positioning information, allowing autonomous vehicles to correctly position themselves within their lanes, which is crucial for lane departure warning, lane keeping, and trajectory planning. Therefore, both drivable area detection and lane detection are indispensable components of autonomous vehicles.
[0003] Due to obstructions from motor vehicles, non-motor vehicles, pedestrians, rain, snow, and road damage, drivable areas often lack regular shape features, making them difficult to detect using traditional methods. Lane detection methods based on traditional feature extraction struggle with lane lines in complex scenes. However, semantic segmentation methods based on deep learning have achieved better detection results. Unlike object detection, which only needs to predict object information in an image, semantic segmentation identifies and understands the content of all pixels in an image. It refines the classification problem to each pixel, assigning each pixel a predefined category, such as drivable area, lane line, or background. Compared to object detection, semantic segmentation detects pixel-level boundaries with greater precision. Reference 1 (Caltagirone L, Bellone M, Svensson L, et al. LIDAR-camera fusion for road detection using fully convolutional neural networks[J]. Robotics and Autonomous Systems, 2019, 111:125-131) utilizes an FCN network to detect drivable areas. This method has a significant advantage in detection accuracy compared to traditional geometric modeling methods. However, the model is relatively complex, resulting in lower segmentation efficiency. Reference 2 (Yoo S, Lee HS, Myeong H, et al. End-to-end lane marker detection via row-wise classification[C]. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition Workshops(CVPRW). 2020:1006-1007.) simplifies lane detection using a classification method. This method uses Horizontal Reduction Modules (HRMs) to convert the output of the segmentation backbone into a row representation. Classification-based lane detection methods simplify the detection task into a multi-line classification task, reducing the size of the model and the amount of computation, but at the cost of accuracy and they cannot detect scenarios with many lane lines very well. Summary of the Invention
[0004] The purpose of this invention is to address the aforementioned difficulties in existing technologies by providing a semantic segmentation-based method for constructing a drivable area and lane line detection model that enables real-time, accurate, and efficient detection. An improved YOLOv5s model is established by removing the target detection-related branches and adding drivable area and lane line segmentation branches, thereby achieving real-time, accurate, and efficient drivable area and lane line detection and enhancing detection accuracy and speed.
[0005] The technical solution of this invention is to address the problem of drivable region and lane line detection from the perspective of semantic segmentation. Based on an improved YOLOv5s network, a simple and efficient segmentation network is constructed to enhance the detection accuracy and speed of drivable regions and lane lines. This invention includes the following steps:
[0006] Step 1: Improve the YOLOv5s model structure and propose an improved YOLOv5s model structure to enhance the model's ability to detect targets;
[0007] Step 2: Establish the drivable area and lane line detection model structure, which mainly consists of a backbone network, a neck structure, a drivable area head, and a lane line head.
[0008] Step 3: After cropping the original input image, normalize it by standardizing each RGB channel;
[0009] Step 4: Design the YOLOv5 model for drivable area and lane line detection, including model inference and post-processing;
[0010] Step 5: Design the loss function for the drivable area and lane line detection model, analyze the training quality of the model based on the magnitude of the loss value, and determine whether the model has converged;
[0011] Step 6: Construct a dataset for detecting drivable areas and lane lines for autonomous vehicles, and design segmentation labels for drivable areas and lane lines;
[0012] Step 7: Train the drivable area and lane line detection model, and design evaluation metrics for the drivable area and lane line detection model.
[0013] In step 1, the specific steps for improving the YOLOv5s model structure and proposing an improved YOLOv5s model structure can be as follows:
[0014] Step 1: Add a module to the FPN structure of the original YOLOv5s model so that the Neck end can make full use of the shallow high-resolution features of the P2 level.
[0015] Step 2: Replace the P5 level feature W / 32×H / 32 in YOLOv5s with the P2 level high-resolution feature W / 4×H / 4 to improve the ability to detect small targets.
[0016] In step 2, the establishment of the drivable area and lane line detection model structure specifically includes the following steps;
[0017] Step 1: Design a backbone network consisting of Focus and CBS_CSP modules to extract features from the input image;
[0018] Step 2: Design the neck structure using feature pyramid network and path aggregation network to achieve feature fusion at different scales;
[0019] Step 3: Remove the PANet structure from the Detection Head and Neck of the improved YOLOv5s model;
[0020] Step 4: Add driving area and lane line segmentation branches to the FPN structure in the Neck end of the improved YOLOv5s model to achieve a simple and efficient segmentation network;
[0021] Step 5: Design the Drivable Area Head and Lane Line Head, and transform the width and height of the feature map back to the resolution of the input image.
[0022] In step 3, the original input image is cropped and normalized, and each RGB channel is standardized. Specifically, the original input image is cropped to a resolution of W×H, the input image is normalized, the pixel values of the three RGB channels [0,255] are scaled to the range of [0,1], and each channel is standardized.
[0023] In step 4, the model inference and post-processing of the YOLOv5 model for designing drivable areas and lane line detection specifically include the following steps:
[0024] Step 1: After the input image passes through the drivable area or lane line detection model, an output feature map of the same size as the input image is generated. The semantic probability output is generated according to the Sigmoid formula and the output is mapped to the range [0,1].
[0025] Step 2: Convert the semantic classification probability to obtain the pixel category, adjust the segmentation map to the size of the original image using bilinear interpolation, and encode the classification result into different colors on the original image.
[0026] In step 5, the specific steps for designing the loss function of the drivable area and lane line detection model, analyzing the quality of model training based on the magnitude of the loss value, and determining whether the model has converged include:
[0027] Step 1: The binary cross-entropy (BCE) loss function is used for calculation, and the sigmoid function is used to transform the input data of the BCE loss function to the range [0,1]. The loss of the entire image is the average of the loss of each pixel.
[0028] Step 2: The semantic segmentation loss is composed of the drivable region detection loss L. drivable Lane detection loss L lane It consists of two parts, and uses the same loss function to calculate the segmentation loss.
[0029] In step 6, the specific steps for constructing the drivable area and lane line detection dataset for autonomous vehicles and designing drivable area and lane line segmentation labels can be as follows:
[0030] Step 1: Connect the coordinates of the drivable area directly, fit the curve part using a cubic Bézier curve, and then fill the closed area to obtain the drivable area segmentation label.
[0031] The second step is to directly connect the coordinates of the lane lines, fit the lane line curve using a polynomial curve, and calculate the center line of the two lines as the new lane line according to the ENet-SAD method, thus obtaining the final lane line label.
[0032] In step 7, the specific steps for training the drivable area and lane line detection model and designing the evaluation index for the drivable area and lane line detection model include:
[0033] Step 1: Set the initial learning rate to 0.001, the number of iterations to 125, the batch size to 16, the weight of the drivable area to 1, the weight of the lane line to 1, and use the SGD optimizer for optimization training.
[0034] Step 2: Driving area detection uses MIoU as the accuracy evaluation metric, which is the average ratio of the intersection to the union of the predicted results and the actual values across all categories.
[0035] Step 3: Based on ENet-SAD, lane accuracy and IoU are selected as evaluation metrics for lane line detection. Lane accuracy is defined as the ratio of correctly predicted lane lines to all actual lane lines.
[0036] This invention employs semantic segmentation to detect drivable areas and lane lines. Based on the improved YOLOv5s object detection model, the PANet structure at the Detection Head and Neck ends of the object detection model is removed, and drivable area and lane line segmentation branches are added after the FPN structure at the Neck end of the object detection model. This results in a simple and efficient segmentation network that enables real-time, accurate, and efficient detection of drivable areas and lane lines, enhancing detection accuracy and speed. Attached Figure Description
[0037] Figure 1 This is a structural diagram of the drivable area and lane line detection model of the present invention. Detailed Implementation
[0038] The following embodiments will further illustrate the present invention with reference to the accompanying drawings.
[0039] like Figure 1 As shown, the embodiments of the present invention include the following steps:
[0040] Step 1: Improve the YOLOv5s model structure and propose an improved YOLOv5s model structure to enhance the model's ability to detect targets.
[0041] Step 1.1: Add a module to the FPN structure of the original YOLOv5s model so that the Neck end can make full use of the P2 level shallow high-resolution features;
[0042] Step 1.2: Replace the P5 level feature W / 32×H / 32 in YOLOv5s with the P2 level high-resolution feature W / 4×H / 4 to improve the ability to detect small targets.
[0043] Step 2: Establish a drivable area and lane line detection model based on the improved YOLOv5s. This model mainly consists of a backbone network, a neck structure, a drivable area head, and a lane line head.
[0044] Step 2.1: Design a backbone network consisting of Focus and CBS_CSP modules to extract features from the input image.
[0045] Step 2.1.1: Slice and expand the four adjacent pixels according to the channel dimension of the input image, and then concatenate the four slices to obtain a feature map with a width and height of half that of the input and a channel number of four times that of the input. Then, pass it through the CBS (k=3, s=1) module to finally obtain the output feature map of the Focus module.
[0046] Step 2.1.2: The CBS_CSP module consists of a CBS module and a CSP module. The CBS module uses a convolutional layer with a stride of 2 to perform downsampling. The CSP module introduces residual units, which can effectively alleviate the gradient vanishing problem caused by deep neural networks.
[0047] Step 2.1.3: The CSP module splits the input into two branches. The first branch uses a CBS(k=1, s=1) module and n Res Units stacked together. One Res Unit consists of CBS(k=1, s=1), CBS(k=3, s=1), and the Add module. The second branch is processed by the CBS(k=1, s=1) module and then concatenated with the first branch. It is then processed by the CBS(k=1, s=1) module again, finally obtaining the output feature map of the CSP module.
[0048] Step 2.2: Design a neck structure using a feature pyramid network and a path aggregation network structure to achieve top-down feature fusion at different scales.
[0049] Step 2.2.1: Use FPN to upsample deep features to achieve top-down feature fusion at different scales, and at the same time propagate deep feature information to shallow layers, thereby obtaining more strong semantic information to improve the network classification ability.
[0050] Step 2.2.2: Remove the PANet structure from the Detection Head and Neck of the improved YOLOv5s model.
[0051] Step 2.2.3: Add driving area and lane line segmentation branches to the FPN structure in the Neck end of the improved YOLOv5s model to achieve a simple and efficient segmentation network.
[0052] Step 2.3: Design the Drivable Area Head and Lane Line Head, and transform the width and height of the feature map back to the resolution of the input image.
[0053] Step 2.3.1: The output feature map of the UpConcat layer has a resolution of W / 4 × H / 4. Two CSPup layers are used to restore the feature map resolution, and the restored feature map resolution is W × H.
[0054] Step 2.3.2: Then, the CBS module is used to transform the number of channels into 2 channels to correspond to the number of categories. Finally, the semantic probability output of the drivable area and lane line segmentation is generated through the Sigmoid layer.
[0055] Step 2.3.3: The CSPUP layer consists of CSP1, CBS and Upsample modules. CSP1 and CBS modules perform feature extraction, ensuring that the feature map resolution remains unchanged, and further generate a denser feature map. The Upsample module restores the feature map resolution by upsampling by 2 times, and uses the nearest neighbor interpolation method to achieve upsampling.
[0056] Step 3: Crop the original input image to a resolution of W×H and normalize the input image.
[0057] Step 3.1: First, use equation (1) to scale the pixel values of the three RGB channels [0, 255] to the range [0, 1].
[0058]
[0059] Step 3.2: Then standardize each channel using equation (2).
[0060]
[0061] In the formula, x i,j (R) represents the image position (i,j) i,j G i,j B i,j The pixel values are: mean = (0.485, 0.456, 0.406) and standard deviation = (0.229, 0.224, 0.225).
[0062] Step 4: Design the model inference and post-processing for the drivable area and lane line detection model.
[0063] Step 4.1: After the input image passes through the drivable area or lane line detection model, an output feature map of the same size as the input image is generated. The semantic probability output is generated according to the Sigmoid calculation formula of Equation (3), and the output is mapped to the range [0,1].
[0064]
[0065] In the formula, y represents the model output; y′ represents the model semantic probability output.
[0066] Step 4.2: First, the semantic classification probability is converted to obtain the pixel category using equation (4). Then, the segmentation map is adjusted to the size of the original image using bilinear interpolation. Finally, the classification result is encoded into different colors on the original image.
[0067]
[0068] In the formula, The classification result represents the image location (i,j).
[0069] Step 5: Design the loss function for the drivable area and lane line detection model, analyze the quality of model training based on the magnitude of the loss value, and further determine whether the model has converged.
[0070] Step 5.1: The binary cross-entropy (BCE) loss function is used for calculation, and the sigmoid function is used to transform the input data of the BCE loss function to the range [0,1]. The loss of the entire image is the average of the loss of each pixel.
[0071] Step 5.2: The semantic segmentation loss is composed of the drivable region detection loss L. drivable Lane detection loss L lane It consists of two parts, and uses the same loss function to calculate the segmentation loss. The calculation formula is as follows:
[0072]
[0073] In the formula, W and H represent the width and height of the final output feature map of the segmentation model; S represents the true label for semantic segmentation; S represents the predicted semantic information.
[0074] Step 6: Construct a dataset for detecting drivable areas and lane lines for autonomous vehicles, and design segmentation labels for drivable areas and lane lines.
[0075] Step 6.1: Connect the coordinates of the drivable area directly, fit the curve part with a cubic Bézier curve, and then fill the closed area to obtain the drivable area segmentation label.
[0076] Step 6.2: Connect the coordinates of the lane lines directly, fit the lane line curve using a polynomial curve, and calculate the center line of the two lines as the new lane line according to the ENet-SAD method, thus obtaining the final lane line label.
[0077] Step 7: Train the drivable area and lane line detection model, and design evaluation metrics for the drivable area and lane line detection model.
[0078] Step 7.1: Set the initial learning rate to 0.001, the number of iterations to 125, the batch size to 16, the weight of the drivable area to 1, the weight of the lane line to 1, and use the SGD optimizer for optimization training.
[0079] Step 7.2: The accuracy of drivable area detection is evaluated using MIoU, which is the average of the ratio of the intersection to the union of the predicted results and the actual values in all categories. The calculation formula is shown in Equation (6).
[0080]
[0081] In the formula, k+1 represents the semantic segmentation category + background; p ij This represents the number of pixels that belong to category i but are predicted to belong to category j; p ii This represents the number of pixels that belong to the i-th category and are predicted to be in the i-th category, equivalent to the TP of the confusion matrix, because p ii Since p is calculated once by each of the first two terms in the denominator, it needs to be subtracted from the denominator once more. ii .
[0082] Step 7.3: Select Lane Accuracy and IoU as evaluation metrics for lane detection based on ENet-SAD. Lane Accuracy is defined as the ratio of correctly predicted lane lines to all actual lane lines, and the calculation formula is shown in Equation (7). IoU is defined as the ratio of the intersection and union of the predicted results and the actual values, and the calculation formula is shown in Equation (8).
[0083]
[0084]
[0085] In the formula, k represents the semantic segmentation category; p ij This represents the number of pixels that originally belonged to category i but were predicted to belong to category j; p ii This represents the number of pixels that belong to category i and are predicted to be category i, which is equivalent to TP in the confusion matrix.
[0086] The above description, in conjunction with preferred technical solutions, provides a further detailed explanation of the present invention and should not be construed as limiting the specific implementation of the invention to these descriptions. For those skilled in the art, simple deductions and substitutions can be made without departing from the concept of the present invention, and all such deductions and substitutions should be considered within the scope of protection of the present invention.
Claims
1. A method for constructing a drivable region and lane line detection model based on semantic segmentation, characterized in that... Includes the following steps: Step 1: Improve the YOLOv5s model structure by proposing an improved YOLOv5s model structure to enhance the model's target detection capability. The specific steps are as follows: Step 1: Add a module to the FPN structure of the original YOLOv5s model to enable the Neck end to fully utilize the shallow high-resolution features of the P2 level. Step 2: Replace the P5 level feature W / 32×H / 32 in YOLOv5s with the P2 level high-resolution feature W / 4×H / 4 to improve the ability to detect small targets; Step 2: Establish the drivable area and lane line detection model structure. This model mainly consists of a backbone network, a neck structure, a drivable area detection head, and a lane line detection head. The establishment of the drivable area and lane line detection model structure specifically includes the following steps; Step 1: Design a backbone network consisting of Focus and CBS_CSP modules to extract features from the input image; Step 2: Design the neck structure using feature pyramid network and path aggregation network to achieve feature fusion at different scales; Step 3: Remove the PANet structure from the Detection Head and Neck of the improved YOLOv5s model; Step 4: Add driving area and lane line segmentation branches to the FPN structure in the Neck end of the improved YOLOv5s model to achieve a simple and efficient segmentation network; Step 5: Design the drivable area detection head and lane line detection head, and transform the width and height of the feature map back to the resolution of the input image; Step 3: After cropping the original input image, normalize it by standardizing each RGB channel; Step 4: Design the YOLOv5 model for drivable area and lane line detection, including model inference and post-processing; Step 5: Design the loss function for the drivable area and lane line detection model, analyze the training quality of the model based on the magnitude of the loss value, and determine whether the model has converged; Step 6: Construct a dataset for detecting drivable areas and lane lines for autonomous vehicles, and design segmentation labels for drivable areas and lane lines; Step 7: Train the drivable area and lane line detection model, and design evaluation metrics for the drivable area and lane line detection model.
2. The method for constructing a drivable region and lane line detection model based on semantic segmentation as described in claim 1, characterized in that... In step 3, the original input image is cropped and normalized. The specific steps are to crop the original input image to a resolution of W×H, normalize the input image, scale the pixel values of the three RGB channels [0, 255] to the range of [0, 1], and normalize each channel.
3. The method for constructing a drivable region and lane line detection model based on semantic segmentation as described in claim 1, characterized in that... In step 4, the model inference and post-processing of the YOLOv5 model for designing drivable areas and lane line detection specifically include the following steps: Step 1: After the input image passes through the drivable area or lane line detection model, an output feature map of the same size as the input image is generated. The semantic probability output is generated according to the Sigmoid formula and the output is mapped to the range [0, 1]. The second step is to convert the semantic classification probability into pixel categories, adjust the segmentation map to the size of the original image using bilinear interpolation, and encode the classification results into different colors on the original image.
4. The method for constructing a drivable region and lane line detection model based on semantic segmentation as described in claim 1, characterized in that... In step 5, the specific steps for designing the loss function of the drivable area and lane line detection model, analyzing the quality of model training based on the magnitude of the loss value, and determining whether the model has converged include: Step 1: The binary cross-entropy (BCE) loss function is used for calculation, and the sigmoid function is used to transform the input data of the BCE loss function to the range [0, 1]. The loss of the entire image is the average of the loss of each pixel. Step 2: Semantic segmentation loss is derived from drivable region detection loss. Lane detection loss It consists of two parts, and uses the same loss function to calculate the segmentation loss.
5. The method for constructing a drivable region and lane line detection model based on semantic segmentation as described in claim 1, characterized in that... In step 6, the specific steps for constructing the drivable area and lane line detection dataset for autonomous vehicles and designing drivable area and lane line segmentation labels are as follows: Step 1: Connect the coordinates of the drivable area directly, fit the curve part with a cubic Bézier curve, and then fill the closed area to obtain the drivable area segmentation label. The second step is to directly connect the coordinates of the lane lines, fit the lane line curve using a polynomial curve, and calculate the center line of the two lines as the new lane line according to the ENet-SAD method, thus obtaining the final lane line label.
6. The method for constructing a drivable region and lane line detection model based on semantic segmentation as described in claim 1, characterized in that... In step 7, the specific steps for training the drivable area and lane line detection model and designing the evaluation index for the drivable area and lane line detection model include: Step 1: Set the initial learning rate to 0.001, the number of iterations to 125, the batch size to 16, the weight of the drivable area to 1, the weight of the lane line to 1, and use the SGD optimizer for optimization training. Step 2: Driving area detection uses MIoU as the accuracy evaluation index, which is the average of the ratio of the intersection to the union of the predicted results and the actual values in all categories; Step 3: Based on ENet-SAD, lane accuracy and IoU are selected as evaluation metrics for lane line detection; lane accuracy is defined as the ratio of correctly predicted lane lines to all actual lane lines.