Lane line detection method and device based on difficulty adaptive adjustment
By combining the CLRNet model with the difficulty prediction module, adaptive adjustment of the lane line detection method is achieved, which improves detection speed and accuracy and solves the problem of detection resource waste in different scenarios.
Patent Information
- Application Number
- CN202311270852.8
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2023-09-27
- Publication Date
- 2025-09-16
- Estimated Expiration
- 2043-09-27
AI Technical Summary
Existing lane line detection methods are difficult to adaptively adjust according to the detection difficulty in different scenarios, resulting in insufficient detection accuracy and slow speed in high-difficulty scenarios, and waste of computing resources in low-difficulty scenarios.
The lane detection model is constructed using the CLRNet model, introducing a difficulty prediction module and a lightweight network structure. Through adaptive adjustment of shallow and deep head networks, combined with separable convolution and uncertainty loss functions, the image difficulty is evaluated and the model structure is adjusted to meet the detection needs of different scenarios.
The speed and accuracy of lane line detection are improved, and the waste of computing resources is reduced, especially by accelerating detection in low-difficulty scenarios while maintaining high accuracy in high-difficulty scenarios.
Smart Images
Figure CN117315608B_ABST
Abstract
Description
Technical Field
[0001] The present invention belongs to the fields of computer vision and autonomous driving systems. Background Art
[0002] With the rapid development of autonomous driving technology, lane detection has become a key task in autonomous driving systems, providing crucial spatial positioning and path planning information. Traditional lane detection uses image segmentation to segment lane instances. However, such models often have large parameter counts and slow inference speed, limiting their practical applications. Consequently, approaches such as LaneATT and CLRNet have emerged that interpret lane lines as planar straight lines. These methods approximate the ground truth lane lines by predicting the starting point coordinates and slope of each line, and fit curved lane lines by predicting a series of lateral offsets for each line, significantly improving the speed and performance of lane detection. However, lane detection varies in different scenarios, with detection particularly challenging in some scenarios where lane lines are obscured or worn. Traditional methods focus on designing model networks for challenging scenarios, treating all scenarios equally. No work has yet analyzed how to use detection difficulty to guide model detection.
[0003] Predicting detection difficulty has precedent in other areas of deep learning, and it is also feasible for lane detection tasks. Lane detection scenarios vary significantly in difficulty. The performance of existing methods shows that the most challenging scenarios often have distinct characteristics, such as lane lines with large curvature turns or lane lines without obvious markings on the ground. In contrast, less difficult scenarios have clearly marked lane lines on the ground that are very close to straight lines. Therefore, the model can be used to adaptively distinguish the lane detection difficulty of the input image. High-speed vehicles often travel on wide, well-marked roads, so speed is more important. In complex scenarios, autonomous driving systems require higher detection accuracy to locate lane lines. Furthermore, experimental observations by the present invention show that in most scenarios, existing method models can accurately predict lane lines using a portion of their model structure, while the full model is only required in some difficult scenarios. Taking the CLRNet model as an example, its complete model uses the three-layer structure of FPN and three detection head networks for prediction, with an F1 score of up to 79.58%. If the present invention directly reads the output of the intermediate detection head network of the model, or retrains with this structure, the F1 scores reach 79.18% and 79.21%, respectively. This shows that in most scenarios, the output of the intermediate layer has the same effect as the complete structure. If a partial network structure is used for prediction of low-difficulty images, the detection speed can be greatly improved. At the same time, since the full network prediction is used for high-difficulty images, a significant decrease in network accuracy can also be avoided. Summary of the Invention
[0004] The present invention aims to solve one of the technical problems in the related art at least to a certain extent.
[0005] To this end, the purpose of the present invention is to propose a lane line detection method based on adaptive difficulty adjustment to improve the speed of lane line detection.
[0006] To achieve the above objectives, a first embodiment of the present invention provides a lane detection method based on adaptive difficulty adjustment, comprising:
[0007] Obtain the lane line image to be detected;
[0008] Building a lane detection model based on the CLRNet model; the lane detection model includes a backbone network, a head network, and a difficulty prediction module, wherein the head network includes a shallow head network and a deep head network;
[0009] Inputting the lane line image to be detected into the lane line detection model, and performing feature extraction on the lane line image to be detected through the backbone network to obtain image features;
[0010] Decoding the image features through the shallow head network to obtain a first detection result;
[0011] Predicting the lane line image to be detected by the difficulty prediction module to obtain a difficulty prediction score;
[0012] If the difficulty prediction score is greater than a preset threshold, the deep head network is used to detect the first detection result to obtain a detection result.
[0013] In addition, the lane line detection method based on adaptive difficulty adjustment according to the above embodiment of the present invention may also have the following additional technical features:
[0014] Furthermore, in one embodiment of the present invention, constructing a lane detection model based on the CLRNet model includes:
[0015] In the process of building the lane detection model, separable convolution is introduced instead of aggregate convolution to extract feature vectors; the sampling position is determined by point-slant anchor lines, and an offset is only added in the final prediction.
[0016] Furthermore, in one embodiment of the present invention, it further comprises:
[0017] Uncertainty loss is introduced during the lane detection model training process to assign a different weight to each row of lane points in the lane line image to be detected. The network then adaptively determines the weight ratio required for points at different positions when calculating the curve regression loss to suppress the difference in distance.
[0018] Furthermore, in one embodiment of the present invention, the difficulty prediction module predicts the lane line image to be detected to obtain a difficulty prediction score, including:
[0019] The difficulty prediction module is used to evaluate the detection difficulty of the lane line image to be detected, thereby adaptively adjusting the model structure for images of different difficulty levels to perform lane line detection.
[0020] Furthermore, in one embodiment of the present invention, after the difficulty prediction module predicts the lane line image to be detected and obtains a difficulty prediction score, the method further includes:
[0021] If the difficulty prediction score is less than a preset threshold, the first detection result is used as the detection result.
[0022] To achieve the above objectives, a second embodiment of the present invention provides a lane detection device based on adaptive difficulty adjustment, comprising the following modules:
[0023] An acquisition module is used to acquire the lane line image to be detected;
[0024] A construction module is used to build a lane detection model based on the CLRNet model; the lane detection model includes a backbone network, a head network, and a difficulty prediction module, wherein the head network includes a shallow head network and a deep head network;
[0025] An extraction module, configured to input the lane line image to be detected into a lane line detection model, and extract features of the lane line image to be detected through the backbone network to obtain image features;
[0026] A first detection module, configured to decode the image features through the shallow head network to obtain a first detection result;
[0027] A scoring module, configured to predict the lane line image to be detected using the difficulty prediction module to obtain a difficulty prediction score;
[0028] The second detection module uses the deep head network to detect the first detection result if the difficulty prediction score is greater than a preset threshold to obtain a detection result.
[0029] Furthermore, in one embodiment of the present invention, the building block is further configured to:
[0030] In the process of building the lane detection model, separable convolution is introduced instead of aggregate convolution to extract feature vectors; the sampling position is determined by point-slant anchor lines, and an offset is only added in the final prediction.
[0031] Furthermore, in one embodiment of the present invention, the building block is further configured to:
[0032] Uncertainty loss is introduced during the lane detection model training process to assign a different weight to each row of lane points in the lane line image to be detected. The network then adaptively determines the weight ratio required for points at different positions when calculating the curve regression loss to suppress the difference in distance.
[0033] Furthermore, in one embodiment of the present invention, the scoring module is further configured to:
[0034] The difficulty prediction module is used to evaluate the detection difficulty of the lane line image to be detected, thereby adaptively adjusting the model structure for images of different difficulty levels to perform lane line detection.
[0035] Furthermore, in one embodiment of the present invention, the scoring module is further configured to:
[0036] If the difficulty prediction score is less than a preset threshold, the first detection result is used as the detection result.
[0037] The lane line detection method based on adaptive difficulty adjustment proposed in an embodiment of the present invention first makes lightweight improvements to the network structure of the CLRNet model, and then adds an adaptive difficulty scoring module based on this improved network. For low-difficulty images, only part of the network is intercepted for prediction, while the entire network structure is retained for high-difficulty images, achieving adaptive model structure adjustment based on difficulty. In addition, since images are characterized by being larger near and smaller far, in order to better score the detection difficulty, the present invention adds differentiated weights for near and far points to the original curve regression loss function, weakening the negative impact of the large near and small far differences, thereby more accurately providing supervision signals for difficulty assessment and lane line detection. BRIEF DESCRIPTION OF THE DRAWINGS
[0038] The above and / or additional aspects and advantages of the present invention will become apparent and readily understood from the following description of the embodiments in conjunction with the accompanying drawings, in which:
[0039] Figure 1 A flowchart of a lane detection method based on adaptive difficulty adjustment provided by an embodiment of the present invention.
[0040] Figure 2 A schematic diagram of the network structure of a lane detection model provided by an embodiment of the present invention.
[0041] Figure 3 A schematic diagram of a lane detection device based on adaptive difficulty adjustment provided by an embodiment of the present invention. DETAILED DESCRIPTION
[0042] The following describes embodiments of the present invention in detail, examples of which are shown in the accompanying drawings, wherein the same or similar reference numerals throughout represent the same or similar elements or elements having the same or similar functions. The embodiments described below with reference to the accompanying drawings are exemplary and are intended to be used to explain the present invention, and are not to be construed as limiting the present invention.
[0043] The following describes a lane line detection method based on adaptive difficulty adjustment according to an embodiment of the present invention with reference to the accompanying drawings.
[0044] Figure 1 A flowchart of a lane detection method based on adaptive difficulty adjustment provided by an embodiment of the present invention.
[0045] like Figure 1 As shown, the lane line detection method based on difficulty adaptive adjustment includes the following steps:
[0046] S101: Acquire a lane line image to be detected;
[0047] S102: Build a lane detection model based on the CLRNet model. The lane detection model includes a backbone network, a head network, and a difficulty prediction module. The head network includes a shallow head network and a deep head network.
[0048] S103: Inputting the lane line image to be detected into the lane line detection model, and performing feature extraction on the lane line image to be detected through the backbone network to obtain image features;
[0049] S104: Decoding the image features through a shallow head network to obtain a first detection result;
[0050] S105: Predicting the lane line image to be detected by the difficulty prediction module to obtain a difficulty prediction score;
[0051] S106: If the difficulty prediction score is greater than a preset threshold, a deep head network is used to detect the first detection result to obtain a detection result.
[0052] Furthermore, in one embodiment of the present invention, a lane detection model is constructed based on the CLRNet model, including:
[0053] In the process of building the lane detection model, separable convolution is introduced instead of aggregate convolution to extract feature vectors; the sampling position is determined by point-slant anchor lines, and an offset is only added in the final prediction.
[0054] Specifically, inspired by the CLRNet network structure, the present invention designs a more lightweight network structure, such as Figure 2As shown. For each layer of feature maps coming out of FPN, the present invention abandons the cumbersome attention mechanism and introduces separable convolution instead of the aggregate convolution to extract feature vectors and ultimately predict the detection results, thereby sampling the feature map of the next layer of FPN. When sampling and predicting, unlike the traditional anchor line + offset method, the present invention only uses a point-slope anchor line to determine the sampling position, and only adds an offset in the final prediction. Such a lightweight design effectively improves the inference speed of the network model with only a slight decrease in the F1 score, which is more in line with the real-time requirements in actual scenarios of autonomous driving.
[0055] Furthermore, in one embodiment of the present invention, it further comprises:
[0056] Uncertainty loss is introduced during the lane detection model training process to assign a different weight to each row of lane points in the lane line image to be detected. The network then adaptively determines the weight ratio required for points at different positions when calculating the curve regression loss to suppress the difference in distance.
[0057] Unlike traditional target detection, lane lines, a special instance, often span most of the vertical axis range of the image. Therefore, the present invention adds a weight for the difference between near and far points to the L1 loss function used in existing lane line detection methods, thereby compensating for the problem of loss values being larger near and smaller far due to the transformation between the camera coordinate system and the world coordinate system. In specific implementation, the present invention introduces uncertainty loss to assign a different weight to each row of lane line points, and the network adaptively determines the weight ratio required for points in different positions when calculating the curve regression loss. Such a curve regression loss can effectively suppress the difference between near and far points, provide a more accurate supervision signal for the difficulty score, and thus effectively improve the accuracy of model reasoning.
[0058] Furthermore, in one embodiment of the present invention, the difficulty prediction module predicts the lane line image to be detected to obtain a difficulty prediction score, including:
[0059] The difficulty prediction module is used to evaluate the detection difficulty of the lane line image to be detected, so as to adaptively adjust the model structure for lane line detection for images of different difficulty levels.
[0060] Specifically, the present invention designs a plug-and-play adaptive difficulty scoring module and applies it to the above-mentioned lightweight design model. The difficulty scoring module can be inserted before the output of the intermediate detection head network of the network, adaptively judges the difficulty of detecting lane lines in the input image, and determines the position of the output detection result according to a preset threshold. If the prediction difficulty is less than the threshold, it means that the input image is easy to detect, and accurate prediction can be achieved using the output of the current layer head network; if the prediction difficulty is greater than the threshold, it means that the input image is difficult to detect, and it is necessary to open a deeper layer head network to extract features from a larger feature map to obtain a more accurate prediction. It should be emphasized that the difficulty scoring module of the present invention is not only applicable to the model designed by the present invention, but can be used for all models with an intermediate detection head network. Therefore, the module of the present invention can well adapt to the complex designs made by existing methods for high-difficulty scenarios, and assist these methods in effectively reducing the amount of calculation when predicting low-difficulty scenarios, further improving the overall reasoning speed.
[0061] Furthermore, in one embodiment of the present invention, after the difficulty prediction module predicts the lane line image to be detected and obtains the difficulty prediction score, the method further includes:
[0062] If the difficulty prediction score is less than a preset threshold, the first test result is used as the test result.
[0063] The lane line detection method based on adaptive difficulty adjustment proposed in an embodiment of the present invention first makes lightweight improvements to the network structure of the CLRNet model, and then adds an adaptive difficulty scoring module based on this improved network. For low-difficulty images, only part of the network is intercepted for prediction, while the entire network structure is retained for high-difficulty images, achieving adaptive model structure adjustment based on difficulty. In addition, since images are characterized by being larger near and smaller far, in order to better score the detection difficulty, the present invention adds differentiated weights for near and far points to the original curve regression loss function, weakening the negative impact of the large near and small far differences, thereby more accurately providing supervision signals for difficulty assessment and lane line detection.
[0064] In order to implement the above embodiment, the present invention also proposes a lane line detection device based on adaptive difficulty adjustment.
[0065] Figure 3 A schematic structural diagram of a lane detection device based on adaptive difficulty adjustment provided by an embodiment of the present invention.
[0066] like Figure 3 As shown, the lane line detection device based on adaptive difficulty adjustment includes: an acquisition module 100, a construction module 200, an extraction module 300, a first detection module 400, a scoring module 500, and a second detection module 600, wherein:
[0067] An acquisition module is used to acquire the lane line image to be detected;
[0068] A construction module for building a lane detection model based on the CLRNet model. The lane detection model includes a backbone network, a head network, and a difficulty prediction module. The head network includes a shallow head network and a deep head network.
[0069] The extraction module is used to input the lane line image to be detected into the lane line detection model, and extract features from the lane line image to be detected through the backbone network to obtain image features;
[0070] A first detection module is used to decode image features through a shallow head network to obtain a first detection result;
[0071] A scoring module is used to predict the lane line image to be detected through the difficulty prediction module to obtain a difficulty prediction score;
[0072] The second detection module uses a deep head network to detect the first detection result if the difficulty prediction score is greater than a preset threshold to obtain a detection result.
[0073] Furthermore, in one embodiment of the present invention, the building module is further configured to:
[0074] In the process of building the lane detection model, separable convolution is introduced instead of aggregate convolution to extract feature vectors; the sampling position is determined by point-slant anchor lines, and an offset is only added in the final prediction.
[0075] Furthermore, in one embodiment of the present invention, the building module is further configured to:
[0076] Uncertainty loss is introduced during the lane detection model training process to assign a different weight to each row of lane points in the lane line image to be detected. The network then adaptively determines the weight ratio required for points at different positions when calculating the curve regression loss to suppress the difference in distance.
[0077] Furthermore, in one embodiment of the present invention, the scoring module is further configured to:
[0078] The difficulty prediction module is used to evaluate the detection difficulty of the lane line image to be detected, so as to adaptively adjust the model structure for lane line detection for images of different difficulty levels.
[0079] Furthermore, in one embodiment of the present invention, the scoring module is further configured to:
[0080] If the difficulty prediction score is less than a preset threshold, the first test result is used as the test result.
[0081] In the description of this specification, the reference terms "one embodiment", "some embodiments", "example", "specific example", or "some examples" mean that the specific features, structures, materials or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the present invention. In this specification, the schematic representations of the above terms do not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described can be combined in any one or more embodiments or examples in a suitable manner. In addition, those skilled in the art can combine and combine different embodiments or examples described in this specification and features of different embodiments or examples without contradiction.
[0082] Furthermore, the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of the technical features being referred to. Thus, a feature defined as "first" or "second" may explicitly or implicitly include at least one such feature. In the description of the present invention, "plurality" means at least two, such as two, three, etc., unless otherwise specifically defined.
[0083] Although the embodiments of the present invention have been shown and described above, it will be understood that the above embodiments are illustrative and are not to be construed as limiting the present invention. A person skilled in the art may change, modify, replace, and modify the above embodiments within the scope of the present invention.
Claims
1. A lane detection method based on adaptive difficulty adjustment, characterized in that: The following steps are involved: Obtain the lane line image to be detected; A lane detection model is constructed based on the CLRNet model, including: introducing separable convolution instead of aggregate convolution to extract feature vectors during the lane detection model construction process; using point-slant anchor lines to determine sampling positions, and only adding offsets during the final prediction; the lane detection model includes a backbone network, a head network, and a difficulty prediction module, wherein the head network includes a shallow head network and a deep head network; Uncertainty loss is introduced during lane detection model training to assign a different weight to each row of lane points in the lane line image to be detected. The network then adaptively determines the weight ratio required when calculating the curve regression loss for points at different locations to suppress differences in distance. Inputting the lane line image to be detected into the lane line detection model, and performing feature extraction on the lane line image to be detected through the backbone network to obtain image features; Decoding the image features through the shallow head network to obtain a first detection result; Predicting the lane line image to be detected by the difficulty prediction module to obtain a difficulty prediction score; If the difficulty prediction score is greater than a preset threshold, the deep head network is used to detect the first detection result to obtain a detection result; If the difficulty prediction score is less than a preset threshold, the first detection result is used as the detection result.
2. The method according to claim 1, characterized in that The predicting the lane line image to be detected by the difficulty prediction module to obtain a difficulty prediction score includes: The difficulty prediction module is used to evaluate the detection difficulty of the lane line image to be detected, thereby adaptively adjusting the model structure for images of different difficulty levels to perform lane line detection.
3. A lane line detection device based on adaptive difficulty adjustment, characterized in that: include: An acquisition module is used to acquire the lane line image to be detected; A construction module for building a lane detection model based on the CLRNet model, including: introducing separable convolution instead of aggregate convolution to extract feature vectors during the lane detection model construction process; determining sampling positions using point-slant anchor lines, and only adding offsets during the final prediction; the lane detection model includes a backbone network, a head network, and a difficulty prediction module, where the head network includes a shallow head network and a deep head network; The building block is further used to: Uncertainty loss is introduced during lane detection model training to assign a different weight to each row of lane points in the lane line image to be detected. The network then adaptively determines the weight ratio required when calculating the curve regression loss for points at different locations to suppress differences in distance. An extraction module, configured to input the lane line image to be detected into a lane line detection model, and extract features of the lane line image to be detected through the backbone network to obtain image features; A first detection module, configured to decode the image features through the shallow head network to obtain a first detection result; A scoring module, configured to predict the lane line image to be detected using the difficulty prediction module to obtain a difficulty prediction score; The second detection module uses the deep head network to detect the first detection result to obtain a detection result if the difficulty prediction score is greater than a preset threshold; and uses the first detection result as the detection result if the difficulty prediction score is less than the preset threshold.
4. The device according to claim 3, characterized in that The scoring module is further used to: The difficulty prediction module is used to evaluate the detection difficulty of the lane line image to be detected, thereby adaptively adjusting the model structure for images of different difficulty levels to perform lane line detection.