METHOD FOR LANE AND ROAD DETECTION
Patent Information
- Application Number
- DE502019012392
- Authority / Receiving Office
- DE · DE
- Patent Type
- Patents
- Current Assignee / Owner
- Priority Date
- 2018-08-07
- Filing Date
- 2019-05-06
- Publication Date
- 2024-11-21
- Estimated Expiration
- 2039-05-06
AI Technical Summary
Existing lane and roadway detection methods face challenges in accurately recognizing boundaries, especially when markings are missing or obscured by dynamic objects, leading to difficulties in extracting reliable lane and road information.
A multitask deep neural network (DNN) architecture comprising a convolutional neural network (CNN) encoder and two decoders for semantic segmentation and key point regression is used to provide pixel-precise classification and sub-pixel-precise boundary detection, with redundancy for estimating detection confidence.
Enables reliable detection of lanes and roadways in challenging scenarios by combining semantic segmentation and key point regression outputs, enhancing detection confidence and accuracy for downstream applications like lane departure warning systems.
Description
[0001] The invention relates to a method for detecting lanes and roadways.
[0002] For example, state-of-the-art methods are known that detect lane markings or lane boundaries and orient themselves accordingly. In known methods, all driver assistance functions are also aligned to the lane markings and / or lane boundaries.
[0003] A disadvantage of the existing methods is that they exclusively use the markings and / or boundaries as a reference for the lane or roadway layout, which often leads to problems when the markings are missing or not visible. Without correctly recognized markings, a lane departure warning system, for example, cannot function.
[0004] Furthermore, certain methods, such as semantic segmentation and the regression approach, are known from the state of the art.
[0005] Semantic segmentation, i.e., pixel-by-pixel classification, is a frequently used approach with high classification performance. However, extracting the relevant information, namely lane / road boundaries, is not always easy or unambiguous, even from correctly classified image regions, because such systems typically depict visible regions, e.g., a road, but not its boundaries, such as the roadside.
[0006] Especially in the presence of dynamic objects such as vehicles that obscure lane or road boundaries, the latter are omitted, which significantly complicates the subsequent extraction of the boundary from the pixel-labeled image material.
[0007] The use of regression approaches offers the possibility of directly outputting boundaries in parametric representation. Such an approach is capable of interpolating intermediate values from the image context when the boundary is hidden or uncertain.
[0008] However, it is difficult to identify in which areas the network provides correct detection and in which areas the network has interpolated.
[0009] The publication "Marvin Teichmann et al.: "MultiNet: Real-time Joint Semantic Reasoning for Autonomous Driving" describes the use of a system consisting of a CNN encoder and three downstream parallel decoders. This system is intended to provide an approach for joint classification, detection, and semantic segmentation using a common architecture.
[0010] It is therefore an object of the present invention to provide a method for lane and roadway detection which enables reliable detection of lanes and roadways even in difficult scenarios.
[0011] This problem is solved by the subject matter of independent claim 1.
[0012] Further advantageous embodiments and embodiments of the invention are the subject of the subclaims.
[0013] Initial considerations included using a deep neural network (DNN) to reliably detect the lane, the vehicle's own and adjacent lanes, and the accessible road area. This network enables a high detection rate even in challenging scenarios. This also includes estimating detection confidence.
[0014] According to the invention, a method for lane and roadway detection is proposed, wherein a multitask DNN architecture is used for detection, which consists of at least one encoder and at least one first and one second decoder, comprising the following steps: Providing an input image by means of an optical detection device, filtering the input image by means of the encoder, generating a first representation by the encoder, forwarding the first representation to the first and second decoders, processing the first representation in the first and second decoders, outputting two further different representations of the first and second decoders, merging the two representations of the first and second decoders, outputting detected lanes and lane boundaries and / or the roadway.
[0015] The encoder is preferably a convolutional neural network (CNN). The first and second decoders can also be CNNs. The input image can be an RGB or YUV image, or even a grayscale image. The detected lanes or the detected roadway and their respective boundaries can be output to a driver assistance system, such as a lane keeping assistant. It would also be conceivable to use the output data for autonomous driving.
[0016] In a preferred embodiment, the encoder representation provides learned features in the form of convolution filters. These convolution filters are particularly relevant for lane detection. Edge detection is possible thanks to the convolution filters.
[0017] According to the invention, the first decoder performs semantic segmentation, and the second decoder performs keypoint regression. Semantic segmentation determines a clear area of the lane and / or roadway. An additional keypoint regression directly detects the clearly visible boundaries of the lane and / or roadway and enables the interpolation of uncertain or obscured boundaries.
[0018] Preferably, the representation of the first decoder provides a pixel-precise classification of the input image of a drivable area. The drivable area is preferably a roadway and / or lane. This allows classification into various classes such as first lane, left adjacent lane, right adjacent lane, second left adjacent lane, etc., as well as the classification of the entire drivable roadway. Currently, a binary image with the classification result is output for each class.
[0019] In a preferred embodiment, the representation of the second decoder provides visible boundaries of the lane and / or roadway as continuous value pairs in image coordinates. The lane and / or roadway boundaries are thus output as continuous value pairs in image coordinates using so-called key points (xi, yi) of a number k, where 1<=i<=k. This results in a subpixel-accurate output, since the CNN of the second decoder, in contrast to the CNN of the first decoder, provides a continuous output in the form of floating-point values.
[0020] The k key points, each of which contains two values for x and y position in the image, are divided into an approximation of the left and right lane boundaries, i.e. k / 2 key points per side.
[0021] The key points are specified in 2D image coordinates. However, the approach can also be converted to 3D world coordinates.
[0022] As a result of the entire DNN, two different representations of the lanes and / or roadway are available.
[0023] Preferably, a recognition confidence for the respective lane and / or roadway boundary is estimated based on the different representations of the first and second decoder.
[0024] Based on these two different representations, the DNN provides a certain degree of redundancy. This redundancy can be used to estimate the DNN's detection confidence for the respective lane boundary.
[0025] The more accurately the respective lane / road boundary, which is provided by the KeyPoint output of the second decoder, replicates the edge area of the pixel-wise classification of the first decoder of the lane / road, the more reliable the output of the DNN and the higher the recognition confidence, and any downstream function such as lane controller can react accordingly.
[0026] The shared encoder with two separate decoders for different representations of the lane and / or roadway enables features to be learned in the encoder during training that are advantageous for both representations.
[0027] Thus, the representations support each other in finding these features and additionally allow an increased generalization capability of the proposed DNN, since the found features must supply two different representations.
[0028] Further advantageous embodiments and embodiments are apparent from the figures, in which: Fig. 1 : an example representation of a roadway recognized by semantic segmentation; Fig. 2 : a schematic exemplary representation of an embodiment of the invention; Fig. 3 : a further schematic exemplary representation of a further embodiment of the invention; Fig. 4 : a representation of a roadway with lanes detected according to an embodiment of the invention; Fig. 5 : a schematic flow diagram of a method according to the invention.
[0029] Figure 1shows an example of a roadway 1 detected by semantic segmentation. It is clearly visible that the road boundaries are not explicitly detected, but only the area between them. Especially in the presence of dynamic objects 2, such as vehicles that obscure lane or road boundaries, the latter are omitted, which significantly complicates the subsequent extraction of the boundaries from the pixel-labeled image material. The right road edge in Figure 1 cannot be clearly extracted from pixel-wise segmentation.
[0030] Figure 2shows a schematic exemplary representation of an embodiment of the invention. This shows an exemplary multitask DNN architecture, which consists of a CNN encoder, a semantic segmentation decoder, and a decoder for key-point regression. The semantic segmentation decoder provides a pixel classification as output, and the key-point regression provides key points as output. These two outputs are fused together, and a confidence estimation is performed. By merging the two representations, it is possible to estimate in which areas the network output is reliable and in which areas interpolation was carried out and / or uncertainty in the detection exists.
[0031] Figure 3shows a further schematic exemplary representation of another embodiment of the invention. This shows a generic structure of a Deep Neural Network (DNN), which can be used for reliable detection of a lane and adjacent lanes as well as lane and / or roadway boundaries. The components correspond to the structure of Figure 2 .
[0032] Figure 4shows a representation of a roadway 1 with lanes S1-S3 detected according to an embodiment of the invention. Here, the roadway 1 was divided into ego lane S2, left adjacent lane S3, and right adjacent lane S1 through pixel-by-pixel classification. Furthermore, the key points KPl, KPr supplied by the key point regression decoder for the left and right lane markings of the ego lane S2 are shown. These key points were connected to form a line and thus represent the lane marking. The reference symbol KPl denotes the key points located to the left in the direction of travel, and the reference symbol KPr denotes the key points located to the right in the direction of travel.
[0033] Figure 5shows a schematic flow diagram of a method according to the invention. In method step VS1, an input image is provided by an optical detection device. This input image is filtered by the encoder in a subsequent method step VS2. In a further step VS3, a first representation of the lane and / or roadway is generated by the encoder. In step VS4, the first representation is forwarded to the first and second decoders. This first representation is processed in the first and second decoders in step VS5.
[0034] In step VS6, two additional different representations of the first and second decoders are output. In step VS7, the two representations of the first and second decoders are combined. In a final step VS8, the detected lanes and lane markings, as well as the road surface, are output. List of reference symbols
[0035] 1Roadway 2Object S1-S3Lanes KPlKey-Points left KPrKey-Points right VS1-VS8Procedure steps
Claims
1. Method for lane (S1-S3) detection and road (1) detection, wherein a multitask DNN architecture consisting of at least one encoder and at least a first and a second decoder is used for detection, comprising the following steps of: - providing (VS1) an input image by means of an optical capture device, - filtering (VS2) the input image by means of the encoder, - generating (VS3) a first representation of the lane (S1-S3) and / or road (1) by means of the encoder, - forwarding (VS4) the first representation to the first and second decoders, - processing (VS5) the first representation in the first and second decoders, - outputting (VS6) two further different representations from the first and second decoders, wherein the first decoder carries out a semantic segmentation and the second decoder carries out a key point regression, and wherein the first decoder provides a pixel classification and the second decoder provides key points as the output, wherein the pixel classification provides a free area of the road and / or lane, and the key points provide visible boundaries of the road and / or lane, - combining (VS7) the two representations from the first and second decoders, wherein the two representations are fused when combining them, and - outputting (VS8) identified lanes (S1-S3) and lane boundaries and the road (1).
2. Method according to Claim 1, wherein the representation from the encoder provides learnt features in the form of convolution filters.
3. Method according to Claim 1, wherein the representation from the first decoder provides a pixel-accurate classification of the input image of a navigable region.
4. Method according to Claim 3, wherein the navigable region is a road (1) and / or a lane (S1-S3).
5. Method according to Claim 1, wherein the representation from the second decoder provides visible boundaries of the lane (S1-S3) and / or of the road (1) as continuous pairs of values in image coordinates.
6. Method according to one of the preceding claims, wherein an identification confidence for the respective lane and / or road boundary is estimated on the basis of the different representations from the first and second decoders.
