Methods for detecting lanes and roads
Through the multi-task deep neural network architecture, combined with semantic segmentation method and key point regression method, the accuracy problem in the existing technology is solved, and high recognition rate and high confidence detection in difficult situations is achieved .
Patent Information
- Application Number
- CN201980024009.7
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Priority Date
- 2018-08-07
- Filing Date
- 2019-05-06
- Publication Date
- 2025-02-07
- Estimated Expiration
- 2039-05-06
AI Technical Summary
The prior art is difficult to accurately identify the boundaries between lanes and roads when detecting lanes and roads, especially when markings are missing or blocked, resulting in instability of auxiliary systems.
Using a multi-task deep neural network (DNN) architecture, including an encoder and two decoders, different representations of lanes and roads are generated through semantic segmentation and key point regression, and the accuracy of detection is improved by combining these representations and reliability.
It realizes the reliable detection of lanes and roads in difficult situations, improves the recognition rate and detection confidence, and enhances the stability of the auxiliary system.
Smart Images
Figure CN111937004B_ABST
Abstract
Description
Technical Field
[0001] The invention relates to a method for detecting lanes and roads. Background Art
[0002] From the prior art, for example, methods for recognizing lane markings or road boundaries and being guided by them are known. In the known methods, all driver assistance functions are also guided by lane markings and / or road boundaries.
[0003] A disadvantage of the known methods is that they use only markings and / or boundaries as a reference for the lane or road course, which often leads to problems when markings are missing or not visible. For example, a lane keeping assist system cannot be operated without correctly identified markings.
[0004] Furthermore, specific methods such as semantic segmentation methods and regression methods are known from the prior art.
[0005] Semantic segmentation, or pixel-by-pixel classification, is a common approach with high classification performance. However, even extracting relevant information such as lane / road boundaries from correctly classified image regions is not always convenient or clear, as such systems typically image visible areas such as a road, but not boundaries such as the edge of the road.
[0006] In particular, when there are dynamic objects such as vehicles that obscure / occlude lane boundaries or road boundaries, boundaries such as road edges are ignored in the imaging, which makes it more difficult to subsequently extract the corresponding boundaries from the pixel-labeled image material.
[0007] The use of regression methods offers the possibility to output the boundaries directly in parametric form. In cases where the boundaries are obscured or uncertain, this approach can interpolate intermediate values from the image context.
[0008] However, it is difficult to identify in which areas the network provides correct detection and in which areas network interpolation has been implemented. Summary of the invention
[0009] It is therefore an object of the present invention to provide a method for detecting lanes and roads which enables reliable detection of lanes and roads even in difficult situations.
[0010] This object is achieved by the subject matter of independent claim 1 .
[0011] Further advantageous embodiments and embodiments of the invention are the subject matter of the dependent claims.
[0012] The first thing to consider is the reliable detection of lanes, own lane and adjacent lanes as well as drivable road areas using deep neural networks (DNNs) that achieve high recognition rates even in difficult situations. This also includes analytical evaluation of the detection confidence.
[0013] According to the present invention, a method for detecting lanes and roads is proposed, wherein the detection is performed using a multi-task DNN (Deep Neural Network) architecture comprising at least one encoder and at least one first decoder and one second decoder, the method comprising the following steps:
[0014] - providing an input image by means of an optical detection device,
[0015] - filter the input image with the help of the encoder,
[0016] - generating a first representation by an encoder,
[0017] - forwarding the first representation to the first decoder and the second decoder,
[0018] - processing the first representation in the first decoder and the second decoder,
[0019] - output two other different representations of the first decoder and the second decoder,
[0020] - merging the two representations of the first decoder and the second decoder,
[0021] - Output of detected lanes, lane limits and / or roads.
[0022] The encoder is preferably a convolutional neural network (CNN). The first decoder and the second decoder can also be convolutional neural networks (CNN). The input image can be an image in a three-primary color light pattern (RGB) or YUV format, or it can also be a grayscale image. The identified lane or the identified road and the corresponding associated boundaries can be output to a driver assistance system such as a lane keeping assistance system, for example. It is also conceivable to use the output data for autonomous driving.
[0023] In a preferred embodiment, the representation of the encoder provides learned features in the form of convolutional filters. These convolutional filters are particularly relevant for lane detection. Edges can be identified by convolutional filters.
[0024] In a particularly preferred embodiment, the first decoder performs a semantic segmentation method and the second decoder performs a keypoint regression method. The unoccupied area of the lane and / or road is determined with the aid of the semantic segmentation method. The additional keypoint regression method allows the direct detection of clearly visible lane and / or road boundaries and the interpolation of uncertain or obscured boundaries.
[0025] The first decoder characterization preferably provides a pixel-accurate classification of the drivable area input image. The drivable area is preferably a road and / or a lane. It can be classified into different categories such as the lane where the vehicle is located, the adjacent left lane, the adjacent right lane, the adjacent second left lane, etc., as well as the entire drivable road. Currently, for this output, each category output includes a binary image of the classification result.
[0026] In a preferred embodiment, the second decoder characterizes the visible boundaries of the lane and / or road as continuous value pairs in image coordinates. The lane and / or road boundaries are respectively represented by a number k of so-called key points (x i ,y i ) is output as a continuous value pair in image coordinates, where 1<=i<=k. In contrast to the convolutional neural network (CNN) of the first decoder, since the convolutional neural network (CNN) of the second decoder provides continuous output in the form of floating point values, an output accurate to sub-pixel can be obtained here.
[0027] The k keypoints, which contain two values of the x and y position in the image respectively, are here divided into the approximations of the left road limit and the right road limit, i.e. k / 2 keypoints on each side.
[0028] Here, the keypoints are given in 2D image coordinates. However, the method can also be converted to 3D world coordinates.
[0029] As a result of the entire deep neural network (DNN), lanes and / or roads respectively have two different representations.
[0030] Based on the different characterizations of the first decoder and the second decoder, a recognition confidence of the respective lane and / or road boundary is preferably estimated.
[0031] Based on these two different representations, the DNN itself provides a certain amount of redundancy, which can be used to estimate the DNN recognition confidence of the corresponding lane boundary.
[0032] The more accurate the corresponding lane / road boundaries of the edge areas classified pixel by pixel by the first decoder simulating the lane / road provided by the key point output of the second decoder, the more reliable the output of the deep neural network (DNN), the higher the recognition confidence, and possible subsequent functions such as driving path control can respond accordingly on this basis.
[0033] By using a common encoder for two separate decoders for different representations of lanes and / or roads, features that are beneficial for both representations can be learned in the encoder during training.
[0034] Therefore, the representations support each other in finding these features and also allow the proposed deep neural network (DNN) to have enhanced generalizability since the found features must provide two different representations. BRIEF DESCRIPTION OF THE DRAWINGS
[0035] Other useful designs and implementations can be found in the accompanying drawings. Among them:
[0036] Figure 1 : This is an exemplary illustration of roads identified by semantic segmentation;
[0037] Figure 2 : This is an exemplary schematic diagram of an embodiment of the present invention;
[0038] Figure 3 : This is another exemplary schematic diagram of another embodiment of the present invention;
[0039] Figure 4 : This is a road diagram with lanes identified according to an embodiment of the present invention;
[0040] Figure 5 : This is a schematic diagram of the process according to the method of the present invention. DETAILED DESCRIPTION
[0041] Figure 1 An exemplary illustration of a road 1 identified by semantic segmentation is shown. It can be clearly seen here that the road boundaries cannot be identified unambiguously, only the areas between the boundaries. In particular, when there are dynamic objects 2 such as vehicles that obscure lane or road boundaries, boundaries such as road edges are ignored in the imaging, which makes it more difficult to subsequently extract the corresponding boundaries from the pixel-labeled image material. Figure 1 The right edge of the road.
[0042] Figure 2 An exemplary schematic diagram showing an embodiment of the present invention. Here, an exemplary multi-task deep neural network (DNN) architecture is shown, which consists of a convolutional neural network (CNN) encoder, a semantic segmentation decoder, and a decoder for keypoint regression. The semantic segmentation decoder provides pixel classification as output, and the keypoint regression method provides keypoints as output. The two outputs are fused and confidence assessed. By merging the two representations, it is possible to estimate in which areas the network output is reliable, and in which areas interpolation is performed and / or there is uncertainty in the detection.
[0043] Figure 3Another exemplary schematic diagram showing another embodiment of the present invention. Here, the general structure of a deep neural network (DNN) is shown, which can be used to reliably detect lanes, adjacent lanes, and lane and / or road boundaries. The corresponding parts correspond to Figure 2 structure.
[0044] Figure 4 A diagram of a road 1 with lanes S1-S3 identified according to an embodiment of the present invention is shown. Here, the road 1 is divided into the host lane S2, the adjacent left lane S3 and the adjacent right lane S1 by pixel classification. In addition, for the left and right lane markings of the host lane S2, key points KPl, KPr provided by the key point regression decoder are shown, which have been connected to form a line and thus represent the lane markings. Here, the reference numeral KPl describes a key point located on the left side of the driving direction, and the reference numeral KPr describes a key point located on the right side of the driving direction.
[0045] Figure 5 A schematic flow chart of the method according to the invention is shown. In method step VS1, an input image is provided by means of an optical detection device. The input image is filtered by means of an encoder in a subsequent method step VS2. In a further step VS3, a first representation of a lane and / or road is generated by the encoder. In step VS4, the first representation is forwarded to a first decoder and a second decoder. In step VS5, the first representation is processed in the first decoder and the second decoder.
[0046] In step VS6, two further different representations of the first decoder and the second decoder are output. In step VS7, the two representations of the first decoder and the second decoder are merged. In the final step VS8, the identified lanes and lane limits and roads are output.
[0047] Reference Mark List
[0048] 1 Road
[0049] 2 Objectives
[0050] Lanes S1-S3
[0051] KPl Left Key Point
[0052] KPr right key point
[0053] VS1-VS8 Method Steps
Claims
1. A method for detecting lanes (S1-S3) and roads (1), wherein: The detection is performed using a multi-task deep neural network architecture including at least one encoder and at least one first decoder and at least one second decoder, the method comprising the following steps: - providing (VS1) an input image by means of an optical detection device, - filter the input image with the help of the encoder (VS2), - generating (VS3) a first representation of the lane (S1-S3) and / or the road (1) by means of an encoder, - forwarding (VS4) the first representation to the first decoder and the second decoder, - processing (VS5) the first representation in the first decoder and the second decoder, - output (VS6) two further different representations of the first decoder and the second decoder, wherein the first decoder performs a semantic segmentation method and the second decoder performs a keypoint regression method, wherein the first decoder provides pixel classification as output and the second decoder provides keypoints as output, wherein the pixel classification provides roads and / or lanes and the keypoint regression method provides visible boundaries of lanes and / or roads, - merging (VS7) the two representations of the first decoder and the second decoder, wherein the pixel classification and the keypoints are fused in the merging, and - Output (VS8) the identified lanes (S1-S3), lane boundaries and road (1).
2. The method according to claim 1, in, The encoder representation provides learned features in the form of convolutional filters.
3. The method according to claim 1, in, The first decoder's representation provides a pixel-accurate classification of the input image's drivable area.
4. The method according to claim 3, in, The drivable area is a road (1) and / or a lane (S1-S3).
5. The method according to claim 1, in, The representation of the second decoder provides the visible limits of the lanes (S1-S3) and / or the road (1) as pairs of consecutive values in image coordinates.
6. A method according to any one of the preceding claims, in, A recognition confidence level of the corresponding lane and / or road boundary is estimated based on the different representations of the first decoder and the second decoder.