Apparatus and method for searching a lane in which a vehicle can travel

Abstract

Apparatus and method for searching for a lane in which a vehicle can travel. An apparatus for searching for a lane in which a vehicle can travel, wherein the apparatus is configured to receive an image taken by a camera, the image showing an area in front of the vehicle, to detect lane markings in the image, to determine, for each of the lane markings, whether the respective lane marking is of a first type representing a first type of road condition or of a second type representing a second type of road condition, to create candidate lanes from the lane markings, to classify the candidate lanes into candidate lane classes depending on the type of the lane markings of the candidate lanes, and to search for a lane in which the vehicle can travel from the candidate lane classes.

Description

[0001] This application is a divisional application of patent application filed on May 10, 2019, with application number 201910387702.2 and invention title "Apparatus and Method for Searching for Lanes Where Vehicles Can Drive". Technical Field

[0002] This invention relates to an apparatus and method for searching lanes in which vehicles can travel. Background Technology

[0003] Lane detection (LD) systems provide crucial sensing capabilities for lane departure warning (LDW) systems, lane keeping assist (LKA) systems, or autonomous driving. In situations where modern lane detection systems can detect multiple lane markings, certain scenarios become challenging, such as construction sites where multiple lane markings (e.g., white and yellow) are painted on the road. For successful lane detection in a construction site area, the lane detection system needs to accurately estimate all markings of their respective colors and, for the application, estimate the correct autonomous lane the vehicle should be traveling in.

[0004] Even when multiple lane markings of the same color are applied, correctly selecting the corresponding lane marking and adjacent lane markings is not always straightforward. Furthermore, there are situations where lane markings are missing or incorrectly detected. In such cases, selecting the correct and most reasonable lane marking in chaotic situations becomes challenging.

[0005] Conventional single-lane detection systems cannot provide solutions for these challenging situations. For example, when vehicles pass through a construction site, a common solution in conventional lane detection systems is to shut down the system during construction when a yellow lane marking is detected. This avoids false detections, but it also limits the system's availability. Summary of the Invention

[0006] A fundamental objective of this invention is to provide an apparatus for searching lanes that vehicles can travel even under road conditions that differ from normal road conditions (e.g., in a construction site area). Another objective of this invention is to provide a system comprising an apparatus and method for searching lanes that vehicles can travel in.

[0007] The objectives of this invention are satisfied by the features of the independent claims. Further advantageous developments and aspects of the invention are set forth in the dependent claims.

[0008] In a first aspect of this application, an apparatus is provided for searching lanes (i.e., self-driving lanes) that a vehicle may or should drive in.

[0009] The device is configured to receive images captured by a camera. This camera can be mounted on a vehicle and configured to capture images of the area in front of the vehicle. If the vehicle is traveling on a road, the images captured by the camera show the road conditions ahead of the vehicle.

[0010] The device is configured to detect lane markings in images captured by a camera. For each detected lane marking, the device determines whether the corresponding lane marking is of type one or type two. Type one lane markings indicate type one road conditions, such as normal road conditions. Type two lane markings indicate type two road conditions, such as a construction site.

[0011] The device is configured to create candidate lanes based on lane markings. A candidate lane is a lane a vehicle is likely to travel in. A candidate lane can be a lane between two detected lane markings. However, a candidate lane may also consist of only lane markings on one side of the lane.

[0012] The device is configured to classify candidate lanes into different candidate lane categories. These categories can be predetermined categories, and the candidate lanes within each category can have specific characteristics. In other words, each category includes candidate lanes with predetermined characteristics. One of these characteristics is the type of lane marking. Therefore, candidate lanes are classified into categories based on the type of lane marking they possess. Each category can be associated with a corresponding list, and candidate lanes of the corresponding category are either classified or written into that list.

[0013] The device is also configured to search for lanes (i.e., self-lanes) from candidate lane categories. A self-lane may be found within one of the categories. Alternatively, a self-lane may not be found within the categories.

[0014] This device enables vehicles to travel in different road conditions with different lane markings, such as when a vehicle is passing through a construction site.

[0015] In one embodiment, the device is further configured to determine, for each of the lane markings, whether the corresponding lane marking is a left lane marking or a right lane marking. Left lane markings are located on the left side of the lane or vehicle, while right lane markings are located on the right side of the lane or vehicle. Furthermore, the device can use the orientation of the lane markings, i.e., whether the corresponding lane marking is a left lane marking or a right lane marking, to classify candidate lanes. Therefore, classifying a candidate lane into a candidate lane category may depend not only on the type of lane markings of the candidate lane but also on the orientation of the lane markings.

[0016] In another implementation, the candidate lane categories include at least categories a) to e). For each of categories a) to e), a corresponding list can be provided, in which candidate lanes of the corresponding category can be categorized or written.

[0017] Category a) is for candidate lanes with a second type of left lane marking and a second type of right lane marking.

[0018] Category b) is for candidate lanes with left lane markings and right lane markings, where one of these two lane markings is of type 1 and the other of these two lane markings is of type 2.

[0019] Category c) is for lane markings that are only on one side and that lane marking is a candidate lane of type 2. There are no lane markings on the other side of the lane.

[0020] Category d) is for candidate lanes with first-type left lane markings and first-type right lane markings.

[0021] Category e) is for lane markings that are only on one side and that lane marking is a candidate lane of type 1. There are no lane markings on the other side of the lane.

[0022] The device can also be configured to sequentially search for lanes that a vehicle can drive in from candidate lane categories. Specifically, the categories are searched in a given order, meaning there is a predetermined hierarchical structure between the categories. For example, the device first searches category a) to see if it contains candidate lanes that can be used as ego lanes for vehicles. Then, the device searches these categories in the order of categories b), c), d), and e) to check if at least one of these categories contains an ego lane.

[0023] The device can also be configured to terminate the search for candidate lane categories once a lane that the vehicle can drive in has been selected. For example, if a self-lane is found in category a), the device terminates the search and does not search for a self-lane in categories b), c), d), and e).

[0024] The device can also be configured to assign a candidate lane to one of the candidate lane categories only when the candidate lane meets one or more criteria. The criteria can be selected from three options, where any combination of criteria is possible, and additional or alternative criteria can be added.

[0025] 1. If the width of a candidate lane is equal to or greater than the predetermined width, then the candidate lane is assigned to one of the categories.

[0026] 2. If a candidate lane has lane markings on only one side and not on the other side, the candidate lane is assigned to one of the categories if the lane markings of the candidate lane are within a predetermined first distance relative to the vehicle.

[0027] 3. If a candidate lane is within a predetermined second distance relative to a lane that has been previously selected for use by vehicles, then the candidate lane is assigned to one of the categories.

[0028] Furthermore, the device can be configured to select candidate lanes from one of the categories as lanes that vehicles can drive on. For this decision, one or more of the aforementioned three criteria can be used. Moreover, when selecting a candidate lane as a lane that vehicles can drive on, the overall geometric information of the candidate lane can be considered, such as offset, slope, curvature, rate of change of curvature, etc. For example, if other candidates are available, a candidate lane with two lane markings that has good width in front of the vehicle but subsequently intersects is not a good candidate. Furthermore, each lane marking has its appearance information, such as features extracted from the image and its classification type, such as solid or dashed lane markings. Additionally, each lane marking has its metadata, such as how confident the classifier is about the type of lane marking (solid, dashed, etc.) and how confident the tracker is about the geometry of the candidate lane. Furthermore, not only the image of the current frame but also images of previous frames can be considered. For selecting a candidate lane from one of the categories as a lane that vehicles can drive on, the device can use one or more of the criteria described herein. Alternative or additional criteria for selecting candidate lanes as lanes that vehicles can drive in are that the candidate lane is stable and / or the candidate lane has no steep protrusions and / or the candidate lane has no sudden changes in width.

[0029] In another embodiment, the first type of road condition is a normal road condition, that is, not a construction site, while the second type of road condition is a construction site.

[0030] In another embodiment, the first type of lane marking has a predetermined first color, particularly white or non-yellow, and the second type of lane marking has a predetermined second color, particularly yellow.

[0031] According to a second aspect of this application, a system for searching lanes where vehicles can travel includes a camera for capturing images and the device described above.

[0032] According to a third aspect of this application, a method for searching for lanes in which vehicles can travel includes the following steps:

[0033] - Receives images captured by a camera, showing the area in front of the vehicle.

[0034] - Detect lane markings in the image.

[0035] - For each lane marking, determine whether the corresponding lane marking represents Type 1 of the first type of road condition or Type 2 of the second type of road condition.

[0036] - Create candidate lanes based on lane markings

[0037] - Based on the type of lane markings of the candidate lanes, the candidate lanes are classified into candidate lane categories, and

[0038] - Search for lanes that the vehicle can travel in from the candidate lane categories.

[0039] The method according to the third aspect of this application may include the embodiments disclosed above in conjunction with the apparatus according to the first aspect of this application. Attached Figure Description

[0040] The invention will now be described in more detail by way of example with reference to embodiments and the accompanying drawings. In these drawings:

[0041] Figure 1 This is a schematic diagram of an exemplary implementation of a system for searching lanes where vehicles can travel;

[0042] Figure 2 This is a schematic diagram of an exemplary implementation of a method for searching lanes in which vehicles can travel;

[0043] Figures 3A to 3E This is a diagram illustrating images captured by a camera installed on a vehicle; and

[0044] Figure 4 This is a diagram illustrating the search for your own lane. Detailed Implementation

[0045] Figure 1 A system 10 for searching lanes where vehicles can travel is schematically illustrated. The system 10 includes a camera 11 and a device 12. The system 10 is installed in a vehicle that is itself.

[0046] Camera 11 is mounted on the vehicle and captures images 13 of the area in front of the vehicle.

[0047] The image 13 captured by camera 11 is sent to device 12. Device 12 executes method 20 for searching lanes in which the vehicle can travel and generates an output signal 14 containing information about the lanes in which the vehicle can travel. Figure 2 The method is illustrated schematically in Figure 20.

[0048] The apparatus 12, system 10, and method 20 are exemplary embodiments of the first, second, and third aspects of this application, respectively.

[0049] exist Figure 2 In step 21 of method 20, camera 11 captures image 13 of the scene in front of the vehicle. If the vehicle is traveling on a road, then image 13 shows the road. Camera 11 captures image 13 in consecutive frames. For example, a single image 13 is captured in each frame.

[0050] In step 22, device 12 receives image 13 captured by camera 11 in step 21.

[0051] In step 23, device 12 searches for lane markings in image 13. If image 13 shows lane markings, then device 12 detects these lane markings. Device 12 can detect n individual and independent lane markings. n can be 0, greater than 0, or greater than 4.

[0052] In step 24, for each of the lane markings, device 12 determines whether the corresponding lane marking is a left lane marking (located on the left side of the vehicle) or a right lane marking (located on the right side of the vehicle). In other words, the detected lane markings are divided into left and right groups based on their lateral offset relative to the vehicle.

[0053] The task of device 12 is to correctly select the lane markings that indicate the vehicle's own lane (i.e., the lane the vehicle can / should travel in) and also comply with traffic rules. Selecting the correct lane is particularly challenging during construction site operations, i.e., when vehicles are passing through a construction site on the road.

[0054] Lane markings at construction sites have specific characteristics; specifically, the color of lane markings at construction sites differs from that at non-construction sites. In many countries, lane markings at construction sites are yellow, while those at non-construction sites are non-yellow, often white.

[0055] The rule during construction is that yellow lane markings cover existing white lane markings. However, depending on each individual construction site and the roadwork there, not all necessary lane markings are repainted in yellow; rather, existing white lane markings are often used alongside yellow lane markings. Alternatively, yellow lane markings may be painted over existing white lane markings.

[0056] A proper self-lane can include self-left lane markings and self-right lane markings, each of which can be yellow or white. It's also possible for a self-lane to have markings only on one side of the lane (yellow or white). Of course, there are also cases where there are no self-lane markings at all.

[0057] In step 25 of method 20, device 12 determines for each of the lane markings detected in step 23 whether the corresponding lane marking represents a first type of road condition or a second type of road condition. In this embodiment, the first type of road condition is a non-construction site, while the second type of road condition is a construction site. Furthermore, in this embodiment, the first type of lane marking is white, and the second type of lane marking is yellow.

[0058] In step 26, device 12 creates candidate lanes based on lane markings. Candidate lanes may have lane markings on both sides of the lane, or only on one side. Obviously, if a lane includes two lane markings, then these markings must be one on the left side of the vehicle and one on the right side. What may be less obvious is that in the real world, a lane marking may be missing, and a single lane marking may also indicate a self-contained lane.

[0059] There are multiple candidate lane categories, specifically categories a) to e). Figures 3A to 3E Examples of candidate lanes for categories a) through e) are shown respectively. Figures 3A to 3E This is a schematic diagram of image 13 taken by camera 11. Figures 3A to 3E The image shows road 40, a portion of the vehicle 41, a left lane marking 42 on the left side of the vehicle, and a right lane marking 43 on the right side of the vehicle. The white lane markings have no shading, while the yellow lane markings... Figures 3A to 3E It is marked with a shading line.

[0060] like Figure 3A As shown, lanes or candidate lanes of category a) have yellow left lane markings 42 and yellow right lane markings 43.

[0061] Lanes or candidate lanes in category b) have left lane marking 42 and right lane marking 43, wherein one of lane markings 42 and 43 is yellow, and the other of lane markings 42 and 43 is white. Figure 3B In the example, left lane marking 42 is white, while right lane marking 43 is yellow.

[0062] Category c) lanes or candidate lanes have lane markings only on one side of the lane, where these lane markings are yellow. Figure 3CIn the example, there is only a yellow left lane marker 42, but no right lane marker 43.

[0063] Lanes or candidate lanes in category d) have white left lane markings 42 and white right lane markings 43, such as Figure 3D As shown.

[0064] Lanes or candidate lanes in category e) have lane markings only on one side of the lane, where these lane markings are white. Figure 3E In the example, there is only a white left lane marking 42, but no right lane marking 43.

[0065] In step 27 of method 20, device 12 classifies the candidate lanes created in step 26 into candidate lane categories a through e) based on the type of lane markings of the candidate lanes. Therefore, each candidate lane is classified as a lane in one of categories a) through e). Furthermore, for each of categories a) through e), a corresponding list exists, into which the candidate lanes identified in step 27 can be written.

[0066] To become a feasible candidate in step 27, a candidate lane must meet certain predetermined criteria; that is, in step 27, candidate lanes that do not meet the predetermined criteria are not assigned to one of categories a) to e). Examples of these criteria are given below; however, other or alternative criteria may also be used:

[0067] a) The width of a candidate lane with two lane markings (left and right) must fall within a reasonable spacing. For example, a minimum width of 2.1 meters can be applied during construction site work. Alternatively, a width of 6 meters can be applied during highway entrance ramps. Therefore, if the width of a candidate lane is equal to or greater than the predetermined width, the candidate lane can be assigned to one of categories a) through e).

[0068] b) The lane marking offset of a candidate lane that has lane markings on only one side must be close enough to the vehicle. Therefore, if a candidate lane has lane markings on only one side, the candidate lane can be assigned to one of categories a) to e) if the lane markings of the candidate lane are within a predetermined first distance relative to the vehicle.

[0069] c) A viable candidate lane must be close to the lane selected in the last image frame. Therefore, if a candidate lane is within a predetermined second distance relative to a lane previously selected for a vehicle, it can be assigned to one of categories a) to e).

[0070] In step 28, device 12 searches for candidate lane categories a) to e), specifically, searches for lanes that vehicles can drive in from the list associated with categories a) to e), i.e., self-driving lanes.

[0071] Device 12 searches for its own lane from candidate lanes in categories a) to e) in a given order (i.e., according to their priority). Device 12 starts with category a), then sequentially moves to categories b), c), and d), up to category e). If the list of any category contains more than one candidate lane, then device 12 can select the most suitable candidate lane from the list of that category as its own lane. For this decision, geometric information of the candidate lanes from the current frame and previous frames and / or metadata and / or other standards can be used, for example.

[0072] Once a self-positioned lane is found in the list of one of the categories, device 12 terminates the search for candidate lane categories. It then skips the list of the remaining categories. Figure 4 The process is illustrated schematically in the diagram.

[0073] For example, if both left and right lane markings are painted yellow during construction, then a possible candidate lane will be found in the list for category a). The lists for categories b) through e) can then be skipped.

[0074] If, during construction, lane markings are painted yellow only on one side of the lane, and existing white lane markings still exist on the other side of the lane, then the list for category a) should be empty, but the list for category b) should include the correct self-lane as a candidate lane.

[0075] It can also be seen that during normal road driving without yellow lane markings, the lists for categories a), b), and c) should be empty. Only the lists for categories d) and e) contain candidate lanes. Therefore, method 20 always attempts to find a feasible pair of self-lane markings for its own lane.

[0076] The selected autonomous lane is output by device 12 as output signal 14 for further processing, such as guiding the vehicle to the selected autonomous lane.

[0077] For simplicity, Figures 3A to 3E The example shows only one lane. Of course, device 12 can also select the correct lane for multiple lanes. Moreover, lane markings of different colors may overlap, yet device 12 can still correctly select its own lane.

[0078] List of labels

[0079] 10 system

[0080] 11 cameras

[0081] 12 devices

[0082] 13 images

[0083] 14 Output Signals

[0084] 20 methods

[0085] 21 steps

[0086] 22 steps

[0087] 23 steps

[0088] 24 steps

[0089] 25 steps

[0090] 26 steps

[0091] 27 steps

[0092] 28 steps

[0093] 40 Road

[0094] Part 41

[0095] 42 Left Lane Markings

[0096] 43 Right Lane Markings

Claims

1. A device (12) for searching for drivable lanes for vehicles, wherein, The device (12) is configured to Receive an image (13) captured by the camera (11), the image (13) showing the area in front of the vehicle, Detect lane markings (42, 43) in the image (13). For each of the lane markings (42, 43), determine whether the corresponding lane marking represents the first type of road condition or the second type of road condition. For each of the lane markings (42, 43), determine whether the corresponding lane marking is a left lane marking (42) or a right lane marking (43). Candidate lanes are created based on the lane markings (42, 43). Based on the type of lane markings (42, 43) of the candidate lanes, the candidate lanes are classified into candidate lane categories, wherein the candidate lane categories include one or more categories that contain candidate lanes with lane markings (42, 43) only on one side of the lane, and Search for lanes that the vehicle can drive in from the candidate lane categories.

2. The apparatus (12) according to claim 1, wherein, The device (12) is also configured to sequentially search for lanes that the vehicle can drive in from the candidate lane categories.

3. The apparatus (12) according to claim 1, wherein, The device (12) is also configured to terminate the search for the candidate lane category when a lane that the vehicle can drive in has already been selected.

4. The apparatus (12) according to claim 1, wherein, The device (12) is also configured to assign a candidate lane to one of the categories only if the candidate lane meets one or more criteria.

5. The apparatus (12) according to claim 4, wherein, One of the criteria is that the width of the candidate lane is equal to or greater than a predetermined width.

6. The apparatus (12) according to claim 4 or 5, wherein, One of the criteria is that, if the candidate lane has lane markings (42, 43) on only one side, the lane markings (42, 43) of the candidate lane are within a predetermined first distance relative to the vehicle.

7. The apparatus (12) according to claim 4, wherein, One of the criteria is that the candidate lane is within a predetermined second distance relative to a lane previously selected for the vehicle's travel.

8. The apparatus (12) according to claim 1, wherein, The first type of road condition is a non-construction site, while the second type of road condition is a construction site.

9. The apparatus (12) according to claim 1, wherein, The first type of lane markings (42, 43) have a predetermined first color, and the second type of lane markings (42, 43) have a predetermined second color.

10. The apparatus (12) according to claim 9, wherein, The predetermined first color is white or a color other than yellow, and the predetermined second color is yellow.

11. A system (10) for searching for drivable lanes for vehicles, the system (10) comprising a camera (11) for capturing images (13) and a device (12) according to any one of claims 1 to 10.

12. A method (20) for searching for drivable lanes for vehicles, the method comprising the steps of: Receive an image (13) captured by the camera (11), the image (13) showing the area in front of the vehicle, Detect lane markings (42, 43) in the image (13). For each of the lane markings (42, 43), determine whether the corresponding lane marking (42, 43) represents the first type of road condition or the second type of road condition. For each of the lane markings (42, 43), determine whether the corresponding lane marking is a left lane marking (42) or a right lane marking (43). Candidate lanes are created based on the lane markings (42, 43). Based on the type of lane markings (42, 43) of the candidate lanes, the candidate lanes are classified into candidate lane categories, wherein the candidate lane categories include one or more categories that contain candidate lanes with lane markings (42, 43) only on one side of the lane, and Search for lanes that the vehicle can drive in from the candidate lane categories.

Images