Fault isolation and mitigation for mis-detection of lane markings on a roadway

Abstract

This invention relates to fault isolation and mitigation in the event of false detection of lane markings on a road. A system for a vehicle includes a controller and a plurality of sensors on the vehicle. A first sensor of the plurality of sensors is configured to detect lane markings on the road. The controller is configured to store data from the plurality of sensors. In response to receiving an indication based on data received from the first sensor indicating a false detection of lane markings on the road, the controller is configured to: execute in parallel a plurality of processes configured to detect, respectively, a plurality of causes of false detection of lane markings based on the stored data; isolate one of these causes as the root cause of the false detection of lane markings; and provide a response for mitigating the false detection of lane markings on the road based on the root cause of the false detection of lane markings.

Description

Technical Field

[0001] The information provided in this section is for the purpose of presenting the overall context of this disclosure. The work of the currently attributed inventors, to the extent described in this section, and in all aspects of that description which at the time of filing may not be regarded as prior art, is neither expressly nor implicitly considered prior art to this disclosure.

[0002] This disclosure relates generally to vehicle control systems, and more particularly to fault isolation and mitigation in the event of false detection of lane markings on a road. Background Technology

[0003] Today, many vehicles are equipped with lane departure warning systems to assist drivers. For example, lane departure warning systems include sensors (such as cameras) that monitor lane markings and warn the driver when the vehicle tends to deviate from its lane. Specifically, data from these sensors is processed to determine whether the vehicle is staying in its lane and to detect when it is deviating. When the vehicle deviates from its lane, a warning is issued to the driver. Some vehicles are also equipped with perception systems that include sensors (such as cameras, radar, and lidar) that monitor the vehicle's surroundings and assist the driver in tasks such as parking, changing lanes, and more. Summary of the Invention

[0004] A system for a vehicle includes a controller and a plurality of sensors on the vehicle. A first sensor of the plurality of sensors is configured to detect lane markings on a road. The controller is configured to store data from the plurality of sensors. In response to receiving an indication based on data received from the first sensor indicating a false detection of lane markings on the road, the controller is configured to: execute a plurality of procedures in parallel, the plurality of procedures being configured to detect, respectively, a plurality of causes of false detection of lane markings based on the stored data; isolate one of these causes as the root cause of the false detection of lane markings; and provide a response for mitigating false detection of lane markings on the road based on the root cause of the false detection of lane markings.

[0005] Among other features, these processes are configured to detect whether a false detection of lane markings is caused by any of the following: a faulty first sensor; the absence of lane markings on the road; rain or snow that obscures lane markings on the road; glare or shadows that obscure lane markings on the road; an obstacle in the field of view of the first sensor; construction work on the road; a change in the lane configuration of the road; and unpaved road (or "removed road surface").

[0006] Among other features, the response includes: a second vehicle following in front of the vehicle, notifying the vehicle's occupants to take over control of the driving vehicle, switching to a second of the plurality of sensors for lane marking detection, and / or dispatching services.

[0007] Among other features, one of the processes executed by the controller is configured to: process an image captured by a first sensor; perform clustering and filtering of pixels in the image; perform first and second curve fitting on the filtered pixels; and determine whether lane markings exist on the road based on the second curve fitting.

[0008] Among other features, one of the processes executed by the controller is configured to: process images captured by two of the plurality of sensors; for each of the two sensors, determine the number of images with and without lane markings; for each of the two sensors, calculate the ratio of the number of images with or without lane markings to the total number of images processed; and determine whether lane markings exist on the road based on the ratio between the two sensors.

[0009] Among other features, one of the processes executed by the controller is configured to confirm whether the first sensor is faulty by comparing the original image of the road captured by the first sensor with the original image of the road captured by the second sensor among the plurality of sensors.

[0010] Among other features, one of the processes executed by the controller is configured to confirm whether the first sensor is faulty by comparing objects detected in an image of the road captured by the first sensor with objects detected in an image of the road captured by the second sensor among the plurality of sensors.

[0011] Among other features, one of the processes executed by the controller is configured to detect whether the road is unpaved based on images captured by the first sensor, map information about the road, and data from the inertial measurement unit in the vehicle.

[0012] Among other features, one of the processes executed by the controller is configured to use a machine learning model to detect the presence of rain or snow in an image captured by the first sensor.

[0013] Among other features, one of the processes executed by the controller is configured to detect the presence of glare or shadows in images based on pixel intensity in images captured by the first sensor.

[0014] Among other features, one of the processes executed by the controller is configured to detect the presence of construction objects on the road based on construction objects detected in images captured by one of the plurality of sensors.

[0015] Among the other features, one of the processes executed by the controller is configured to detect changes in the lane configuration of the road by detecting at least one of changes in the number of lanes in the road and changes in the relationship between the number of lanes and the lanes occupied by vehicles.

[0016] Among other features, one of the processes executed by the controller is configured to: determine whether a second vehicle exists within a predetermined distance in front of the vehicle, and determine the size of the second vehicle, based on an image captured by a second sensor among the plurality of sensors and data received from a third sensor among the plurality of sensors.

[0017] Among the other features, one of the processes executed by the controller is configured to determine whether to follow the second vehicle based on a predetermined distance.

[0018] Among other features, one of the processes executed by the controller is configured to determine whether to drive the actuator on another track in response to the predetermined distance being greater than a predetermined distance.

[0019] Among other features, a method for a vehicle includes: storing data from a plurality of sensors on the vehicle; and receiving an indication of false detection of lane markings on a road based on data received from a first sensor of the plurality of sensors. The method includes: performing in parallel a plurality of processes in response to receiving the indication, the plurality of processes being configured to: detect, respectively, a plurality of causes of false detection of lane markings based on the stored data; identify one of these causes as the root cause of the false detection of lane markings; and provide a response for mitigating false detection of lane markings on the road based on the root cause of the false detection of lane markings. The response includes: a second vehicle following in front of the vehicle; notifying the occupants of the vehicle to take over control of the driving vehicle; switching to a second sensor of the plurality of sensors for lane marking detection; and / or dispatching services.

[0020] Among other features, the method further includes using these processes to detect whether a false detection of lane markings is caused by any of the following: a faulty first sensor; the absence of lane markings on the road; rain or snow that obscures lane markings on the road; glare or shadows that obscure lane markings on the road; an obstacle in the field of view of the first sensor; construction work on the road; changes in the lane configuration of the road; and an unpaved road.

[0021] Among other features, the method further includes: processing an image captured by a first sensor; performing clustering and filtering of pixels in the image; performing first and second curve fitting on the filtered pixels; and determining whether lane markings exist on the road based on the second curve fitting.

[0022] Among other features, the method further includes: processing images captured by two of the plurality of sensors; determining, for each of the two sensors, the number of images with and without lane markings; calculating, for each of the two sensors, a ratio of the number of images with or without lane markings to the total number of images processed; and determining, based on the ratio of the two sensors, whether lane markings exist on the road.

[0023] Among other features, the method further includes: confirming whether the first sensor is faulty by comparing an original image of the road captured by the first sensor with an original image of the road captured by a second sensor among the plurality of sensors; or comparing an object detected in an image of the road captured by the first sensor with an object detected in an image of the road captured by the second sensor among the plurality of sensors.

[0024] The present invention also discloses the following technical solutions:

[0025] Option 1. A system for a vehicle, the system comprising:

[0026] The vehicle has multiple sensors, wherein a first sensor of the multiple sensors is configured to detect lane markings on the road; and

[0027] The controller is constructed as follows:

[0028] Storing data from the plurality of sensors; and

[0029] In response to receiving an indication based on data received from the first sensor indicating a false detection of lane markings on the road:

[0030] Multiple processes are executed in parallel, the multiple processes being configured to: detect multiple causes of the false detection of lane markings based on stored data;

[0031] Isolate one of the aforementioned causes as the root cause of the false detection of lane markings; and

[0032] Based on the root cause of the false detection of lane markings, a response is provided to mitigate the false detection of lane markings on the road.

[0033] Option 2. The system according to Option 1, wherein the process is configured to detect whether the false detection of the lane marking is caused by any of the following:

[0034] The first sensor is faulty;

[0035] There are no lane markings on the road in question;

[0036] There is rain or snow that makes the lane markings on the road blurred;

[0037] There is glare or shadow that obscures lane markings on the road;

[0038] There is an obstacle in the field of view of the first sensor;

[0039] There are construction works on the road;

[0040] The lane configuration changes of the road; and

[0041] The road was unpaved.

[0042] Option 3. The system according to Option 1, wherein the response includes: a second vehicle following in front of the vehicle, notifying the occupants of the vehicle to take over control of driving the vehicle, switching to a second sensor among the plurality of sensors for lane marking detection, and / or dispatching services.

[0043] Option 4. The system according to Option 1, wherein one of the processes executed by the controller is configured as follows:

[0044] Process the image captured by the first sensor;

[0045] Perform clustering and filtering of pixels in the image;

[0046] Perform first curve fitting and second curve fitting on the filtered pixels; and

[0047] The presence of lane markings on the road is determined based on the second curve fitting.

[0048] Option 5. The system according to Option 1, wherein one of the processes executed by the controller is configured as follows:

[0049] Process images captured by two of the plurality of sensors;

[0050] For each of the two sensors, determine the number of images with and without lane markings;

[0051] For each of the two sensors, calculate the ratio of the number of images with or without lane markings to the total number of images processed; and

[0052] The presence of lane markings on the road is determined based on the ratio of the two sensors.

[0053] Option 6. The system according to Option 1, wherein one of the processes performed by the controller is configured to: confirm whether the first sensor is faulty by comparing an original image of the road captured by the first sensor with an original image of the road captured by a second sensor among the plurality of sensors.

[0054] Option 7. The system according to Option 1, wherein one of the processes performed by the controller is configured to: confirm whether the first sensor is faulty by comparing an object detected in an image of the road captured by the first sensor with an object detected in an image of the road captured by a second sensor among the plurality of sensors.

[0055] Option 8. The system according to Option 1, wherein one of the processes executed by the controller is configured to detect whether the road is unpaved based on an image captured by the first sensor, map information about the road, and data from an inertial measurement unit in the vehicle.

[0056] Option 9. The system according to Option 1, wherein one of the processes executed by the controller is configured to: use a machine learning model to detect the presence of rain or snow in an image captured by the first sensor.

[0057] Option 10. The system according to Option 1, wherein one of the processes performed by the controller is configured to detect the presence of glare or shadow in the image based on the pixel intensity in the image captured by the first sensor.

[0058] Option 11. The system according to Option 1, wherein one of the processes executed by the controller is configured to detect the presence of construction objects on the road based on construction objects detected in images captured by one of the plurality of sensors.

[0059] Option 12. The system according to Option 1, wherein one of the processes performed by the controller is configured to detect a change in the lane configuration of the road by detecting at least one of a change in the number of lanes in the road and a change in the relationship between the number of lanes and the lanes occupied by the vehicle.

[0060] Option 13. The system according to Option 1, wherein one of the processes executed by the controller is configured to: determine whether a second vehicle exists within a predetermined distance in front of the vehicle, and determine the size of the second vehicle, based on an image captured by a second sensor among the plurality of sensors and data received from a third sensor among the plurality of sensors.

[0061] Option 14. The system according to Option 13, wherein one of the processes executed by the controller is configured to determine whether to follow the second vehicle based on the predetermined distance.

[0062] Option 15. The system according to Option 13, wherein one of the processes executed by the controller is configured to: determine whether to drive the driver on another trajectory in response to the predetermined distance being greater than a predetermined distance.

[0063] Option 16. A method for a vehicle, the method comprising:

[0064] Store data from multiple sensors on the vehicle;

[0065] Receive an indication of a false detection of lane markings on the road based on data received from a first sensor among the plurality of sensors;

[0066] In response to receiving the instruction, multiple processes are executed in parallel, the multiple processes being configured to: detect multiple causes of the false detection of lane markings based on stored data;

[0067] One of the aforementioned causes was identified as the root cause of the false detection of lane markings; and

[0068] Based on the root cause of the false detection of lane markings, a response is provided to mitigate the false detection of lane markings on the road.

[0069] The responses include: a second vehicle following in front of the vehicle, notifying the occupants of the vehicle to take over control of driving the vehicle, switching to a second sensor among the plurality of sensors for lane marking detection, and / or dispatching services.

[0070] Option 17. The method according to Option 16, further comprising: using the process to detect whether the false detection of the lane markings is caused by any of the following:

[0071] The first sensor is faulty;

[0072] There are no lane markings on the road in question;

[0073] There is rain or snow that makes the lane markings on the road blurred;

[0074] There is glare or shadow that obscures lane markings on the road;

[0075] There is an obstacle in the field of view of the first sensor;

[0076] There are construction works on the road;

[0077] The lane configuration changes of the road; and

[0078] The road was unpaved.

[0079] Option 18. The method according to Option 16, further comprising:

[0080] Process the image captured by the first sensor;

[0081] Perform clustering and filtering of pixels in the image;

[0082] Perform first curve fitting and second curve fitting on the filtered pixels; and

[0083] The presence of lane markings on the road is determined based on the second curve fitting.

[0084] Option 19. The method according to Option 16, further comprising:

[0085] Process images captured by two of the plurality of sensors;

[0086] For each of the two sensors, determine the number of images with and without lane markings;

[0087] For each of the two sensors, calculate the ratio of the number of images with or without lane markings to the total number of images processed; and

[0088] The presence of lane markings on the road is determined based on the ratio of the two sensors.

[0089] Option 20. The method according to Option 16 further includes: confirming whether the first sensor is faulty by:

[0090] The original image of the road captured by the first sensor is compared with the original image of the road captured by the second sensor among the plurality of sensors; or

[0091] Objects detected in an image of the road captured by the first sensor are compared with objects detected in an image of the road captured by a second sensor among the plurality of sensors.

[0092] Further applicability of this disclosure will become apparent from the detailed description, claims, and drawings. The detailed description and specific examples are intended for illustrative purposes only and are not intended to limit the scope of this disclosure. Attached Figure Description

[0093] This disclosure will be more fully understood from the detailed description and accompanying drawings, in which:

[0094] Figure 1 It shows that it can be implemented Figure 3-18 A simplified example of a distributed computing system illustrating the systems and methods shown;

[0095] Figure 2 It shows Figure 1 A simplified example of a server in a distributed computing system;

[0096] Figure 3 It shows that it can be used Figure 1 Examples of systems for fault isolation and mitigation implemented in vehicles, servers, or combinations thereof as shown;

[0097] Figure 4 It shows that it can be made by Figure 3 Examples of scenarios or events detected by the system, and examples of outputs that can be generated based on fault isolation and mitigation detection;

[0098] Figure 5 This demonstrates how false detection of lane markings can occur. Figure 3 The general method of system execution;

[0099] Figure 6 A method for performing lane marking error detection analysis is shown;

[0100] Figure 7 A general method for determining one or more causes of lane marking misdetection and providing mitigation strategies is shown;

[0101] Figures 8A-8B The first method for detecting the presence of lane markings is shown;

[0102] Figure 9 A second method for detecting the presence of lane markings is shown;

[0103] Figure 10 A method for diagnosing sensors used to detect lane markings is shown;

[0104] Figure 11 A method for detecting unpaved roads is shown;

[0105] Figure 12 A method for detecting rain and / or snow that could cause false detections of lane markings is shown;

[0106] Figure 13 A method for detecting glare and / or shadows that could cause false detection of lane markings is shown;

[0107] Figure 14 A method for detecting construction zones that could lead to false detections of lane markings is shown;

[0108] Figure 15 and Figure 16 A method is shown for detecting lane exits, entrances, and / or bifurcations that could cause false detection of lane markings;

[0109] Figure 17 A method for detecting obstacles to perform lane marking detection is shown; and

[0110] Figure 18 It shows the use in a vehicle Figure 3 The system provides an example of a method for one or more mitigation strategies.

[0111] In the accompanying drawings, "Y" represents "yes" and "N" represents "no", and the reference numerals may be reused to identify similar and / or identical elements. Detailed Implementation

[0112] Many conventional and autonomous vehicles include automated driving systems (see...) Figure 3 (Examples shown). For example, automated driving systems include lane departure warning systems and perception systems that use various sensors on the vehicle (including, but not limited to, cameras, radar, and lidar) to perceive the vehicle's surroundings. Camera-dependent lane departure warning and perception systems can fail in scenarios that blur the camera's field of view, obscure lane markings, or render lane markings undetectable. Non-limiting examples of such scenarios include heavy rain or snow, construction work on the road, obstacles in the sensor's field of view, the presence of glare / shadows, sensor malfunction, unpaved roads, and so on. In such scenarios, it is impossible to confirm the presence or absence of lane markings; and if lane markings are present, it is not easy to determine the root cause of their failure to be detected. Detecting lane markings is helpful for driving both conventional and autonomous vehicles. False detection of lane markings can lead to the incorrect inference that lanes are not present on the road, forcing the driver to take control of the automated driving system, which is an undesirable situation.

[0113] This disclosure addresses the above problems by providing a system for isolating faults associated with lane marking detection and providing mitigation strategies in the event of false detection of lane markings. As used herein, false detection of lane markings or the false detection of lane markings includes the failure to detect the presence of lane markings. For example, failure may occur due to one or more causes, including but not limited to: sensor failure, such as a camera in a perception system used to sense lane markings; environmental factors such as rain, snow, glare, and shadows; obstacles to the sensor; construction work on the road; changes in lane configuration; and so on. Failure may also occur due to unpaved roads or the absence of lane markings, where the absence of lane markings is normal but is falsely detected as a failure. The system isolates the root cause of the fault by considering sensor failure, the actual ground conditions of lane presence, scenarios affecting lane marking detection, and scenarios affecting the detection of lane presence. The system provides mitigation strategies based on the fault isolation results.

[0114] The system uses additional sensors (e.g., LiDAR, side-mounted cameras, etc.) along with a machine learning model trained to detect the presence of lane markings to detect (pseudo) ground conditions. The system can also use clustering-based methods that leverage machine learning models trained to detect lane markings (e.g., neural networks). The system uses machine learning methods and other rule-based methods to determine the root causes of false lane marking detections. These methods can identify scenarios or events such as rain, snow, glare / shadow, obstacles, construction zones, lane exits / entries / forks, and unpaved roads as root causes of false lane marking detections. The system implements a hierarchical mitigation strategy to enable driver assistance and improve the performance of the perception system in such scenarios.

[0115] The system for fault isolation and mitigation according to this disclosure can reside in a vehicle, in one or more servers in the cloud, or a combination thereof. The system improves the field of automated driving systems by providing online analysis of lane marking misdetection and fault isolation, enabling real-time mitigation / adaptation of the lane marking detection system and the automated driving system during fault isolation, and minimizing the engineering effort required to troubleshoot faults in the lane marking detection system. These and other features of the system of this disclosure are described in detail below.

[0116] This disclosure is organized as follows. Initially, referenced... Figure 1 and Figure 2 Distributed computing systems for fault isolation and mitigation that can be partially or fully implemented according to this disclosure are shown and described. References Figure 3 Systems for fault isolation and mitigation according to this disclosure are shown and described, and can be implemented partially or completely. References Figure 4-18 Various methods for fault isolation and mitigation according to this disclosure are shown and described, which can be used in... Figure 1 and Figure 2 Distributed computing systems Figure 3 Implemented in a system or combination thereof.

[0117] Figure 1 A simplified example of a distributed computing system 100 is shown. The distributed computing system 100 may partially or fully implement the systems and methods for fault isolation and mitigation according to this disclosure, which are referenced below. Figure 3-18 To illustrate and describe. Distributed computing system 100 includes distributed communication system 110, one or more vehicles 120-1, 120-2, ... and 120-M (collectively referred to as vehicles 120), and one or more servers 130-1, 130-2, ... and 130-N (collectively referred to as servers 130). M and N are integers greater than or equal to one.

[0118] The distributed communication system 110 may include a local area network (LAN), a wide area network (WAN) (such as the Internet), and / or other types of networks. Vehicle 120 and server 130 may be located in different geographical locations and can communicate with each other via the distributed communication system 110. For example, server 130 may be located in a data center in the cloud. Vehicle 120 and server 130 can connect to the distributed communication system 110 using wireless and / or wired connections.

[0119] Vehicle 120 may be included in the following reference Figure 3 The system shown and described herein is capable of executing software applications. These software applications can implement various methods for fault isolation and mitigation according to this disclosure, which are referenced below. Figure 4-18 To illustrate and describe. Server 130 may provide various services to vehicle 120. For example, server 130 may execute software applications. These software applications may implement various methods for fault isolation and mitigation according to this disclosure, which are referred to below. Figure 4-18 To illustrate and describe. Server 130 can host multiple databases on which these software applications rely to provide services to the occupants of vehicle 120.

[0120] Figure 2A simplified example of server 130-1 is shown. Server 130-1 may include one or more CPUs or processors 170, a network interface 178, memory 180, and mass storage 182. In some embodiments, server 130-1 may be a general-purpose server and may include one or more input devices 172 (e.g., keypad, touchpad, mouse, etc.) and a display subsystem 174, which includes a monitor 176. Network interface 178 connects server 130-1 to distributed communication system 110. For example, network interface 178 may include a wired interface (e.g., Ethernet interface) and / or a wireless interface (e.g., Wi-Fi, Bluetooth, Near Field Communication (NFC), or another wireless interface). Memory 180 may include volatile or non-volatile memory, cache, or other types of memory. Mass storage 182 may include flash memory, one or more magnetic hard disk drives (HDDs), or other mass storage devices.

[0121] The processor 170 of server 130-1 executes an operating system (OS) 184 and one or more software applications 186. The software applications 186 may implement various methods for fault isolation and mitigation according to this disclosure, which are referenced below. Figure 4-18 To illustrate and describe. In some examples, software application 186 may be implemented in a hypervisor or containerized architecture. Mass storage 182 may store one or more databases 188, which store data structures used by software application 186 to perform corresponding functions.

[0122] Figure 3 This disclosure illustrates the application of this method in vehicles (e.g., Figure 1 Example of a system 200 for fault isolation and mitigation implemented in any of the vehicles 140 shown. System 200 includes a controller 201, an infotainment subsystem 202, an automated driving subsystem 204, and a communication subsystem 212. The communication subsystem 212 can communicate with the one or more servers 130 via a distributed communication system 110. The controller 201 implements various methods for fault isolation and mitigation according to this disclosure, which are referenced below. Figure 4-18To illustrate and describe. Instead, in some examples, the one or more servers 130 may execute these methods, and the controller 201 may send necessary data to the one or more servers 130 and receive the results of the processed data from the one or more servers 130. In other examples, the controller 201 may execute some of these methods, while the one or more servers 130 may execute the remaining methods. Further, the controller 201 and one of the servers 130 may execute portions of any of these methods. In any of these examples, the infotainment subsystem 202 provides the results of these methods to the occupants of the vehicle. For example, the infotainment subsystem 202 provides alarms and other messages regarding fault isolation and mitigation to the occupants of the vehicle.

[0123] The infotainment subsystem 202 includes an audiovisual interface that allows vehicle occupants to interact with various vehicle subsystems. For example, results provided by these methods may be output via the infotainment subsystem 202 in the form of visual messages displayed on a display of the infotainment subsystem 202, audio messages output via speakers of the infotainment subsystem 202, or a combination thereof. For example, the infotainment subsystem 202 may include a touchscreen through which vehicle occupants can select commands from drop-down menus displayed on a graphical user interface (GUI). For example, the infotainment subsystem 202 may include a microphone through which vehicle occupants can issue commands to vehicle subsystems, etc. These commands may, for example, be responses to results provided by these methods (e.g., alarms or other messages) (e.g., to gain control of the automated driving subsystem 204).

[0124] The automated driving subsystem 204 uses multiple navigation sensors 206 to perform driving operations. For example, the navigation sensors 206 may include cameras installed throughout the vehicle, a Global Positioning System (GPS), an Inertial Measurement Unit (IMU), radar systems (short-range and long-range), a lidar system, and so on. Using data from these sensors, the automated driving subsystem 204 can perform driving operations. For example, the automated driving subsystem 204 can control the vehicle's steering subsystem 208 and braking subsystem 210. Depending on the results provided by one or more methods described below, the automated driving subsystem 204 can drive the vehicle or allow the vehicle's occupants to control the vehicle.

[0125] Figure 4 Different scenarios or events that can be detected by system 200 (e.g., controller 201) and different outputs that can be generated based on the detection are illustrated. As generally shown at 300, controller 201 can perform the following in parallel. Figure 5-18Various methods are described to perform these detections and provide mitigation strategies during their execution. Detection of these scenarios is triggered by a lane marking misdetection event, as shown at 302. For example, a lane marking misdetection event occurs when a sensor in the perception system (such as a camera in navigation sensor 206) misdetects lane markings in a lane tracked by the sensor while the vehicle is driving on the road. Scene detection is generally shown at 304, which includes checking for sensor failure, as shown at 310, and checking for the presence of lane markings on the ground, as shown at 312 (both described in detail below). Additional scenarios will be described below. The output generated by controller 201 based on the scene detection performed at 304 is generally shown at 306 and detailed in Table 1 shown below.

[0126] At point 304, when the lane marking misdetection event shown at point 302 is triggered, controller 201 operates in parallel. Figure 5-18 The method shown detects any of the following scenarios that could potentially cause or trigger lane marking misdetection events to determine the root cause of the triggering. For example, these scenarios can be of two types: the first type generally shown at 320, and the second type generally shown at 322.

[0127] For example, scenario 320 of the first type may include rain (and / or snow) detection 330, glare and / or shadow detection 332, obstacle detection 334, and construction zone detection 336. Scenario 322 of the second type may include lane exit, entrance, and / or fork detection 340 (collectively referred to as exit / entrance / fork detection 340), unpaved road detection 342, and construction zone detection 336. These detections, along with the ground conditions shown at 310 (checking for sensor failure) and 312 (checking for lane marking presence), are described in further detail below with reference to the following figures.

[0128] Based on the detection shown at 304, controller 201 generates one or more outputs X1, X2, X3, and X4 and provides mitigation strategies, which are shown in Table 1 below. If a sensor failure is detected (e.g., a camera malfunction), controller 201 sets output X1 to Y, and if no sensor failure is detected, controller 201 sets output X1 to N. If a lane exists (i.e., lane markings are confirmed), controller 201 sets output X2 to Y, and if no lane exists (e.g., if the road is unpaved), controller 201 sets output X2 to N. If any of the first type of scenario 320 is detected, controller 201 sets output X3 to Y, and if no one of the first type of scenario 320 is detected, controller 201 sets output X3 to N. If any of the second type of scenario 322 is detected, controller 201 sets output X4 to Y, and if no one of the second type of scenario 322 is detected, controller 201 sets output X4 to N.

[0129] Table 1

[0130] .

[0131] Figure 5-18 Various methods for detection and mitigation described above and in Table 1 are illustrated. One or more of these methods may be executed by controller 201, the one or more servers 130, or a combination thereof. Controller 201 executes these methods in parallel and uses the results output by these methods according to the table shown above to provide a mitigation strategy.

[0132] Figure 5 A general method 400 is shown when a lane marking is falsely detected. The detailed steps of method 400 are described below with reference to subsequent figures. At 402, method 400 determines whether a lane marking has been falsely detected. If a lane marking has been falsely detected, then at 404, method 400 performs a fault isolation process in parallel and generates, as shown in the figures below. Figure 4 And one or more outputs X1, X2, X3, and X4 shown in Table 1. Various methods for performing fault isolation and determining the root cause of the fault (i.e., lane marking misdetection) are described in detail below. At 406, when isolating the fault, method 400 performs one or more mitigation strategies based on the output of the fault isolation method (e.g., see Table 1).

[0133] Figure 6A method 410 for performing lane marking false detection analysis is shown. At 412, method 410 initializes a cyclic buffer of predetermined size B (e.g., in controller 201 or in automated driving subsystem 204), which is calibrable. For example, the size B of the buffer can be selected such that the cyclic buffer can store one or more sensors from the vehicle before and after the lane marking false detection occurs (e.g., ...). Figure 3 Sufficient amount of data from the navigation sensor 206 shown. At 414, method 410 stores relevant data from the one or more sensors on the vehicle in a cyclic buffer. For example, the data stored in the cyclic buffer may include, but is not limited to, the following: data from the IMU, images captured by the front-facing camera, images captured by one or more side-facing cameras, data from short-range and long-range radars, data from lidar, the result of data processed by a perception system (which may include one or more cameras), vehicle trajectory after lane marking misdetection (if the driver takes over), etc.

[0134] At 416, method 410 determines whether the lane markings were falsely detected (e.g., by a perception system in the vehicle). If the lane markings were correctly detected (i.e., no false detection of lane markings occurred), method 410 returns to 414. If the lane markings were falsely detected, at 418, method 410 stores additional data in a circular buffer for a predetermined time period T following the false detection of lane markings, where T is also calibrable. At 420, method 410 performs fault isolation and implements one or more mitigation strategies, examples of which are shown in Table 1. Various methods for performing fault isolation and determining the root cause of the fault are described in detail below.

[0135] Figure 7 A general method 450 is shown for determining the cause of a lane marking misdetection and implementing a mitigation strategy when a failure is detected. The detailed steps of method 450 are described below with reference to Figures 8-18. At 430, method 450 determines whether a lane marking has been misdetected. If the lane marking has not been misdetected (i.e., if the lane marking was detected correctly), method 450 returns to 432. If the lane marking has been misdetected, at 452, method 450 determines the presence of the lane marking. See below for... Figure 8A and Figure 8B as well as Figure 9 Two methods, 452-1 and 452-2, are used to illustrate and describe the ground conditions used to determine the presence of lane markings.

[0136] At point 432, method 450 identifies one or more causes of lane marking misdetection. The following is from the reference... Figure 10These methods are described in terms of "onwards". At 434, when determining one or more causes of lane marking misdetection, method 450 executes one or more mitigation strategies, examples of which are shown in Table 1 and described in further detail below.

[0137] Figures 8A-8B and Figure 9 Two different methods, 452-1 and 452-2, are shown for detecting ground conditions regarding the presence of lane markings. Figure 8A and Figure 8B The first method 452-1 for detecting ground conditions regarding the presence of lane markings is shown. Figure 9 A second method 452-2 is shown for detecting ground conditions regarding the presence of lane markings. These methods correspond to... Figure 4 The ground condition at point 312 shows the inspection of lane markings. Any of these methods can be used. These methods provide the corresponding results to controller 201. Controller 201 uses these results to isolate faults (i.e., diagnose the root cause of lane marking misdetection) and provide mitigation strategies according to Table 1.

[0138] exist Figure 8A In the middle, at 454, method 452-1 uses the front-facing camera (e.g., Figure 3 (One of the navigation sensors 206 shown) is used to capture an image of a portion of the road on which the vehicle is traveling. At 456, method 452-1 performs clustering and spatial filtering of the pixels in the captured image. Figure 8B Examples of clustering and spatial filtering are shown in the figure.

[0139] exist Figure 8B For example, the captured image may include pixels 440 that may be related to lane markings. The captured image may also include pixels 442 as outliers. Further, the captured image may also include pixels 444 that are not related to objects related to lane markings. For example, pixel 444 may be related to a portion of a vehicle's hood or any other part of the vehicle that obscures the camera's field of view in the captured image. Method 452-1 determines the mean and standard distribution of the pixel clusters in the captured image. Method 452-1 uses spatial filtering (e.g., a sphere) 446 to filter out pixels 442 as outliers and pixels 444 that are not related to lane markings. Method 452-1 selects pixels 440 corresponding to the same pixel cluster.

[0140] At position 458, method 452-1 performs the first iteration of curve fitting on the selected pixel 440. The first iteration of curve fitting removes pixels that are relatively far from the center of the fitted curve. Therefore, the first iteration of curve fitting is an additional form of filtering performed on the selected pixel 440.

[0141] At 460, method 452-1 performs a second iteration of curve fitting on the pixels remaining after the first iteration of curve fitting to obtain filtered lane pixels. At 462, method 452-1 determines whether the number and / or density of pixels in the filtered lane pixels obtained after the second curve fitting is sufficient (e.g., greater than a threshold) to conclude that a lane marking exists. If the number and / or density of pixels in the filtered lane pixels obtained after the second curve fitting is sufficient, then at 464, method 452-1 concludes that a lane marking exists. If the number and / or density of pixels in the filtered lane pixels obtained after the second curve fitting is insufficient, then at 466, method 452-1 concludes that a lane marking does not exist.

[0142] Figure 9 An alternative method 452-2 for determining the presence of lane markings is illustrated. Method 452-2 processes images captured in parallel by two cameras on the vehicle (e.g., a left camera and a right camera). At 500, method 452-2 loads a first model (e.g., a CNN model, where CNN is a convolutional neural network) trained to detect lane markings using machine learning to process the image from the first camera (e.g., the left camera). At 502, method 452-2 loads a second model (e.g., a CNN model) trained to detect lane markings using machine learning to process the image from the second camera (e.g., the right camera).

[0143] At point 504, using the first model from the images received from the first camera, method 452-2 determines the number of images with and without lane markings. For example, method 452-2 can select as referenced above. Figure 6 This determination is made by interpreting the images stored in the loop buffer before and after the lane marking misdetection.

[0144] At point 506, using the second model, method 452-2 determines the number of images with and without lane markings from the images received from the second camera. For example, method 452-2 can be achieved by selecting images as referenced above. Figure 6 This determination is made by interpreting the images stored in the loop buffer before and after the lane marking misdetection.

[0145] At point 508, for the processed image from the left camera, method 452-2 calculates the ratio of the number of images without lane markings to the total number of images. Alternatively, method 452-2 may calculate the ratio of the number of images with lane markings to the total number of images.

[0146] At point 510, for the processed image from the right camera, method 452-2 calculates the ratio of the number of images without lane markings to the total number of images. Alternatively, method 452-2 may calculate the ratio of the number of images with lane markings to the total number of images.

[0147] At 512, method 452-2 determines whether the ratio calculated for the left camera is greater than or equal to a threshold. Alternatively, if method 452-2 calculates the ratio of the number of images with lane markings to the total number of images, then method 452-2 determines whether the ratio calculated for the left camera is less than or equal to a threshold.

[0148] At point 514, method 452-2 determines whether the ratio calculated for the right camera is greater than or equal to a threshold. Alternatively, if method 452-2 calculates the ratio of the number of images with lane markings to the total number of images, then method 452-2 determines whether the ratio calculated for the right camera is less than or equal to a threshold. These thresholds are calibrable.

[0149] At point 516, method 452-2 determines whether the ratios calculated for the left and right cameras are not greater than or equal to a threshold (i.e., if the decisions at points 512 and 514 are N and N). If the decisions at points 512 and 514 are N and N, then method 452-2 determines that lane markings exist in the processed image. If the decisions at points 512 and 514 are Y and Y, Y and N, or N and Y, then method 452-2 determines that lane markings do not exist in the processed image. If these ratios are calculated alternatively as mentioned above, method 452-2 can make a similar determination.

[0150] Figure 10 The diagram shows the diagnostics used to detect lane markings (i.e., to check in). Figure 4 Method 600 describes a sensor (i.e., camera) failure (as shown at 310 in the diagram). Method 600 provides the results to controller 201. Controller 201 uses these results to isolate the fault (i.e., diagnose the root cause of lane marking misdetection) and provide mitigation strategies according to Table 1.

[0151] Typically, the perception systems in a vehicle (e.g., in Figure 3 The automated driving subsystem 204 shown includes cameras (e.g., Figure 3(One of the navigation sensors 206 shown). The perception system monitors the operation of the camera. The perception system can generate diagnostic fault codes (DTCs) for the camera using the on-board diagnostic (OBD) system. At 602, method 600 reads the camera's DTC. At 604, method 600 determines whether a DTC (indicating a fault in the camera) is set. If a DTC is set, then at 620, method 600 determines that the sensor (i.e., the camera) has failed and the lane marking misdetection is due to the camera failure.

[0152] However, if DTC is not set (i.e., the camera is functioning correctly) and lane markings are still falsely detected, then at 606, method 600 cross-validates the results from the camera and other sensors. For example, method 600 compares the detections made by the camera with those made by another sensor that has a field of view (FOV) overlapping with the camera. For example, the other sensor could be another camera (e.g., a side-mounted camera) or a lidar.

[0153] Method 600 can perform two alternating types of processing. The first type of processing involves determining the actual objects detected by the two sensors (i.e., the camera and another sensor with an overlapping field of view), and comparing these detected objects; this is computationally intensive. Alternatively, the second type of processing involves comparing the raw images captured by the two sensors, which is less computationally intensive than the first type of processing.

[0154] At 610, method 600 performs the first type of processing. Method 600 processes images captured by the two sensors (i.e., a camera and another sensor) and detects objects in the images captured by the two sensors. Method 600 compares the objects detected in the images captured by the two sensors and determines whether the objects detected in the images captured by the two sensors match. If the objects detected in the images captured by the two sensors do not match, at 620, method 600 determines that the sensor (i.e., the camera) has failed and the lane marking misdetection is due to the camera failure. If the objects detected in the images captured by the two sensors match, at 614, method 600 determines that the sensor (i.e., the camera) is healthy (i.e., operating normally) and the lane marking misdetection is not due to the camera failure.

[0155] Alternatively, at 612, method 600 performs the second type of processing. Method 600 compares the original images captured by the two sensors and determines whether the original images captured by the two sensors match. If the original images captured by the two sensors do not match, then at 620, method 600 determines that the sensor (i.e., the camera) has failed and the lane marking misdetection is due to the camera failure. If the original images captured by the two sensors match, then at 614, method 600 determines that the sensor (i.e., the camera) is healthy (i.e., operating normally) and the lane marking misdetection is not due to the camera failure.

[0156] Figure 11 A method 700 for detecting unpaved roads is shown (i.e., Figure 4 The unpaved road detection shown is 342). Method 700 can use a combination of various techniques described below to detect unpaved roads. Method 700 provides the results to controller 201. Controller 201 uses these results to isolate faults (i.e., diagnose the root cause of lane marking misdetection) and provide mitigation strategies according to Table 1.

[0157] At 702, method 700 classifies images of roads captured by sensors (e.g., cameras in navigation sensor 206) as images of paved or unpaved roads. For example, method 700 can select, as referenced above... Figure 6 The interpretation involves storing images in a circular buffer before and after lane marking false detections for this classification. For example, to perform classification, method 700 can use a machine learning model (e.g., a CNN model) trained to detect paved and unpaved roads. Using the trained model, method 700 can classify images selected from the circular buffer.

[0158] At 710, method 700 calculates the ratio of the number of images indicating that the road is unpaved to the total number of images. Alternatively, method 700 may calculate the ratio of the number of images indicating that the road is paved to the total number of images. If the ratio is greater than or equal to a predetermined threshold (which is calibrable), then method 700 determines that the road is unpaved (or, if an alternative ratio is used, that the road is paved). At 720, if the road is unpaved, method 700 sets parameter X. CNN = 0. Alternatively, at 722, if the road is paved, then method 700 sets parameter X. CNN = 1.

[0159] Additionally, at 704, method 700 retrieves (i.e., reads) map information about the road the vehicle is traveling on. For example, controller 201 (in Figure 3(As shown in the diagram) Map information is obtained from one of the servers 130. Controller 201 uses GPS (see...) Figure 3 The navigation sensor 206 in the system provides GPS data and map information to detect the road the vehicle is traveling on. For example, method 700 can select GPS data from data stored in a cyclic buffer, as referenced above. Figure 6 As explained. At 712, method 700 determines whether a road is paved or unpaved based on the detected road and map information. At 724, if the road is unpaved, method 700 sets parameter X. MAP = 0. Alternatively, at 726, if the road is paved, then method 700 sets parameter X. MAP = 1.

[0160] Furthermore, at 706, method 700 calculates the data from the vehicle's IMU (e.g., see...). Figure 3 The power spectrum of road vibration sensed by the navigation sensor 206 in the system. Again, method 700 can select IMU data from the data stored in the cyclic buffer, as referenced above. Figure 6 As explained. At 714, method 700 compares the power spectrum (e.g., the energy of frequencies in the power spectrum) with a predetermined threshold E. th A (calibrable) comparison is used to determine whether the road the vehicle is traveling on is paved or unpaved. For example, if the power spectrum (e.g., the energy of frequencies in the power spectrum) is greater than or equal to a predetermined threshold E th Method 700 then concludes that the road is unpaved. At point 728, if the road is unpaved, method 700 sets parameter X. IMU = 0. Alternatively, at 730, if the road is paved, then method 700 sets parameter X. IMU = 1.

[0161] At position 740, method 700 calculates the weighted metric M = W1X. CNN + W2X MAP + W3X IMU The weights W1, W2, and W3 are calibrable. At 742, method 700 compares the weighted metric M with a predetermined threshold M. th The predetermined threshold is also calibrable. At 744, if the weighted metric M is less than the predetermined threshold M... th Method 700 then confirms that the road is unpaved. Alternatively, at 746, if the weighted metric M is greater than or equal to a predetermined threshold M... thIf the road is unpaved, then method 700 confirms that the road is unpaved. Therefore, by using a combination of three different techniques (shown at 702, 704 and 706), method 700 robustly determines whether the road on which the vehicle is traveling is paved or unpaved.

[0162] Figure 12 The diagram shows the method for detecting rain and / or snow (i.e., Figure 4 The method 800 (rain / snow detection 330) shown in the diagram may cause lane marking misdetection due to rain and / or snow. Method 800 provides the results to controller 201. Controller 201 uses these results to isolate faults (i.e., diagnose the root cause of lane marking misdetection) and provide mitigation strategies according to Table 1.

[0163] At 802, method 800 loads a model (e.g., a CNN model) trained to use machine learning to detect rain and / or snow to process data from sensors (e.g., images from a front-facing camera that brings the vehicle's windshield into view). For example, the trained model can detect windshield wiper movement, combinations of blurred and relatively sharp images captured by the sensors, etc. These can be signs of rain and / or snow. For example, rain and snow can be distinguished by further combining ambient temperature with these signs. For example, method 800 may choose to refer to the above description. Figure 6 The image and temperature data stored in the loop buffer before and after lane marking misdetection are explained for this processing.

[0164] At 804, method 800 determines the number of images with and without rain and / or snow indications. At 806, method 800 calculates the ratio of images with rain and / or snow indications to the total number of images processed. At 808, if the ratio is greater than or equal to a predetermined threshold, method 800 determines that the scene is rain or snow. Alternatively, at 810, if the ratio is less than the predetermined threshold, method 800 determines that the scene is not rain or snow.

[0165] Figure 13 The method for detecting glare and / or shadows (i.e., Figure 4 Method 850 (332) for glare and / or shadow detection shown in the diagram, where glare and / or shadow may cause false detection of lane markings. Method 850 provides the results to controller 201. Controller 201 uses these results to isolate faults (i.e., diagnose the root cause of false lane marking detection) and provide mitigation strategies according to Table 1.

[0166] At 852, method 850 creates a histogram of pixels in the region of interest from an image captured by a sensor (e.g., from a front-facing camera that brings the vehicle's windshield into view). For example, method 850 may select as referenced above. Figure 6The images stored in the circular buffer before and after the lane marking false detection are explained for this processing. Note that although method 850 uses histograms as described below, method 850 may alternatively use other machine learning methods that employ image classification (e.g., using a model trained to directly classify whether an image contains glare or shadow) or object detection (e.g., using a model trained to detect glare or shadow in an image).

[0167] At 854, method 850 determines the number of pixels N1 with an intensity greater than or equal to a first value T1, where both N1 and T1 are calibrable. At 856, method 850 determines the number of pixels N2 with an intensity less than or equal to a second value T2, where both N2 and T2 are also calibrable. At 858, method 850 determines whether N1 is greater than or equal to a first threshold Th1, which is calibrable. If N1 is greater than or equal to the first threshold Th1, then at 860, method 850 detects glare in the lane detection area, and method 850 terminates. Alternatively, if N1 is less than the first threshold Th1, then at 862, method 850 determines whether N2 is greater than or equal to a second threshold Th2, which is also calibrable. If N2 is greater than or equal to the second threshold Th2, then at 864, method 850 detects shadows in the lane detection area, and method 850 terminates. If N2 is less than the second threshold Th2, then at 866, method 850 determines that there is no glare or shadow in the lane detection area, and method 850 terminates.

[0168] Figure 14 The diagram shows the method for detecting the construction area (i.e., Figure 4 The construction zone detection method 880 shown in Figure 336 may cause false detection of lane markings. For example, method 880 may select the method described above. Figure 6 The images stored in the circular buffer before and after the lane marking false detection are interpreted for processing. Note that although method 880 uses object detection as described below, method 880 can alternatively use other machine learning methods that employ image classification (e.g., using a model trained to directly classify whether an image contains a construction zone). Method 880 provides the results to controller 201. Controller 201 uses these results to isolate faults (i.e., diagnose the root cause of lane marking false detection) and provide mitigation strategies according to Table 1.

[0169] At point 882, method 880 detects construction cones, barriers, and / or other markings (collectively referred to as construction objects) in images captured by sensors (e.g., images from a camera on the vehicle). At point 884, method 880 determines the number of images containing construction objects. At point 886, method 880 calculates the ratio of the number of images containing construction objects to the total number of images. At point 888, method 880 determines whether the ratio is greater than or equal to a calibrable threshold. If the ratio is greater than or equal to the threshold, then at point 890, method 880 determines that the vehicle is in a construction zone. Alternatively, if the ratio is less than the threshold, then at point 892, method 880 determines that the vehicle is not in a construction zone.

[0170] Figure 15 and Figure 16 Method 950 for detecting lane exits, entrances, and / or bifurcations is shown (i.e., collectively referred to as...). Figure 4 In the scenario shown in Exit / Entrance / Bifurcation Detection 340, exits, entrances, and / or bifurcations may cause false detection of lane markings. For example, method 950 may select as described above. Figure 6 The images stored in the loop buffer before and after lane marking misdetection are explained for processing. (In description) Figure 16 Before the method shown in 950, refer to Figure 15 Various examples of lane exit, entrance, and / or bifurcation scenarios are shown and described (i.e., various ways in which the lane configuration of a road can be changed).

[0171] exist Figure 15 For example, at position 910, vehicle 902 is in the leftmost lane (lane 1) 903, and there are two other lanes 904 and 905 to the right of lane 903, and lane 906 has been added to the left of lane 903. Lane 906 continues as a separate lane without merging with lane 903. Therefore, before lane 906 was added, there were 3 lanes, and vehicle 902 was in lane 1; after lane 906 was added, there are now 4 lanes, and vehicle 902 is now in lane 2 (lane 903), where, assuming vehicle 902 is in lane 903, the leftmost lane is now the added lane 906. This can be expressed using the following naming convention or notation: lane indices are 1, 2, 2; and the number of lanes is 3, 4, 4. In other words, the vehicle was originally in lane 1 (before lane 906 was added), is now in lane 2 (after lane 906 was added), and continues to be in lane 2 (after lane 906 was added); and the number of lanes was originally 3 (before lane 906 was added), is now 4 (after lane 906 was added), and continues to be 4 (after lane 906 was added). For simplicity, the same naming convention will be used for lane indexes and lane numbers in the following text, and will not be repeated.

[0172] At position 915, vehicle 902 is in the rightmost lane (lane 3) 905, with two other lanes 904 and 903 to the left of lane 905, and lane 907 added to the right of lane 905. Lane 907 continues as a separate lane and does not merge with lane 905. Therefore, before lane 907 was added, there were 3 lanes, and vehicle 902 was in lane 3; after lane 907 was added, there are now 4 lanes, and vehicle 902 is now still in lane 3 (lane 905), where, assuming vehicle 902 remains in lane 905, the rightmost lane is now the added fourth lane 907. This can be expressed using the following naming convention or notation: the lane index is 3, 3, 3; and the number of lanes is 3, 4, 4.

[0173] At position 920, vehicle 902 is in the leftmost lane (lane 1) 903, with two more lanes 904 and 905 to the right of lane 903, and lane 908 is the exit / entry lane to the left of lane 903. Lane 908 does not continue independently but merges with lane 903. Therefore, before lane 908, there are 3 lanes, and vehicle 902 is in lane 1; with lane 908, there are 4 lanes, and vehicle 902 is now in lane 2 (lane 903); and after lane 908 is interrupted (i.e., merged into lane 903), assuming vehicle 902 remains in lane 903, vehicle 902 is again in lane 1 (lane 903); and the number of lanes is again 3. This can be expressed using the following naming convention or notation: the lane index is 1, 2 (during the merging of lane 908 into lane 903), 1; and the number of lanes is 3, 4, 3 (temporarily 4 when lane 908 is merged into lane 903).

[0174] At position 925, vehicle 902 is in the rightmost lane (lane 3) 903, and there are two more lanes 904 and 903 to the left of lane 905, and lane 909 is the exit / entry lane to the right of lane 905. Lane 909 does not continue independently, but merges with lane 905. Therefore, before lane 909, there are 3 lanes, and vehicle 902 is in lane 3; with lane 909, there are 4 lanes, and vehicle 902 is still in lane 3 (lane 905); after lane 909 is interrupted (i.e., merged into lane 905), assuming vehicle 902 remains in lane 905, vehicle 902 is again in lane 3 (lane 905); and the number of lanes is again 3. This can be expressed using the following naming convention or notation: the lane index is 3, 3, 3; and the number of lanes is 3, 4, 3 (temporarily 4 when lane 909 merges into lane 905).

[0175] Similarly, at 930, lane 910 exits from lane 903, but lane 903 continues. The lane indices are 1, 2, 1; and the number of lanes is 3, 4, 3. At 935, lane 912 exits from lane 905, but lane 905 continues. The lane indices are 3, 3, 3; and the number of lanes is 3, 4, 3.

[0176] At 940, a lane bifurcation scenario is shown. Vehicle 902 is in the middle lane 904 (lane 2). The rightmost lane 905 (lane 3) bifurcates into lane 914. After the bifurcation, lane 905 is interrupted, and only two lanes, 904 and 903, continue. Assuming vehicle 902 remains in lane 905, the lane index is 2, 2, 2; and the number of lanes is 3, 4, 2 (temporarily 4 when lane 914 bifurcates from lane 905). Similarly, the leftmost lane 903 can also bifurcate. Many additional lane change scenarios (i.e., ways in which the lane configuration of the road can change) are possible and can be understood by those skilled in the art.

[0177] Figure 16 Method 950 (collectively referred to as) for detecting lane exits, entrances, and / or bifurcations is shown. Figure 4 Exit / entry / fork detection (340) in the road markings, exits, entrances, and / or forks may cause false detections of lane markings. Method 950 can detect lane configuration changes in the road as follows. For example, Method 950 can select regarding changes mentioned above. Figure 6 The data of the number of lanes L and the lane index LI stored in the cyclic buffer before and after lane marking misdetection are explained for this processing.

[0178] At 952, method 950 loads data regarding the number of lanes L and lane index LI for the vehicle (i.e., if the vehicle is being driven). At 954, based on this data, method 950 determines whether the vehicle is in the rightmost lane. If the vehicle is in the rightmost lane, then at 956, method 950 determines whether the values ​​of L and / or LI have changed. If the values ​​of L and / or LI have not changed, then at 958, method 950 determines that the lane marking misdetection was not caused by a lane exit / entrance / fork scenario, and method 950 terminates.

[0179] If the values ​​of L and / or LI have changed, then at 960, method 950 determines whether the values ​​of L and / or LI have been restored (i.e., reverted to their values ​​before the change). If the values ​​of L and / or LI have not been restored, then at 962, method 950 determines that a lane has been added to the right of the vehicle, and method 950 terminates. If the values ​​of L and / or LI have been restored, then at 964, method 950 determines that a lane exit / entry scenario has occurred on the right side of the vehicle, and method 950 terminates.

[0180] At 954, if the vehicle is not in the rightmost lane, method 950 proceeds to 970. At 970, method 950 determines if the vehicle is in the leftmost lane. If the vehicle is in the leftmost lane, at 972, method 950 determines if the value of LI has changed. If the value of LI has not changed, at 974, method 950 determines that the lane marking misdetection was not caused by a lane exit / entrance / fork scenario, and method 950 ends. If the value of LI has changed, at 976, method 950 determines if the value of LI has been restored (i.e., reverted to its value before the change). If the value of LI has not been restored, at 978, method 950 determines that a lane has been added to the left of the vehicle, and method 950 ends. If the value of LI has been restored, at 980, method 950 determines that a lane exit / entrance scenario has occurred to the left of the vehicle, and method 950 ends.

[0181] At point 970, if the vehicle is not in the leftmost lane, method 950 proceeds to point 982. At point 982, method 950 determines whether L has changed. If L has not changed, method 950 proceeds to point 974. If L has changed, at point 984, method 950 determines that a lane fork scenario has occurred, and method 950 terminates. Controller 201 uses the above determinations made by method 950 to determine the root cause of the lane marking misdetection.

[0182] Figure 17 A method 650 for detecting obstacles for lane marking detection is shown (e.g., an obstruction formed by another vehicle in front of the main vehicle, in...). Figure 4 The obstacle is shown as obstacle detection 334, which may cause false detection of lane markings. For example, method 650 may be selected as described above. Figure 6 The images stored in the cyclic buffer before and after the lane marking misdetection are interpreted for processing. Method 650 provides the results to controller 201. Controller 201 uses these results to isolate the fault (i.e., diagnose the root cause of the lane marking misdetection) and provide mitigation strategies according to Table 1.

[0183] At 652, method 650 is loaded by a remote radar (e.g., in...). Figure 3 The position information of the first N objects detected by the navigation sensor 206 in the diagram is shown. At 654, method 650 loads the position information of the first N objects detected by a camera typically used for sensing lane markings (e.g., in the diagram). Figure 3The navigation sensor 206 in the diagram shows the position information of the first N objects detected. At 656, method 650 counts the number of data points D1 indicating the presence of a relatively large vehicle (e.g., a truck) within a predetermined distance in front of the main vehicle, where D1 and the distance are calibrable. At 658, method 650 counts the number of data points D2 indicating the presence of a relatively small vehicle (e.g., a sedan) within a predetermined distance in front of the main vehicle, where D2 and the distance are calibrable.

[0184] At 660, method 650 determines whether D1 is greater than or equal to a predetermined quantity N1, which is calibrable. If D1 is greater than or equal to the predetermined quantity N1, then at 662, method 650 indicates that a large vehicle has been detected in front of the main vehicle, and method 650 terminates.

[0185] If D1 is less than N1, then at 664, method 650 determines whether D2 is greater than or equal to a predetermined quantity N2, which is also calibrable. If D2 is greater than or equal to the predetermined quantity N2, then at 666, method 650 indicates that a small vehicle has been detected in front of the main vehicle, and method 650 terminates.

[0186] If D2 is less than N2, then at point 668, method 650 determines that there is no vehicle in front of the main vehicle (i.e., no obstacle in the field of view of the camera used to sense lane markings), and method 650 terminates. The result provided by method 650 helps to eliminate obstacles in front of the vehicle as a possible cause of false lane marking detection.

[0187] Figure 18 An example of a method 1000 for using system 200 in a vehicle and executing one or more mitigation strategies according to this disclosure is shown. Method 1000 can be executed using a combination of one or more of controller 201 and servers 130 in the vehicle, wherein the combination utilizes data stored in a circular buffer in controller 201, and wherein each of controller 201 and the one or more servers 130 executes a portion of the method described above.

[0188] At 1002, method 1000 waits for a predetermined time T, which is calibrable. After the predetermined time T has elapsed, at 1004, method 1000 receives scene information (e.g., whether the scene is type one or type two). In some examples, scene information may also be received from other vehicles (e.g., via communication via the cloud and server 130) that are near the main vehicle and may also be experiencing lane misdetection. At 1006, method 1000 determines whether lane detection is challenging (e.g., due to heavy rain or snow). If lane detection is not challenging, method 1000 returns to 1002. If lane detection is challenging, at 1008, method 1000 determines whether a lane detection model trained to detect lane markings in the scene (e.g., one of the methods used for the scene described above) is available. If a lane detection model trained to detect lane markings in the scene is available, then at 1010, method 1000 uses the lane detection model trained to detect lane markings in the scene, which may be available in controller 201 and / or in one of the servers 130. These results are used by controller 201 as described above.

[0189] If the lane detection model trained to detect lane markings in the scenario is unavailable, at 1012, method 1000 determines whether the primary vehicle is likely to follow the vehicle in front of it. If so, at 1014, method 1000 allows the primary vehicle to follow the vehicle in front of it. If it is impossible to follow the vehicle in front of the primary vehicle (e.g., if the vehicle in front of the primary vehicle is relatively far away), at 1016, method 100 determines whether a new trajectory (e.g., a different path determined by the automated driving subsystem 204 or a trajectory uploaded by another vehicle near the primary vehicle) is available. If the new trajectory is available, at 1018, method 1000 follows that trajectory (i.e., the vehicle is driven along the available trajectory). If the new trajectory is unavailable, at 1020, method 1000 alerts the vehicle occupants to take over control of the vehicle (e.g., by means of...). Figure 2 The infotainment subsystem 202 shown in the diagram provides messages to the driver of the vehicle, and method 1000 ends.

[0190] After steps 1018 and 1014, method 1000 proceeds to step 1022. At step 1022, method 1000 determines whether the scenario is still (i.e., continues) challenging. If the scenario is not challenging or is no longer challenging, method 1000 returns to step 1002. If the scenario is still challenging, the method returns to step 1012.

[0191] Note that the example of Method 1000 assumes a single challenging scenario occurring at a time. However, if any other challenging scenarios occur simultaneously, Method 1000 notifies the driver of the predominantly challenging scenarios and alerts the driver to take over control of the vehicle (e.g., via...). Figure 2 The infotainment subsystem 202 shown provides one or more messages to the driver.

[0192] The foregoing description is merely illustrative in nature and is not intended to limit this disclosure, its application, or use. The broad teachings of this disclosure can be implemented in various forms. Therefore, while this disclosure includes specific examples, its true scope should not be so limited, as other modifications will become apparent upon examination of the drawings, specification, and the following claims. It should be understood that one or more steps within the method may be performed in a different order (or simultaneously) without altering the principles of this disclosure. Furthermore, while each of the embodiments described above is described as having certain features, any one or more of those features described with respect to any embodiment of this disclosure may be implemented in and / or combined with features of any of those in other embodiments, even if such combination is not explicitly described. In other words, the described embodiments are not mutually exclusive, and the arrangement of one or more embodiments with each other remains within the scope of this disclosure.

[0193] Spatial and functional relationships between components (e.g., between subsystems, controllers, circuit elements, semiconductor layers, etc.) are described using various terms, including “connected,” “joined,” “linked,” “adjacent,” “closely adjacent,” “on top of,” “above,” “below,” and “set.” Unless explicitly described as “direct,” when describing the relationship between a first and a second component in the above disclosure, the relationship can be a direct relationship in which no other intervening components exist between the first and second components, or it can be an indirect relationship (spatially or functionally) between the first and second components. As used herein, at least one of the phrases A, B, and C should be interpreted using non-exclusive logic or referential logic (A or B or C) and should not be interpreted as meaning “at least one of A, at least one of B, and at least one of C.”

[0194] In the accompanying drawings, the direction of the arrows generally illustrates the flow of information of interest (such as data or instructions). For example, when components A and B exchange various types of information, but the information transmitted from component A to component B is relevant to the illustration, the arrow may point from component A to component B. This unidirectional arrow does not imply that no other information is transmitted from component B to component A. Furthermore, for information sent from component A to component B, component B may send a request for the information to component A or receive acknowledgment of the information.

[0195] In this application (including the definitions below), the term "controller" may be replaced by the term "circuit". The term "controller" may refer to, be part of, or include the following: application-specific integrated circuit (ASIC); digital, analog, or mixed-signal analog / digital discrete circuit; digital, analog, or mixed-signal analog / digital integrated circuit; combinational logic circuit; field-programmable gate array (FPGA); processor circuitry (shared, dedicated, or grouped) that executes code; memory circuitry (shared, dedicated, or grouped) that stores code executed by the processor circuitry; other suitable hardware components that provide the described functionality; or combinations of some or all of the above, such as in a system-on-a-chip.

[0196] The controller may include one or more interface circuits. In some examples, the interface circuits may include wired or wireless interfaces connected to a local area network (LAN), the Internet, a wide area network (WAN), or a combination thereof. The functionality of the controller disclosed herein may be distributed among multiple controllers connected via the interface circuits. For example, various subsystems of this disclosure may include corresponding controllers. For example, multiple controllers may allow load balancing. In a further example, a server (also referred to as a remote or cloud) controller may perform a function on behalf of a client controller in a vehicle.

[0197] As used above, the term code may include software, firmware, and / or microcode, and may refer to programs, routines, functions, classes, data structures, and / or objects. The term shared processor circuitry covers a single processor circuit that executes some or all of the code from multiple controllers. The term group processor circuitry covers a processor circuit that, in combination with additional processor circuitry, executes some or all of the code from one or more controllers. The reference to multiple processor circuitry covers multiple processor circuits on a discrete die, multiple processor circuits on a single die, multiple cores of a single processor circuit, multiple threads of a single processor circuit, or a combination of the above. The term shared memory circuitry covers a single memory circuit that stores some or all of the code from multiple controllers. The term group memory circuitry covers a memory circuit that, in combination with additional memory, stores some or all of the code from one or more controllers.

[0198] The term memory circuit is a subset of the term computer-readable medium. As used herein, the term computer-readable medium does not cover transient electrical or electromagnetic signals propagating through a medium (such as a carrier wave); therefore, the term computer-readable medium can be considered tangible and non-transitory. Non-limiting examples of non-transitory, tangible computer-readable media are non-volatile memory circuits (such as flash memory circuits, erasable programmable read-only memory circuits, or mask read-only memory circuits), volatile memory circuits (such as static random access memory circuits or dynamic random access memory circuits), magnetic storage media (such as analog or digital magnetic tape or hard disk drives), and optical storage media (such as CDs, DVDs, or Blu-ray discs).

[0199] The apparatus and methods described in this application can be implemented, in part or in whole, by a special-purpose computer created by causing a general-purpose computing mechanism to perform one or more specific functions embodied in a computer program. The function blocks, flowchart components, and other elements described above serve as software specifications that can be routinely converted into computer programs by skilled technicians or programmers.

[0200] A computer program includes processor-executable instructions stored on at least one non-transitory, tangible, computer-readable medium. A computer program may also include or depend on stored data. A computer program may encompass a basic input / output system (BIOS) for interacting with the hardware of a special-purpose computer, device drivers for interacting with specific devices of a special-purpose computer, one or more operating systems, user applications, background services, background applications, etc.

[0201] Computer programs may include: (i) descriptive text to be parsed, such as HTML (Hypertext Markup Language), XML (Extensible Markup Language), or JSON (JavaScript Object Notation); (ii) assembly code; (iii) object code generated from source code by a compiler; (iv) source code executed by an interpreter; and (v) source code compiled and executed by a just-in-time (JIT) compiler, etc. As an example only, source code can be written using syntax in languages ​​including C, C++, C#, Objective-C, Swift, Haskell, Go, SQL, R, Lisp, Java®, Fortran, Perl, Pascal, Curl, OCaml, JavaScript®, HTML5 (Hypertext Markup Language version 5), Ada, ASP (Dynamic Server Web Pages), PHP (PHP: Hypertext Preprocessing Language), Scala, Eiffel, Smalltalk, Erlang, Ruby, Flash®, Visual Basic®, Lua, MATLAB, SIMULINK, and Python®.

Claims

1. A system for a vehicle, the system comprising: The vehicle has multiple sensors, wherein a first sensor of the multiple sensors is configured to detect lane markings on the road; and The controller is constructed as follows: Storing data from the plurality of sensors; and In response to receiving an indication based on data received from the first sensor indicating a false detection of lane markings on the road: Multiple processes are executed in parallel, the multiple processes being configured to: detect multiple causes of the false detection of lane markings based on stored data; Isolate one of the aforementioned causes as the root cause of the false detection of lane markings; and Based on the root cause of the false detection of lane markings, a response is provided to mitigate the false detection of lane markings on the road. The process is configured to detect whether the false detection of lane markings is caused by a malfunction of the first sensor, and One of the processes executed by the controller is configured to: confirm whether the first sensor is faulty by comparing the original image of the road captured by the first sensor with the original image of the road captured by the second sensor among the plurality of sensors, or by comparing objects detected in the image of the road captured by the first sensor with objects detected in the image of the road captured by the second sensor among the plurality of sensors.

2. The system according to claim 1, wherein, The process is configured to detect whether the false detection of lane markings is caused by any of the following: There are no lane markings on the road in question; There is rain or snow that makes the lane markings on the road blurred; There is glare or shadow that obscures lane markings on the road; There is an obstacle in the field of view of the first sensor; There are construction works on the road; The lane configuration changes of the road; and The road was unpaved.

3. The system according to claim 1, wherein, The response includes: a second vehicle following in front of the vehicle, notifying the occupants of the vehicle to take over control of driving the vehicle, switching to a second sensor among the plurality of sensors for lane marking detection, and / or dispatching services.

4. The system according to claim 1, wherein, One of the processes executed by the controller is configured to: Process the image captured by the first sensor; Perform clustering and filtering of pixels in the image; Perform first curve fitting and second curve fitting on the filtered pixels; and The presence of lane markings on the road is determined based on the second curve fitting.

5. The system according to claim 1, wherein, One of the processes executed by the controller is configured to: Process images captured by two of the plurality of sensors; For each of the two sensors, determine the number of images with and without lane markings; For each of the two sensors, calculate the ratio of the number of images with or without lane markings to the total number of images processed; as well as The presence of lane markings on the road is determined based on the ratio of the two sensors.

6. The system according to claim 1, wherein, One of the processes executed by the controller is configured to detect whether the road is unpaved based on images captured by the first sensor, map information about the road, and data from the inertial measurement unit in the vehicle.

7. The system according to claim 1, wherein, One of the processes executed by the controller is configured to use a machine learning model to detect the presence of rain or snow in an image captured by the first sensor.

8. The system according to claim 1, wherein, One of the processes executed by the controller is configured to detect the presence of glare or shadows in the image based on the pixel intensity in the image captured by the first sensor.

9. The system according to claim 1, wherein, One of the processes executed by the controller is configured to detect the presence of construction objects on the road based on construction objects detected in images captured by one of the plurality of sensors.

10. The system according to claim 1, wherein, One of the processes executed by the controller is configured to detect changes in the lane configuration of the road by detecting at least one of changes in the number of lanes in the road and changes in the relationship between the number of lanes and the lanes occupied by the vehicles.

11. The system according to claim 1, wherein, One of the processes executed by the controller is configured to: determine whether a second vehicle exists within a predetermined distance in front of the vehicle, and determine the size of the second vehicle, based on an image captured by a second sensor among the plurality of sensors and data received from a third sensor among the plurality of sensors.

12. The system according to claim 11, wherein, One of the processes executed by the controller is configured to determine whether to follow the second vehicle based on the predetermined distance.

13. The system according to claim 11, wherein, One of the processes executed by the controller is configured to determine whether to drive the vehicle on another trajectory in response to the predetermined distance being greater than a predetermined distance.

14. A method for a vehicle, the method comprising: Store data from multiple sensors on the vehicle; Receive an indication of a false detection of lane markings on the road based on data received from a first sensor among the plurality of sensors; In response to receiving the instruction, multiple processes are executed in parallel, the multiple processes being configured to: detect multiple causes of the false detection of lane markings based on stored data; One of the aforementioned reasons was identified as the root cause of the false detection of lane markings; as well as Based on the root cause of the false detection of lane markings, a response is provided to mitigate the false detection of lane markings on the road. The response includes: a second vehicle following in front of the vehicle; notifying the occupants of the vehicle to take over control of driving the vehicle; switching to a second sensor among the plurality of sensors for lane marking detection; and / or dispatching services. The method further includes: The process is used to detect whether the false detection of lane markings is caused by a malfunction of the first sensor; and The first sensor is confirmed to be faulty by comparing the original image of the road captured by the first sensor with the original image of the road captured by the second sensor among the plurality of sensors, or by comparing objects detected in the image of the road captured by the first sensor with objects detected in the image of the road captured by the second sensor among the plurality of sensors.

15. The method of claim 14, further comprising: The process is used to detect whether the false detection of lane markings is caused by any of the following: There are no lane markings on the road in question; There is rain or snow that makes the lane markings on the road blurred; There is glare or shadow that obscures lane markings on the road; There is an obstacle in the field of view of the first sensor; There are construction works on the road; The lane configuration changes of the road; and The road was unpaved.

16. The method of claim 14, further comprising: Process the image captured by the first sensor; Perform clustering and filtering of pixels in the image; Perform first curve fitting and second curve fitting on the filtered pixels; and The presence of lane markings on the road is determined based on the second curve fitting.

17. The method of claim 14, further comprising: Process images captured by two of the plurality of sensors; For each of the two sensors, determine the number of images with and without lane markings; For each of the two sensors, calculate the ratio of the number of images with or without lane markings to the total number of images processed; as well as The presence of lane markings on the road is determined based on the ratio of the two sensors.

Images