Driver-assisted autonomous driving system that utilizes the driver as a sensor to minimize driver takeovers

Abstract

An autonomous driving system includes a sensing module, an intervention module, and a vehicle control module. The sensing module detects an upcoming situation to be experienced by a vehicle. The intervention module: determines that there is a low level of confidence in a driving decision to be made for the upcoming situation; implements an intervention of an autonomous driving mode and indicates information related to the upcoming situation to a driver of the vehicle via an interface; requests assistance from the driver based on the upcoming situation by (i) indicating available options for the situation via the interface or (ii) requesting information from the driver to aid in making a driving decision; and determines whether input has been received from the driver via the interface, the input indicating (i) a selected one of the available options or (ii) the requested information. The vehicle control module autonomously drives the vehicle based on whether the input has been received.

Description

TECHNICAL FIELD

[0001] The information provided in this section is for the purpose of generally presenting the context of the disclosure. The work of the presently named inventors, to the extent described in this section, and aspects of the description that can not otherwise qualify as prior art at the time of

[0002] The present disclosure relates to driver assistance systems associated with partial and conditional automation levels. BACKGROUND

[0003] The Society of Automotive Engineers (SAE) J3016 standard is an official reference for the Department of Transportation (DoT) for defining levels of vehicle autonomy. There are six levels (i.e., 0-5), where level 0 refers to vehicles with no automation, level 1 refers to vehicles with driver assistance, level 2 refers to vehicles with partial automation, level 3 refers to vehicles with conditional automation, level 4 refers to vehicles with high automation, and level 5 refers to vehicles with full automation. At levels 0-1, driving control is dominated by the driver. At levels 2-3, driving control is preferably executed by a control system in the vehicle, but is handed over to the driver in a significant number of cases. Little or no interaction occurs between the driver of a level 2 or 3 vehicle and the control system in the vehicle. At levels 4-5, driving control is dominated by the control system in the vehicle, where preferably it is unmanned. SUMMARY

[0004] An autonomous driving system is provided and includes a sensing module, an intervention module, and a vehicle control module. The sensing module is configured to detect an upcoming situation to be experienced by a vehicle. The intervention module is configured to: determine that there is a low level of confidence in a driving decision to be made for the upcoming situation; implement an intervention of an autonomous driving mode and indicate information related to the upcoming situation to a driver of the vehicle via an interface; request assistance from the driver based on the upcoming situation by at least one of (i) indicating available options for the situation via the interface or (ii) requesting information from the driver to aid in making the driving decision; and determine whether input has been received from the driver via the interface indicating (i) a selected one of the available options or (ii) at least one of the requested information. The vehicle control module is configured to autonomously drive the vehicle based on whether the input has been received.

[0005] In other features, the interface includes at least one of a touchscreen, a display, a speaker, or a microphone.

[0006] In other features, the vehicle control module is configured to autonomously drive the vehicle based on the input in response to receiving the input from the driver.

[0007] In other features, the intervention module is configured to display a default option as one of the available options. The vehicle control module is configured to autonomously drive the vehicle based on the default option when no input from the driver is received via the interface within a set time period from when assistance is requested. The default option is the most conservative option of the available options.

[0008] In other features, the intervention module is configured to limit intervention by the driver to a limited role of providing driver input without allowing the driver to control actuators of the vehicle.

[0009] In other features, the selected one of the available options is a driving decision.

[0010] In other features, the vehicle control module is configured to make the driving decision based on the input and to autonomously drive the vehicle based on the driving decision.

[0011] In other features, the intervention module is configured to allow direct intervention by the driver in the autonomous driving mode when a set of options is available for a particular scenario.

[0012] In other features, the intervention module is configured to: determine that there is a low level of confidence in a sensing result obtained by the sensing module; allow indirect intervention by the driver in the autonomous driving mode to request information from the driver related to the sensing result; and modify the sensing result based on the requested information as received from the driver.

[0013] In other features, the intervention module is configured to solicit input from the driver when a situation is indicated to the driver via the interface.

[0014] In other features, the vehicle control module is configured to enforce the selected one of the available options selected by the driver for direct intervention.

[0015] In other features, the vehicle control module is configured to modify the sensing result based on the requested information received from the driver via the interface for indirect intervention.

[0016] Among other features, a method for operating an autonomous driving system is provided, the method comprising: detecting an upcoming situation to be experienced by a vehicle; determining the existence of a low confidence level for making a driving decision in response to the upcoming situation; implementing an intervention in an autonomous driving mode and instructing the driver of the vehicle, via an interface, information relating to the upcoming situation; requesting assistance from the driver based on the upcoming situation by (i) instructing available options for the situation via an interface or (ii) requesting information from the driver to assist in making a driving decision; determining whether input has been received from the driver via an interface, the input indicating (i) selection of one of the available options or (ii) at least one of the requested information; and autonomously driving the vehicle based on whether input has been received.

[0017] Among other features, the method further includes: autonomously driving the vehicle based on input received from the driver.

[0018] Among other features, the method further includes: displaying a default option as one of the available options; and autonomously driving the vehicle based on the default option during a set time period from when assistance is requested, without receiving input from the driver via the interface. The default option is the most conservative option among the available options.

[0019] Among other features, the method further includes limiting driver intervention to a restricted role that provides driver input, and disallowing the driver to control the actuators of the vehicle.

[0020] Among other features, the method further includes: for direct intervention, enforcing the selection of one of the available options as a driving decision; and for indirect intervention, making a driving decision based on input and autonomously driving the vehicle based on the driving decision.

[0021] Among other features, the method further includes: allowing direct intervention of the autonomous driving mode by the driver when a set of options is available for a specific scenario; determining the existence of a low confidence level for the obtained sensing results; allowing indirect intervention of the autonomous driving mode by the driver to request information from the driver related to the sensing results; and modifying the sensing results based on the requested information received from the driver.

[0022] Among other features, the method further includes soliciting input from the driver when instructing the driver on the situation via the interface.

[0023] Among other features, the method further includes: for direct intervention, enforcing selection of one of the available options chosen by the driver; and for indirect intervention, modifying the sensing results based on requested information received from the driver via the interface.

[0024] Other suitable aspects of the present disclosure will become apparent from the detailed description, the claims, and the appended drawings. The detailed description and the specific examples are intended for purposes of illustration only and are not intended to limit the scope of the present disclosure. BRIEF DESCRIPTION OF DRAWINGS

[0025] The present disclosure will become more fully understood from the detailed description, the claims, and the accompanying drawings, wherein:

[0026] Figure 1 is a functional block diagram of an example of an autonomous driving system including an intervention module according to the present disclosure;

[0027] Figure 2 is an elevational view of an example of a human-machine interface (HMI) according to the present disclosure;

[0028] Figure 3 a driver interaction and assisted autonomous driving method according to the present disclosure is shown;

[0029] Figure 4 a first example implementation of the method of Figure 3 is shown for a first example scenario;

[0030] Figure 5 is a second example implementation of the method of Figure 3 is shown for a second example scenario;

[0031] Figure 6 is a third example implementation of the method of Figure 3 is shown for a third example scenario; and

[0032] Figure 7 is an example perspective view of a third scenario according to the present disclosure.

[0033] In the drawings, re-usable reference numerals are used to identify like and / or similar elements. DETAILED DESCRIPTION

[0034] SAE J3016 Levels 2-3 (referred to as L2 and L3) have a significant amount of space for reducing situations where driving control needs to be handed over to a human driver. The examples set forth herein relate to reducing the number of situations where driving control is handed over to a human driver. Disclosed is a system in a vehicle that uses a human driver (referred to hereinafter as "driver") as a sensor and not as an actuator. In other words, when certain situations arise, the system requests input from the driver and makes a decision based on the driver input that affects how the actuators (steering actuators, acceleration actuators, deceleration actuators, etc.) are controlled. The system controls the actuators, not the driver. The system includes an intervention module that interacts with the driver, including providing different options for the driver to select from and / or requesting certain information. The driver then makes a selection and / or provides the requested information. The system performs the driving operation based on the feedback provided by the driver. This is done without disengaging from autonomous driving and the driver not taking over driving control.

[0035] The disclosed system helps the driver provide assistance under certain driving situations (referred to as scenarios). This is done while performing (or in parallel with) effective machine learning and training operations that are based on the driver decisions for improved current and future decisions made by the system. The driver assistance reduces the need to disengage from the autonomous (or self-driving) mode. The autonomous driving mode refers to when the system autonomously controls the driving operations (e.g., steering, acceleration, deceleration, etc.). When disengaged, the driver takes control and drives the vehicle.

[0036] The disclosed system performs a cost-effective method to improve the user experience for autonomous driving vehicles, with a focus on supplementing the system capabilities associated with SAE J3016 Levels L2 and L3. This is done by requesting assistance from the driver for driving decisions made by the system. The system makes the driver aware of the path planning, motion control, and certain information and / or possible decisions that can be made with a level of confidence. Then, for information and / or driving decisions where the level of confidence is low, the system can request feedback from the driver. The system communicates various information, e.g., via an HMI (e.g., infotainment system and / or one or more other displays), to allow the driver to visualize the provided information. The driver can decide whether to intervene, if appropriate. The HMI can communicate the intervention information collected from the driver to an intervention module and / or other modules of the vehicle. If information is requested and the driver decides not to intervene and / or provide input, the system operates conservatively (i.e., in a safe manner) and selects a default option that is safe for the vehicle, the vehicle occupants, and surrounding objects.

[0037] The disclosed examples include a system that receives driver assistance, which is different from systems in conventional autonomous vehicles, which typically have a focus on vehicles that assist drivers. Thus, the disclosed examples supplement existing SAE J3016 L2 and L3 operational profiles.

[0038] The systems disclosed herein make autonomous driving decisions that have improved adaptability to road situations, especially to scenarios in which the system has a correspondingly low level of confidence in making a driving decision. The methods implemented by the system reduce the need to disengage autonomous driving for certain situations, which would conventionally require a driver to take over driving control. This allows the system to maintain autonomous driving control for longer and more continuous periods of time. The reduced number of disengagements results in a significant improvement in the driving experience for L2 and L3 autonomous vehicles and / or other vehicles that operate under the assumption that a human driver is present, preferably where driving control is implemented by a system in the vehicle and not by a human driver. The disclosed examples are applicable to passenger vehicles and other forms of transportation.

[0039] The systems in the disclosed vehicles include path planning and vehicle motion control in L2 and L3 vehicles, as well as intervention operations that are conventionally not performed. For example, driver intervention is communicated via an HMI and utilized as an active source for driving decisions with autonomous features. This improves the driving experience by reducing disengagement frequency. When a disengagement occurs, driving decisions are made by the driver and enforced since the vehicle system disengages. Driving decisions are made and enforced more continuously by the vehicle system with fewer disengagements occurring.

[0040] As used herein, the term "scenario" can refer to an abstract representation of a road situation, especially to road situations in which it is difficult for an autonomous driving system to make timely decisions with a high level of confidence. As further described below, the term "scenario" can refer to "scenario inference" or "scenario-specific event solicitation." As used herein, the term "event" can refer to an intervention by a driver and corresponds to one of a limited number of types of scenarios that are feasible for processing by an autonomous driving system. As used herein, the phrase "event processing" can refer to monitoring an intervention event by a driver and modifying sensory results and / or driving decisions based on input received from the driver. These terms and phrases are further defined in the following description in the sense of processing operations. As several examples, the phrase "driving decision" can refer to a selection by an on-board module of a vehicle of one from a plurality of options for a current specific task and / or performance of an operation (e.g., a turn, acceleration, deceleration, moving forward, moving backward, and / or a stay-still operation) based on information requested and received from a driver.

[0041] Figure 1 An autonomous driving system 100 is shown, which is implemented within a vehicle 102 and can be referred to as an in-vehicle system and / or an advanced driver assistance system. As described and disclosed herein, the vehicle 102 can be a L2 or L3 level vehicle configured to perform autonomous driving operations. The autonomous driving system 100 includes a vehicle control module 104, which can communicate with and / or control the operation of various modules and systems of the vehicle 102.

[0042] The vehicle control module 104 can include one or more modules, such as a sensing module 105, for detecting the environment and situations that the vehicle 102 is currently experiencing and / or about to experience. As disclosed herein, the one or more modules can also include an intervention module 106 for implementing driver intervention. This includes assisting and augmenting autonomous driving decisions and associated operations. The one or more modules can be software modules that include executable code for processing the flow of scenarios, events, and event handling associated operations. Some of these operations are illustrated in Figures 3-7

[0043] The autonomous driving system 100 includes a memory 110 and sensors 112. The memory 110 can store parameters 114, data 116, and algorithms 118 (e.g., autonomous driving algorithms, machine learning algorithms, etc.). The memory 110 can store multiple sets of options 119 for a limited number of scenarios and / or request information from a driver for corresponding scenarios. The sensors 112 can be located throughout the vehicle 102 and include cameras 120, infrared (IR) sensors 122, other object detection sensors (e.g., radar and lidar sensors) 124, and / or other sensors 126. The other sensors 126 can include yaw rate sensors, accelerometers, global positioning system (GPS) sensors, air flow sensors, temperature sensors, pressure sensors, vehicle speed sensors, motor speed sensors, etc.

[0044] The vehicle control module 104 and the sensors 112 can communicate directly with each other, can communicate with each other via a controller area network (CAN) bus 130, and / or via an Ethernet switch 132. In the example shown, the sensors 112 are connected to the vehicle control module 104 via the Ethernet switch 132, but can also or alternatively be connected directly to the vehicle control module 104 and / or the CAN bus 130.​

[0045] The vehicle 102 can further include other control modules, for example, a chassis control module 140 that controls a torque source including one or more electric motors 142 and one or more engines (one engine 144 is shown). The chassis control module 140 can control the distribution of output torque to axles of the vehicle 102 via the torque source. The chassis control module 140 can control operation of a propulsion system 146 that includes the electric motor(s) 142 and the engine(s) 144. Each of the engines can include a starter motor 150, a fuel system 152, an ignition system 154, and a throttle system 156.

[0046] In one embodiment, the vehicle control module 104 is a body control module (BCM) that communicates with and / or controls operation of a telematics module 161, a steering system 162, a braking system 163, a navigation system 164, an infotainment system 166, other actuators 172 and devices 174, and other vehicle systems and modules 176. The navigation system 164 can include a GPS 178. The other actuators 172 can include a steering actuator and / or other actuators. The modules and systems 104, 140, 161, 162, 164, 166, 176 can communicate with each other via the CAN bus 130. A power supply 180 can be included and power the vehicle control module 104 and other systems, modules, controllers, memories, devices, and / or components. The power supply 180 can include one or more batteries and / or other power sources. The vehicle control module 104 can perform countermeasures and / or autonomous vehicle operations based on a planned trajectory of the vehicle 102, detected objects, locations of detected objects, and / or other related operations and / or parameters. This can include controlling the torque source and actuators and providing images, indications, and / or instructions via the infotainment system 166.

[0047] The telematics module 161 can include a transceiver 182 and a telematics control module 184 that can be used to communicate with other vehicles, networks, infrastructure devices (e.g., vehicles, traffic signs, buildings, base stations, etc.), edge computing devices, and / or cloud-based devices. The vehicle control module 104 can control the modules and systems 140, 161, 162, 163, 164, 166, 176, and other actuators, devices, and systems (e.g., actuators 172 and devices 174). This control can be based on data from the sensors 112.

[0048] The sensing module 105 can determine the vehicle's surroundings, position relative to objects, upcoming situations, etc. based on information received from the sensors 112, the telematics module 161, the navigation system 164, and / or other devices, modules, and / or systems mentioned herein.

[0049] The intervention module 106 performs intervention operations using an HMI, which can include the infotainment system 166, the microphone 190, and other input devices 192 (e.g., buttons, dials, switches, etc.) and / or output devices (e.g., speakers, displays, lights, etc.). The infotainment system 166 can include the speakers 194 and the display 196. The display can include a touchscreen, a dashboard display, a center console display, a heads-up display (HUD), etc. The HMI can be located in the vehicle's cabin, and can be located in the driver's field of view. The HMI can be located in the driver's field of view. Figure 2 Other examples are shown and described, which show an example HMI 200.

[0050] The HMI 200 has a corresponding user interface (UI) layout for performing interactive operations. Other user interface layouts can be used. The HMI 200 can include a dashboard 202, a steering wheel 204, and a HUD (which can be represented by the view image by the box 206, and can be on the corresponding vehicle's windshield). The dashboard 202 can include various displays 208, 210, 212, which can be touchscreens for receiving input from the driver. The display 212 can include a built-in microphone. The dashboard 202 can further include various other input devices, e.g., buttons, switches, dials, etc. (e.g., buttons 216, 218, 220, 221). The HMI 200 can include output devices other than the displays 206, 208, 210, 212, e.g., one or more speakers (e.g., the speakers 221 or the speakers 194). The steering wheel 204 can include a light-emitting display (LED) strip 222, which includes LEDs. The button 220 or other input device can be dedicated to implementing driver assistance (or intervention) by the driver. The HMI 200 can be used to implement automated driving system functions, including accepting assistance from the driver, where the button 220 is a dedicated component for this function. Figure 1

[0051] The intervention module 106 can provide information requests via the displays 206, 208, 210, 212, the speakers, and / or the LED strip 222. The intervention module 106 can receive input from the driver via the displays 206, 208, 210, 212, the buttons 216, 218, 220, 221, the microphone of the display 212, or the microphone 190 and / or other input devices. Figure 1

[0052] Figure 3 ​​Driver interaction and assisted autonomous driving methods are shown. Operations can be performed by the vehicle control module 104, the intervention module 106, the HMI 200, and / or other devices, modules, and systems mentioned herein. Operations can be performed iteratively.

[0053] The method can begin at 300. At 302, the vehicle control module 104 can collect sensor data from the sensors 112. At 304, the vehicle control module 104 can identify an environmental situation (or scenario). This can be an abstract representation of a road situation. At 306, the sensing module 105 can perform vehicle sensing operations, including performing operations based on environmental conditions if possible. This can include performing re-decision operations based on received one or more driver inputs. At 306A, the sensing module 105 and / or the intervention module 106 can perform operations according to default operations if no user input is received or based on improved decision input received from the driver.

[0054] At 308, the intervention module 106 can perform a scenario inference operation to indicate to the driver the environmental situation. This can include indicating the situation and / or conditions for which a near-term decision will be made. Relevant notification information can be communicated to the driver via the HMI 200 and / or other output devices mentioned herein. The autonomous driving system 100, which is responsible for sensing, decision making, and HMI interaction, determines via the intervention module 106 that it is difficult to make a high-confidence and timely driving decision for the current scenario. The scenario and / or aspects thereof can be displayed to be visualized by the driver via the HMI 200. This can include text, color indications, image displays, and the like, and can also be accompanied by sound notifications.

[0055] At 310, the intervention module 106 can perform a scenario-specific event solicitation operation to obtain input from the driver. Operation 310 or portions thereof can be performed while performing operation 308. Operation 310 can occur logically after operation 308. The intervention module 106 can indicate to the driver that there is a situation while requesting feedback and / or information from the driver.

[0056] At 312, the intervention module 106 can perform an event operation to obtain driver input. At 312A, the intervention module 106 limits the event handling to a limited type using a user interface (e.g., the HMI 200).

[0057] The driver is made aware of the scenario and / or situation as represented by block A. The driver inputs are referred to as interventions for near-term path planning and / or motion control. The driver can decide which option to select, what modification to perform, and / or what information to provide, and can communicate this intervention indication via voice commands and / or touch actions, which are received as inputs by the intervention module 106. While the driver can input the selection of a default option, the driver can not perform any action when the driver decides to proceed with the default option. If no action is taken by the driver, no input is received, and the default option is automatically selected.

[0058] By providing input and / or information assistance based on events, the driver performs a limited role. The driver can provide inputs via the HMI 200 without touching the vehicle's steering wheel, accelerator pedal, and / or decelerator (e.g., brake) pedal. The driver's role can vary based on the event and / or scenario. The driver's role is similar to a sensor, but with light-weight reasoning capability. The driver does not act as an actuator and, for example, does not turn the steering wheel and / or depress the accelerator or decelerator pedal. This is represented by block B. The driver provides observations and guidance, but does not operate (or actuate) the vehicle's steering, acceleration, and / or deceleration controls. This is represented by block C.

[0059] At 312B, the intervention module 106 receives direct intervention inputs. In the case of direct intervention, the intervention module 106 can present a set of decision options via the HMI 200 from which the driver can choose. The options can include a default option. The default option can be the most conservative (or safest) option among the available options. The driver is not required to explicitly select the default option, but can be enforced based on the driver not selecting any of the options. The set of options can remain available for selection for a limited amount of time, and thereafter, if the driver does not select any of the options, the default option is enforced. The driver can provide inputs via the HMI 200 via voice and / or touch inputs for selecting the scenario-specific options. The driver can be provided with a limited set of options, and the driver can select one of the provided limited set of options. The driver can be prevented from selecting or inputting an option that is not in the limited set of options. As an example, the limited set of options can include "wait," "turn," "accelerate" options for a particular scenario, and the driver can be prevented from, for example, requesting to make a "U-turn."

[0060] At 312C, the intervention module 106 receives indirect intervention inputs. Indirect intervention occurs based on sensing results that lead to a driving decision. This applies to scenarios with high latency time tolerance. In the case of indirect intervention, the intervention module 106 prompts for applicable modifications to the sensing results for the driver to perform. As an example, the intervention module 106 can request certain information from the driver. Figure 6An example is provided where the boundaries of a request for the driver to select and / or indicate a route for the vehicle are defined. The time windows used to allow the driver to select an option, accept an option, and / or provide the requested information can be configured for each specific scenario.

[0061] At 314, intervention module 106 performs event processing operations to enforce an input response received from the driver and / or obtain additional driver input. In the case of direct intervention in driving decisions, an alternative decision is immediately enforced, and this results in a re-evaluation of subsequent recent driving decisions (if any). In the case of indirect intervention in sensing results, further actions may be performed by the driver, including providing additional input so that sensing module 105 generates improved sensing results. The driving decision is enforced after it is re-evaluated based on the improved sensing results.

[0062] Figure 3 This illustrates the processing flow involving human intervention and related limitations when making driving decisions. Operations 306, 308, 310, 312, and 314 are parts of a control loop that can be modified to take into account time constraints on driving decisions by limiting applicable interventions to those feasible for processing by a set of pre-loaded (but scalable) event handling methods. These event handling methods may be implemented by one or more modules (e.g., vehicle control module 104 and / or intervention module 106) and are stored... Figure 1 The memory 110 is used for this purpose. When multiple coordinating software interconnect modules are included to implement the event handling method, the modules can be stored in the memory 110. As an example, modules may be included to perform each of operations 306, 308, 310, 312, and 314.

[0063] exist Figure 3 In one implementation of the method, the following method is used Figure 2 The display 212, speaker 221, and microphone (e.g., the microphone of the display 212 or...) Figure 1 The microphone 190). In this embodiment, the HUD 206, LED strip 222, and button 218 may be omitted. Sensors, such as cameras, radar sensors, lidar sensors, and GPS sensors, may be used to detect objects, map the surrounding environment, and determine the position of the corresponding vehicle relative to other objects. Vehicle-to-the-outside (V2X) airborne units may also be used, for example, Figure 1 Remote information processing module 161.

[0064] During operation 308, the HUD 206, LED strip 222, speaker 221, and display 212 are used to provide notifications to the driver. During operation 310, the display 212 is used to display, for example, the selected options from which to choose, with the default option highlighted. The selected option can then be followed, executed, and / or implemented.

[0065] During operation 312, the intervention module 106 can receive input from the driver. The driver can communicate the selected option to the systems in the vehicle using various methods. As an example, the selected option can be communicated by having the driver press one or more buttons, touch the display 212, and / or provide a voice command. A combination of buttons can be pressed sequentially or simultaneously, and / or a voice command can be provided. The driver can hold or press a button or icon once, depending on the user configuration and / or the type of intervention expected (e.g., voice or touch control). This can include touching an option and / or icon displayed on the display 212.

[0066] During operation 314, the intervention module 106 can display various details of the interaction session with the driver on the display 212 to allow the driver to visualize aspects of the corresponding situation. This can include having the intervention module 106 display the autonomous driving decision that has involved the driver interaction and the driver input. This can be implemented in parallel with displaying the resulting driver decision (or selected option and / or provided information) that is enforced on the display 212.

[0067] The following Figures 4-6 can be implemented via Figure 1 the autonomous driving system 100 and Figure 2 the HMI 200. Figure 4 A first example implementation of the method of Figure 3 is shown for a first example scenario. The method can begin at 400. At 402, the vehicle control module 104 can collect sensor data from the sensors 112. At 404, the vehicle control module 104 can identify an environmental situation (or scenario). This can be an abstract representation of a road situation.

[0068] At 406, the notification communicated by the intervention module 106 is that the system has a low confidence level associated with making a decision regarding a near-term driving situation. As an example, the near-term driving situation can include determining whether to make a right turn at an upcoming or current red light. The challenge for this scenario comes from the difficulty for the intervention module 106 and / or the sensing module 105 to determine whether it is safe to turn into the intended lane (with potentially conflicting traffic) during a specific time window (e.g., several seconds). For this scenario, a direct intervention for the driving decision is applied. The notification can be communicated to the driver via the HMI 200. When the vehicle is about to make a right turn (with significant traffic along the intended direction) at a red light, an indication that the near-term driving decision is preferably with low confidence can be communicated on the HMI 200, where the driver is indicated to visualize the situation in preparation for the intervention.

[0069] At 408, the intervention module 106 provides the driver with options for the driving situation. The options can include: "wait" (default), meaning that the vehicle (referred to as the host vehicle) will wait until no vehicles are detected driving along the path of the intended direction or the traffic light turns green for the current direction; "turn," meaning that the vehicle proceeds at a default speed and acceleration and then turns right; and "accelerate," meaning that the host vehicle proceeds at a speed under the speed limit on the road it is currently traveling on, subject to safety constraints, and then turns right.

[0070] At 410, the intervention module 106 receives a selected driver input based on the driver's observation of the nearby traffic and the timely communication of the selected option by the driver. This operation relies on the driver's observation of the nearby traffic and the timely communication of the selected option, which is feasible for moderately experienced and experienced drivers.

[0071] At 412, the vehicle control module 104 enforces the selected option (referred to as the driving decision) made by the driver and autonomously drives the vehicle based on the driving decision. The driving decision can be the option selected by the driver and enforced. The method can end at 414.

[0072] Figure 5 is shown Figure 3A second example implementation of the method for a second example scenario. The challenge of this scenario comes from the difficulty for the automated driving system to determine the exact lane the vehicle will change to when sensing a certain temporary road situation. The situation can be sensed by the automated driving system based on collected sensor data and / or based on information received from devices outside the vehicle via relevant V2X messages. For example, in case a relevant V2X message is received, the intervention module 106 is able to detect that there is a temporary road situation (e.g. road works or a traffic accident) at a certain distance ahead. Due to technical limitations caused by both V2X standardization (e.g. messages mainly intended for informing human drivers) and solution deployment (lack of vehicle-side and / or infrastructure-side capabilities for timely reflecting precise road situations), the V2X message can not include a precise indication of the lane closure and / or the closure time period. For this reason, the V2X message is not sufficient for directly deriving a driving decision for the scenario. Similarly, the sensing module 105 also has difficulties to deduce which exact lanes are closed as an inference of the temporary road situation.

[0073] The method can start at 500. At 502, the vehicle control module 104 can collect sensor data from the sensors 112. At 504, the vehicle control module 104 can identify an environmental situation (or scenario). This can be an abstract representation of a road situation.

[0074] At 506, the notification communicated by the intervention module 106 is that the automated driving system is not sure which traffic lane the vehicle should head towards. An advance indication of a lane change is indicated, which is triggered by the detected temporary road situation. When the sensing module 105 and / or the intervention module 106 senses a certain temporary road situation at a certain distance ahead, an indication of "not sure which lane to head towards" is communicated on the HMI 200, where the driver is indicated a visualized situation and other forms of notification to prepare for intervention.

[0075] At 508, the intervention module 106 provides lane options to the driver for the current driving situation via the HMI 200. The options can include: "stay", stay on the current lane, the default option; "left 1", change to the left first lane subject to safety guarantees; "left 2", change to the left second lane subject to safety guarantees; "right 1", change to the right first lane subject to safety guarantees; and "right 2", change to the right second lane subject to safety guarantees, etc. For example, these options can be displayed for viewing and selection.

[0076] At 510, the intervention module 106 receives a driver input via the HMI 200 indicating a selection made by the driver. This is based on the observation of nearby traffic by the driver and timely communication of the selected option. This relies on the driver determining which lane the vehicle will change to by observing the actions of the leading vehicles and timely communication of the selected option. This is feasible for moderately experienced and experienced drivers.

[0077] At 512, the vehicle control module 104 enforces the driving decision made by the driver (which is one of the options communicated to the driver) and autonomously drives the vehicle based on the driving decision. The method can end at 514.

[0078] Figure 6 A third example implementation of the method of Figure 3 is shown for a third example scenario. The sensing module 105 can sense a narrow driving lane 700 with an L-shaped turn ahead and a non-smooth corner, as shown in Figure 7 . Figure 7 The vehicle 102 is shown making a right turn, indicated by the arrow 702. The driving lane 700 is near a building structure 704 (e.g., a business, a residence, etc.). The example scenario includes the vehicle 102 performing a right turn on a narrow driving lane with an example width of 2.5 meters, with obstacles near the driving lane. This can be a road situation in a rural residential area.

[0079] The challenge for this scenario comes from the difficulty of the sensing module 105 to detect the exact driving lane boundaries, especially in some rural areas with irregular driving lane appearance and topology. Due to the low speed nature of this scenario, which results in a high latency tolerance, an indirect intervention on the sensing results is applied for this scenario.

[0080] The method can begin at 600. At 602, the vehicle control module 104 can collect sensor data from the sensors 112. At 604, the vehicle control module 104 can identify an environmental situation (or scenario). This can be an abstract representation of a road situation.

[0081] At 606, the intervention module 106 communicates a notification via the HMI 200 that the autonomous driving system 100 has a low level of confidence associated with determining the driving lane (or path) boundaries.

[0082] At 608, the intervention module 106 provides the driver with the boundary options via the HMI 200, or requests the driver to indicate and / or draw the boundary. This can be done, for example, via the display 212, which can be a touch screen. The driver can draw the boundary of the display 212 over the environmental image displayed on the display 212 via the driver's finger. When the sensing module 105 has a low confidence level for the determined lane boundary, in addition to notifying the driver to prepare for intervention, an indication of "low confidence in lane boundary detection" is communicated on the HMI 200. The vehicle control module 104 can adjust the speed of the vehicle 102 to ensure sufficient safety. The driver is prompted to, for example, via the HMI 200, touch to control the delineation of the lane boundary to modify the original sensing result. If it is not safe enough to move forward, the vehicle 102 can automatically slow down (e.g., apply brakes).

[0083] At 610, the intervention module 106 receives the driver input regarding the boundary. The driver provides the required profile for the lane boundary, and confirms the profile.

[0084] At 612, the vehicle control module 104 and / or the intervention module 106 use the lane boundary profile as improved sensing result, based on which driving decisions are made. The lane boundary profile provided by the driver is used as improved sensing result, after which driving decisions are (re)derived and enforced. For example, after the boundary is identified by the driver, the vehicle control module 104 controls the steering, acceleration, and deceleration actuators to drive the vehicle forward, and make a right turn based on the indicated boundary, as shown in Figure 7 .

[0085] If no driver input is received, the vehicle control module 104 can execute a default safe option, which can include staying stationary or moving slowly in the forward direction. The vehicle control module 104 executes the option determined to be the safe option. The method can end at 614.

[0086] The disclosed examples include performing a method to facilitate the preferability of automated driving decisions based on assistance provided by a driver for certain situations. Driver feedback is provided via an HMI that handles intervention from the driver (e.g., voice and / or touch input). The disclosed system mitigates the preferability and robustness deficiencies associated with advanced driver assistance systems in vehicles. The disclosed system also minimizes the hardware complexity required for the modules that perform the sensing and driving decision operations.

[0087] While some of the methods described above are described with respect to certain example scenarios, the methods disclosed herein are applicable to other scenarios. The systems disclosed herein are applicable to performing processing operations for a variety of scenarios and events. The disclosed systems are able to receive software upgrades to recognize additional scenarios and perform similar operations for the additional scenarios.

[0088] Examples reduce the need for repeated interventions. Examples can include storing multiple sets of options for respective scenarios that can occur in different geographic regions. This can be due to country and region traffic regulations, local driving culture, and individual driving habits, for example. These options can be pre-configured during manufacturing and / or prompted for configuration at first use. The options can be used as input for making driving decisions, in particular for determining which of several possible options to present to the driver as a default option, as applicable for a particular scenario.

[0089] Further, for decisional difficulties from very challenging scenarios and not solvable by configuring preference options, a machine learning based function can be performed to learn how the driver intervened or took direct control of the vehicle (provided minimum safety conditions were met). This case enables the autonomous driving system to present a better "default option" for driving decisions based on a sufficient amount of accumulated data related to a certain challenging (but not frequently encountered) scenario.

[0090] The autonomous driving system can be configured to process the accumulated data locally or upload it to a cloud server for processing as training data, and over-the-air (OTA) updates of software modules can be used to deploy training results that are suitable for large-scale deployment (i.e., independent of individual driving habits).

[0091] The foregoing description is merely illustrative in nature and is not intended to limit the disclosure, its application or uses in any way. The broad teachings of the disclosure can be implemented in a variety of forms. Therefore, while this disclosure includes particular examples, the true scope of the disclosure should not be so limited since other modifications will become apparent upon a study of the drawings, specification, and claims. It should be understood that one or more steps within a method can be executed in different order (or concurrently) without altering the principles of the disclosure. Also, although each of the embodiments described herein is described as having certain features, one or more of those features described with respect to any embodiment can be implemented and / or combined with features of the other embodiments, even if that combination is not explicitly described. In other words, the described embodiments are not mutually exclusive, and permutations of one or more embodiments with respect to each other remain within the scope of this disclosure.

[0092] Various terminology (including "connected," "engaged," "coupled," "adjacent," "immediately adjacent," "on top of," "above," "below," and "disposed") is used herein to describe spatial and functional relationships between elements (e.g., modules, circuit elements, semiconductor layers, etc.). Unless explicitly described as "direct," a relationship between or among two elements as described in the above disclosure can be a direct relationship where no other intervening elements are present between or among the two elements, but can also be an indirect relationship where one or more intervening elements are present (either spatially or functionally) between or among the two elements. As used herein, the phrase at least one of A, B, and C should be construed to mean a logical (A OR B OR C) using non-exclusive logical OR, and should not be construed to mean "at least one of A and / or at least one of B and / or at least one of C."

[0093] In the drawings, the direction of an arrow (as indicated by the arrow) generally demonstrates the information flow (e.g., data or instructions) illustrated. For example, when elements A and B exchange a variety of information but the information transferred from A to B is relevant for the illustration, an arrow can point from A to B. This one-way arrow does not imply that no other information is transferred from B to A. In addition, for information sent from A to B, B can send a request for the information or a confirmation of receipt of the information back to A.

[0094] In this application, including the definition below, the term "circuit" can be used in lieu of the term "module" or the term "controller." The term "module" can refer to, be part of, or include an Application Specific Integrated Circuit (ASIC); a digital, analog, or mixed analog / digital discrete circuit; a digital, analog, or mixed analog / digital integrated circuit; a combinational logic circuit; a field programmable gate array (FPGA); a processor circuit (shared, dedicated, or group) that executes code; a memory circuit (shared, dedicated, or group) that stores code executed by the processor circuit; other suitable hardware components that provide the described functionality; or a combination of some or all of the above, such as in a system-on-chip.

[0095] A module can include one or more interface circuits. In some examples, the interface circuits can include wired or wireless interfaces that are connected to a local area network (LAN), the Internet, a wide area network (WAN), or combinations thereof. The functionality of any given module of the present disclosure can be distributed among multiple modules that are connected via interface circuits. For example, a plurality of modules can allow load balancing. In another example, a server (also known as remote, or cloud) module can accomplish some functionality on behalf of a client module.

[0096] As used above, the term code can include software, firmware, and / or microcode, and can refer to program, routines, functions, classes, data structures, and / or objects. The term shared processor circuitry encompasses a single processor circuitry executing some or all code from multiple modules. The term group processor circuitry encompasses processor circuitry in combination with an additional processor circuitry executing some or all code from one or more modules. References to multiple processor circuitries encompass multiple processor circuitries on discrete dies, multiple processor circuitries on a single die, multiple cores of a single processor circuitry, multiple threads of a single processor circuitry, or a combination of the above. The term shared memory circuitry encompasses a single memory circuitry storing some or all code from multiple modules. The term group memory circuitry encompasses memory circuitry in combination with an additional memory storing some or all code from one or more modules.

[0097] The term memory circuitry is a subset of the term computer readable medium. As used herein, the term computer readable medium does not encompass transitory propagating signals per se (e.g., a propagating electromagnetic wave carrying the code). The term computer readable medium therefore can be regarded as tangibly embodied and non-transitory. Non-limiting examples of non-transitory computer readable media are nonvolatile memory circuitry (e.g., flash memory circuitry, erasable programmable read only memory circuitry, or mask read only memory circuitry), volatile memory circuitry (e.g., static random access memory circuitry or dynamic random access memory circuitry), magnetic storage media (e.g., magnetic tapes or magnetic hard drives), and optical storage media (e.g., optical discs, optical tape, or machine readable storage wrapped in shrink wrap or as archives on

[0098] The apparatus and methods described in this application can be implemented partly or entirely by a special purpose computer, formed by configuring a general purpose computer to perform one or more particular functions implemented in computer programs. The functional blocks, flowchart components, and other elements described above act as software specifications, which can be converted to computer programs by routine work of a skilled programmer or engineer.

[0099] A computer program includes processor executable instructions stored on at least one non-transitory computer readable medium. A computer program can also include or rely on stored data. A computer program can encompass a Basic Input / Output system (BIOS) that interacts with hardware of the special purpose computer, device drivers that interact with particular devices of the special purpose computer, one or more operating systems, user applications, background services, background applications, etc.

[0100] A computer program can include: (i) a sequence of instructions (program code) being executed by said at least one processor, (ii) a piece of software causing said at least one processor to execute a sequence of instructions (program code), (iii) a computer program being written to, for example, implement a descriptor according to the application, (iv) a piece of software, and / or (v) a computer program product. The software can include, but does not necessarily include, a computer program guiding the operation of the computer along the lines of the method of the application, and / or a computer program product. The computer program can include: (i) to be parsed descriptive text, for example, HTML (HyperText Markup Language), XML (Extensible Markup Language), or JSON (JavaScript Object Notation), (ii) assembly code, (iii) object code generated from source code by a compiler, (iv) source code for execution by an interpreter, (v) source code for compilation and execution by a just-in-time compiler, etc. By way of example only, source code can be written using a syntax from a language chosen from: C, C++, C#, Objective-C, Swift, Haskell, Go, SQL, R, Lisp, Java®, Fortran, Perl, Pascal, Curl, OCaml, Javascript®, HTML5 (HyperText Markup Language, 5th revision), Ada, ASP (Active Server Pages), PHP (PHP: Hypertext Preprocessor), Scala, Eiffel, Smalltalk, Erlang, Ruby, Flash®, Visual Basic®, Lua, MATLAB, SIMULINK, and Python®.

Claims

1. An autonomous driving system, comprising: a sensing module configured to detect an upcoming situation to be experienced by a vehicle; an intervention module configured to: determine whether a level of confidence in a driving decision to be made for the upcoming situation is at a level that requests driver assistance / feedback, the driving decision referring to steering, acceleration, deceleration, moving forward, moving backward, and / or staying stationary operation, based on the level of confidence in the driving decision being at the level that requests driver assistance / feedback, enable intervention of an autonomous driving mode and indicate information related to the upcoming situation to a driver of the vehicle via an interface, request assistance from the driver based on the upcoming situation by at least one of (i) indicating available options for the situation via the interface or (ii) requesting information from the driver to aid in making the driving decision, and determine whether input has been received from the driver via the interface, the input indicating (i) a selected one of the available options or (ii) at least one of the requested information; and a vehicle control module configured to autonomously drive the vehicle based on whether the input has been received to aid and augment autonomous driving decisions and associated operations.

2. The autonomous driving system of claim 1, wherein: the intervention module is configured to display a default option as one of the available options; the vehicle control module is configured to, within a set time period from when the assistance is requested, autonomously drive the vehicle based on the default option when input from the driver is not received via the interface; and the default option is a most conservative option of the available options.

3. The autonomous driving system of claim 1, wherein, the intervention module is configured to limit the intervention by the driver to a limited role of providing driver input without allowing the driver to control actuators of the vehicle.

4. The autonomous driving system of claim 1, wherein, the selected one of the available options is the driving decision.

5. The autonomous driving system of claim 1, wherein, the vehicle control module is configured to make the driving decision based on the input and autonomously drive the vehicle based on the driving decision.

6. The autonomous driving system of claim 1, wherein, the intervention module is configured to allow direct intervention of the autonomous driving mode by the driver when a set of options is available for a particular scenario.

7. The autonomous driving system of claim 1, wherein, the intervention module is configured to: determine that there is a low level of confidence in a sensing result obtained by the sensing module; allow indirect intervention of the autonomous driving mode by the driver to request information from the driver related to the sensing result; and modify the sensing result based on the requested information as received from the driver. the intervention module is configured to solicit the input from the driver when indicating the situation to the driver via the interface.

8. The autonomous driving system of claim 1, wherein, the vehicle control module is configured to, for direct intervention, enforce the selected one of the available options selected by the driver.

9. The autonomous driving system of claim 1, wherein, the vehicle control module is configured to, for indirect intervention, modify a sensing result based on the requested information received from the driver via the interface.

10. The autonomous driving system of claim 1, wherein, ​

Images