Adaptive alarm and multi-modal dialog HMI for assisted and automated driving systems

Abstract

An aided driving system of a host vehicle includes an adaptive alert module configured to adaptively generate an alert having a variable duration and intensity; a multi-modal dialog interface module configured to implement a large language model to have a dialog with an occupant of the host vehicle using a plurality of modalities; and an aided driving module configured to operate in an aided driving mode and, while in the aided driving mode, perform a plurality of precondition checks, the adaptive alert module is configured to provide an alert to at least one of: i) the adaptive alert module is to operate in an adaptive alert mode and provide the alert via a human machine interface (HMI) of the host vehicle; and ii) the multi-modal dialog interface module is to perform a multi-modal dialog with the occupant via the HMI.

Description

BACKGROUND

[0001] The information provided in this section is presented to provide a context for the present disclosure. Neither the identification of the work of the presently named inventors nor the description of aspects described in this section are necessarily implied to be prior art to the present disclosure.

[0002] The present disclosure relates to assisted and automated driving systems.

[0003] A vehicle can have an assisted and automated driving system implemented to help a driver or a vehicle occupant drive the vehicle. As an example, a vehicle can have a hands-free driving system that controls steering, braking, and acceleration operations during at least a portion of a trip. During hands-free driving, a driver does not control driving operations, but can intervene if the driver disagrees with actions being performed by the vehicle. This can occur, for example, by the driver tapping on a brake pedal or an accelerator pedal and taking over control of the vehicle steering, braking, and acceleration operations. SUMMARY

[0004] An assisted driving system of a host vehicle is disclosed. The assisted driving system includes an adaptive alert module configured to adaptively generate an alert having a variable duration and intensity, a multi-modal dialog interface module configured to implement a large language model to have a dialog with an occupant of the host vehicle using a modality, and an assisted driving module configured to operate in an assisted driving mode and, while in the assisted driving mode, perform a precondition check to implement at least one of: i) the adaptive alert module to operate in an adaptive alert mode and provide the alert via a human-machine interface (HMI) of the host vehicle, and ii) the multi-modal dialog interface module to perform a multi-modal dialog with the occupant via the HMI.

[0005] In other features, the assisted driving mode includes the assisted driving module controlling steering, acceleration, and deceleration of the host vehicle.

[0006] In other features, the assisted driving system further includes a contextual active learning module configured to fine-tune a large language model of the multi-modal dialog interface module.

[0007] In other features, the adaptive alert module is configured to adaptively vary the duration and intensity of the alert provided to the occupant based on a plurality of parameters.

[0008] In other features, the adaptive alert module is configured to adaptively change a duration and intensity of an alert based on a level of inattention and a level of imminent threat of the occupant.

[0009] In other features, the adaptive alert module is configured to adaptively change a duration and intensity of an alert based on a detected posture and gaze of the occupant.

[0010] In other features, the multi-modal dialog interface module is configured to perform at least one of the following: alert and provide a message to the occupant using the modality. The modality includes an audio device, a display, a haptic device, and one or more lights.

[0011] In other features, the multi-modal dialog interface module is configured to control at least one of an ambient lighting, interior lighting, and an audio system to direct a point of focus of the occupant to a point or area of interest.

[0012] In other features, the point or area of interest refers to a display or an area in front of the host vehicle.

[0013] In other features, the multi-modal dialog interface module is configured to control a light bar in an interior of the host vehicle to direct the occupant to the point or area of interest.

[0014] In other features, the multi-modal dialog interface module is configured to redirect the point of focus of the occupant to a center of a road in front of the host vehicle; and re-engage in the assisted driving.

[0015] In other features, the multi-modal dialog interface module is configured to display a spatial cluster related to decision making on a display.

[0016] In other features, the multi-modal dialog interface module is configured to generate reactive and proactive messages for the occupant, the reactive and proactive messages being context-based and domain-specific.

[0017] In other features, the multi-modal dialog interface module is configured to answer questions from the occupant related to the assisted driving mode.

[0018] In other features, the assisted driving system further includes a context-in-the-loop learning module configured to update a personal profile of the occupant. The adaptive alert module is configured to adaptively generate an alert based on the personal profile.

[0019] In other features, the multi-modal dialog interface module is configured to: i) receive explicit feedback and detect implicit feedback from the occupant regarding the alert; and ii) generate a reward based on the explicit feedback and the implicit feedback. The adaptive alert module is configured to change an alert strategy for the occupant based on the reward.

[0020] In other features, an assisted driving method for a host vehicle is disclosed. The method includes operating in an assisted driving mode; while operating in the assisted driving mode, performing a pre-condition check to enable an adaptive alert mode and to enable a multi-modal dialog with an occupant of the host vehicle. During the adaptive alert mode, an alert is adaptively generated with variable duration and intensity and the alert is provided via a human-machine interface (HMI) of the host vehicle. During the multi-modal dialog, a large language model is implemented to have a dialog with the occupant using a modality via the HMI.

[0021] In other features, the assisted driving method further includes, while operating in the assisted driving mode, controlling steering, acceleration, and deceleration of the host vehicle via a control module of the host vehicle.

[0022] In other features, the assisted driving method further includes fine-tuning the large language model based on a preference of the occupant; and changing an alert strategy based on the fine-tuned large language model and a plurality of parameters, including changing a duration and intensity of one or more of the alerts for the occupant.

[0023] In other features, the assisted driving method further includes selectively using the modality to provide the alert based on a personal profile of the occupant.

[0024] Further areas of applicability of the present disclosure will become apparent from the detailed description, claims and drawings. The detailed description and specific examples intended for purposes of illustration only and are not intended to limit the scope of the present disclosure. BRIEF DESCRIPTION OF DRAWINGS

[0025] The present disclosure will become more fully understood from the detailed description and the accompanying drawings, wherein: Figure 1 is a functional block diagram of an example host vehicle including an example driver automation system (DAS) with an assisted driving module in accordance with the present disclosure; Figure 2 is a functional block diagram of an example communication system of a host vehicle in accordance with the present disclosure; Figure 3 is a functional block diagram of an example adaptive driving automation system in accordance with the present disclosure; Figure 4A and 4B (collectively referred to as Figure 4) illustrates an example precondition checking method according to this disclosure; Figure 5 An example driving posture and gaze detection and tracking method for adaptive alarms and multimodal dialogue according to this disclosure is illustrated; Figure 6 The illustration shows an example of an adaptive alarm and multimodal dialogue interface connection method according to this disclosure; Figure 7 This is an example drawing illustrating the duration and intensity of an alert based on inattentiveness and an imminent potential threat, according to this disclosure. Figure 8 This is a perspective view of an example interior of a vehicle that provides directional alarms via an internal lighting system, according to this disclosure; Figure 9 This is a functional block diagram of an example multimodal dialogue interface module for active learning in a domain-specific LLM execution context of freezing and fine-tuning, based on this disclosure; and Figure 10 This is a functional block diagram of the active learning module in the example scenario of this disclosure.

[0026] In the accompanying drawings, reference numerals may be reused to identify similar and / or identical elements. Detailed Implementation

[0027] Hands-free driving systems allow the driver to control the steering, braking, and acceleration of a vehicle when engaged. The driver or passenger can disengage from hands-free (sometimes referred to as assisted or automated) driving by tapping, for example, the brake or accelerator pedal. Traditional hands-free driving systems have limited triggers for engaging and disengaging from hands-free driving, limited or no interaction with the vehicle passenger, and limited functionality.

[0028] Examples disclosed herein include driving automation systems with assisted and / or automated driving, which include adaptive alerts, multimodal dialogue interface connectivity, and context-aware active learning. The examples apply to at least Level 2, Level 2.5, and Level 3 automation. The examples apply to assisted and automated driving with human supervision and intervention to take over control and disengage from assisted and automated driving. Examples include adaptive levels of disengagement, which include adaptive duration and intensity of alerts. For example, the duration and intensity of the alert may be based on how far the host vehicle is turned away from the center of the current driving lane. As another example, the duration and intensity of the alert may be based on how attentive the vehicle occupant is (e.g., is the vehicle occupant slightly turning his or her head, or is the vehicle occupant turning without noticing the road ahead). The source location or modality of the alert may be adapted based on driver temperament or secondary task type. Alerts may be provided to instruct the vehicle occupant to focus on one or more conditions and / or take over driving control.

[0029] Multimodal dialogue interface connectivity includes visual connectivity with a vehicle occupant via one or more displays and / or lights, audible connectivity via speakers and / or other audible devices, and tactile connectivity via haptic devices. Interface connectivity is domain-specific, meaning that the interface connectivity (or communication) with the occupant is related to driving and / or vehicle operation. Domain-specific interface connectivity includes reactive and proactive interface connectivity and may include the use of multimodal dialogue agents and large language models (LLMs). In embodiments, context-aware proactive learning is performed to train the LLM and enhance and improve multimodal interface connectivity, which includes the use of multimodal dialogue human-machine interfaces (HMIs), as further described below. These and other adaptive alarms, multimodal dialogue interface connectivity, and context-aware proactive learning are further described below.

[0030] When driving in open spaces, drivers may want to relax and enjoy the nature and scenery. For this reason, drivers may engage in hands-free driving, as associated with Level 2.5 automation. This type of automated driving only operates on mapped roads and requires the driver to pay close attention to the road, except for a maximum 10-second pause. The examples disclosed herein implement adaptive alerts, a multimodal dialogue HMI, and context-aware active learning after checking multiple preconditions and taking into account driving posture.

[0031] Figure 1A host vehicle 100 is shown, including a Driver Automation System (DAS) 101 with a Driver Assistance Module 102. The Driver Assistance Module 102 includes an Adaptive Alarm Module 103, a Multimodal Dialogue Interface Module 104, and a Context-Based Active Learning Module 105. Although shown as separate modules, two or more of modules 102, 103, 104, and 105 can be combined and implemented as a single module. The Adaptive Alarm Module 103 performs adaptive alarms as described herein. The Multimodal Dialogue Interface Module 104 provides a responsive and proactive interface connection with the vehicle occupants as described herein. The Context-Based Active Learning Module 105 performs context-based active learning as described herein.

[0032] exist Figure 1 Part of DAS101 is shown in the figure, and in Figure 2 , 3 Additional details of the DAS are shown in Figures 9-10. The host vehicle 100 includes a vehicle control module 111, which, as shown, includes a driver assistance module 102. The driver assistance module 102 performs: perception (or situation) determination operations; object detection, recognition, classification, and graphic and visual recognition operations; data retrieval, acquisition, and collection operations; interactive timing operations; driver assistance operations; image overlay operations; dialogue operations, including providing voice, text, and / or haptic messages; and so on. The vehicle control module 111 can perform various operations based on interactions with the user and messages generated as further described below.

[0033] The main vehicle 100 further includes one or more power supplies 109, a telematics module 106, an infotainment module 107, other control modules 108, and a propulsion system 110. The vehicle control module 111 can control the operation of the vehicle 100, including the propulsion system 110. The power supply 109 may include one or more battery packs, a generator, a converter, control circuitry, terminals for high and low voltage loads, and one or more battery sensors 112 for detecting the state of the power supply 109, including voltage, current level, and charging status.

[0034] The telematics module 106 provides wireless communication services within the host vehicle 100 and communicates wirelessly with service providers, network devices, other vehicles, mobile devices, infrastructure equipment, and other devices external to and / or internal to the host vehicle 100. The telematics module 106 can support... Bluetooth, Bluetooth Low Energy (BLE), Ultra Wideband (UWB), Near Field Communication (NFC), Cellular, Legacy (LG) Transmission Control Protocol (TCP), Long Term Evolution (LTE) and / or other wireless communications, and / or according to The transceiver module 106 may operate using BLE, UWB, NFC, cellular, and / or other wireless communication protocols. The telematics module 106 may include one or more transceivers 113 and a navigation module 114, the navigation module 114 having a Global Positioning System (GPS) and GNSS (or Global Navigation Satellite System) receiver 116. The navigation module 114 may include an inertial measurement unit (IMU) 117 and an odometer / wheel sensor 119. The transceiver 113 wirelessly communicates with network devices inside and outside the host vehicle 100, including cloud-based network devices, central stations, back-office environments, and portable network devices. The transceiver 113 may perform pattern recognition, channel addressing, channel access control, and filtering operations.

[0035] Navigation module 114 executes a navigation application to provide navigation services. The navigation services may include location identification services to identify the location of the host vehicle 100. The navigation services may also include guiding the driver and / or directing the host vehicle 100 to a selected location. Navigation module 114 may communicate with a central station to collect map information indicating traffic levels, transport object identification and location (e.g., the location and type of signs), route information, the location of rest areas, gas stations, restaurants, etc. As an example, if the host vehicle 100 is an assisted and / or automated driving vehicle, navigation module 114 may guide vehicle control module 111 along a selected route to a selected destination. GPS and GNSS receiver 116 can provide vehicle speed and / or direction (or heading) and / or global clock timing information for the host vehicle 100 and other vehicles and objects (e.g., pedestrians and cyclists).

[0036] The infotainment module 107 may include and / or be connected to an audio system 122 and / or a video system, the video system including one or more displays (a display 120 is shown). Display 120 and audio system 122 may be part of a human-machine interface. Display 120 may include a cluster and / or center console display, head-up display, etc. In addition to the display and audio system 122, haptic devices 124 (e.g., steering wheel and / or seat vibration devices) may be used to interact with vehicle occupants such as drivers or passengers. This interaction is further described below. Messages may be displayed, audibly played, and / or indicated via display 120, audio system 122, haptic devices 124, and / or via one or more other output devices.

[0037] The infotainment module 107 can provide various informative, warning, and proactive messages including information relating to: upcoming and currently performed operations (e.g., braking, acceleration, turning); detected objects (or obstacles); upcoming and / or nearby gas stations; upcoming and / or nearby restaurants; music services; upcoming and / or nearby shops; vehicle status information; diagnostic information; prognostic information; entertainment features, etc. The infotainment module 107 can be used to guide the vehicle operator to a location, indicate travel estimates (e.g., distance to a selected destination), and other information.

[0038] The propulsion system 110 may include one or more torque sources, such as one or more electric motors and / or one or more engines (e.g., internal combustion engines). Figure 1 In the example shown, the main vehicle 100 includes an engine 130 and one or more electric motors 132. The torque sources are independently controlled. The propulsion system 110 includes an electric motor control system 134, which includes one or more electric motors 132 and an electric motor control module 136, which can control the operation of the one or more electric motors 132 based on signals from the vehicle control module 111.

[0039] Modules 103, 104, 107, 108, and 111 can communicate with each other via one or more buses 140, such as a Controller Area Network (CAN) bus and / or other suitable interfaces. Vehicle control module 111 can control the operation of vehicle modules, equipment, and systems based on feedback from sensors 150.

[0040] Sensor 150 may include external sensor 152, internal sensor 154, and other sensors 156. External sensor 152 may include radar and / or lidar sensor 158 and imaging and audio devices (e.g., a visual spectrum camera, a long-wave infrared camera, a short-wave infrared camera, an ambient light sensor, and a microphone or microphone array) 160. External sensor 152 may be used to detect objects outside the host vehicle 100 and / or in the path of the host vehicle 100.

[0041] Internal sensor 154 may include internal imaging sensors (e.g., a camera) 162, a microphone or microphone array 164, one or more imaging radar sensors 165, and one or more laser scanning sensors 167. Internal sensor 154 may be part of a driver monitoring system (DMS). Internal sensor 154 may be used to monitor vehicle occupants to detect and track head position and / or eyes. The position and movement of the vehicle occupant's head and eyes may be tracked. As an example, internal sensor 154 may track the driver's posture, arm and hand positions, and eyes. This includes: determining eye position and gaze direction, detecting gestures made by the driver, detecting the driver's body orientation, detecting the driver's voice, etc. The monitoring may be used to determine which directions the driver is looking in, what object the driver is looking at (one or more), etc.

[0042] Other sensors 156 may include a vehicle speed sensor 166 as shown, an acceleration sensor (e.g., longitudinal and lateral acceleration sensors) 168, and an oil level sensor 170, as well as other sensors such as an inclinometer, an engine temperature sensor, and an engine oil pressure sensor. Additional sensors may also be included, such as braking system sensors (brake sensor 179 is shown) and steering system sensors (steering angle sensor 181 is shown).

[0043] The driver assistance module 102 can use machine learning for: object classification, including the identification and / or classification of pedestrians, cyclists, and vehicles (e.g., oncoming traffic); and the determination of the probable trajectory for each detected, identified, and / or classified object. The driver assistance module 102 can determine the position of objects based on feedback from sensor 150. The driver assistance module 102 can also detect the driver's (or passenger's) head and eye position and gaze angle, which can be used to calculate and continuously adjust the position of a projected conformal graphic covering the roadway in real time. The graphic includes digital gateways, object identifiers, and other displayed information.

[0044] The vehicle control module 111 may further include a mode selection module 172 and a parameter adjustment module 174. The mode selection module 172 can select a vehicle operation mode. The parameter adjustment module 174 can be used to adjust the parameters of the host vehicle 100. The vehicle control module 111 can perform autonomous operation based on interaction with the vehicle occupant. As an example, the vehicle control module 111 can operate in a fully autonomous mode or a partially autonomous mode, and can control the propulsion system 110, braking system 176, and steering system 178. In an embodiment, the vehicle control module 111 controls the operation of system 110, 176, and 178 based on interaction with the vehicle occupant. The vehicle control module 111 can: i) perform autonomous operations, such as steering, braking, acceleration, etc.; and / or ii) display and / or audibly play messages, perform tactile operations via tactile device 124, and / or output messages and / or corresponding signals via other output devices.

[0045] In this embodiment, the DAS101 uses computer vision, machine learning, and cloud computing to identify, transmit, and evaluate scenarios in which a mobile host vehicle should yield to one or more pedestrians, obstructed carriageways, and / or oncoming (right-of-way) traffic. The DAS101 visualizes and considers pedestrians, carriageway obstacles, and oncoming traffic in real time, and performs actions to provide enhanced situational awareness to vehicle occupants. The DAS101 provides situational awareness in automated driving modes to increase user trust and assist in vehicle takeover.

[0046] In an embodiment, a vehicle occupant (e.g., driver or passenger) can manually override operations performed by the DAS101. This can include, for example, steering, braking, and / or acceleration operations being performed. This can be accomplished, for example, by gently tapping on the brake or accelerator to partially or completely disengage the DAS101. The DAS101 is configured to perceive the road ahead and surrounding area based on the output of sensors (e.g., cameras, radar sensors, and / or lidar sensors) and vehicle-to-everything (V2X) communication, which includes vehicle-to-vehicle communication, vehicle-to-mobile device communication, vehicle-to-infrastructure communication, and other communications (e.g., vehicle-to-distributed network communication).

[0047] The host vehicle 100 may further include a memory 180. The memory 180 may store sensor data 182, parameters 184, applications 186, algorithms 188, historical data 190, domain-specific LLMs 191, external inputs 191 from other devices outside the host vehicle 100, and other data 192. Parameters may include sensor parameters such as vehicle speed, vehicle acceleration, battery charge status, fuel level, etc. Applications 186 may include applications executed by modules 102, 103, 104, 105, 107, 108, and 111.

[0048] Although the memory 180 and the vehicle control module 111 are shown as separate devices, they can be implemented as a single device. The memory 180 may also store historical data 190 and other data 192, such as driver driving modes, driver fuel supply modes, driver stop modes, driver pick-up modes, other driver modes, data acquired and / or collected by at least one of modules 102, 111, traffic data, navigation data, map data, GPS data, route data, speed data, and acceleration data, etc.

[0049] The vehicle control module 111 can control the operation of the propulsion system 110, the video system including the display 120, the audio system 122, the haptic device 124, the braking system 176, the steering system 178, the heating, ventilation and air conditioning (HVAC) system 193, the lighting system 194, the seat system 196, the mirror system 198, and / or other equipment and systems, based on parameters set by modules 102, 107, 108, 111, and 174. The vehicle control module 111 can set at least some of the parameters based on signals received from sensor 150. The lighting system 194 can include various interior lights, a series of light-emitting diodes (LEDs), light strips, etc. The lighting system 194 can also include and / or control the amount of ambient light entering the vehicle by adjusting the tint level of windows, opening and / or closing one or more blackout curtains, etc.

[0050] The vehicle control module 111 can receive power from the power supply 109, which can be supplied to the propulsion system 110, braking system 176, steering system 178, HVAC system 193, lighting system 194, seat system 196, mirror system 198, etc. The power supplied to the haptic device 124, electric motor 132, braking system 176, steering system 178, HVAC system 193, lighting system 194, seat system 196, mirror system 198, and / or their actuators can be controlled by the vehicle control module 111 to adjust, for example: electric motor speed, torque, and / or acceleration; braking pressure; steering wheel angle; pedal position; the state of the haptic device 124; etc. This control can be based on the outputs of sensors 150, navigation module 114, GPS and GNSS receiver 116, data and information received from external devices, and data and information stored in memory 180.

[0051] The vehicle control module 111 can determine various parameters, including vehicle speed, motor speed, gear status, accelerator position, brake pedal position, amount of regenerative (charging) power, amount of automatic start / stop discharge power, and / or other information. The vehicle control module 111 can control the operation of system 110, 176, and 178 based on the declared parameters. The driver assistance module 102 can display vehicle status information based on the declared parameters.

[0052] The driver assistance module 102 monitors the real-time behavior of vehicle occupants, including voice, gaze patterns, head position, and gestures (such as hand gestures, finger gestures, facial expressions, etc.). Gazing patterns include the direction of the occupant's head and eyes. This information is used to determine the location, status, duration, timing, and intensity of alarms and other conversational messages.

[0053] The host vehicle 100 may include various systems for assisting the driver, for performing autonomous operations, and / or for instructing vehicle occupants on information related to the environment of the host vehicle. For example, the host system may include a navigation system that provides map information indicating lane boundaries, street locations, speed limits, the geographic location of the selected destination, etc. The host system may provide the driver with instructions to drive to the selected destination, and / or may perform autonomous operations such as braking, steering, and acceleration to drive the vehicle to the destination based on map information.

[0054] As another example, the host vehicle 100 may include an object detection and collision warning system for detecting approaching objects and executing countermeasures and / or taking evasive actions to prevent a collision. The vehicle control module 111 determines the position of the object relative to the host vehicle 100 and the trajectories of the object and the host vehicle 100. If it is determined that the host vehicle 100 is likely to collide with one of the objects, one or more warning signals may be generated to indicate a potential collision concern to the driver and / or the object. These warnings may be provided in addition to the digital gateway and other information described herein. The vehicle control module 111 may also, or alternatively, execute one or more other countermeasures (e.g., applying braking to decelerate the host vehicle, changing the steering angle of the host vehicle, etc.) to prevent a collision.

[0055] Figure 2 A communication system 200 is shown, including a mainframe vehicle 100, a distributed communication system 202, a cloud-based network device 204, and a back-office 206. Figure 1 Part 210 of DAS101. Part 210 includes a vehicle control module 111, a telematics module 106, a memory 180, and an HMI 212, which may include any interface connection device mentioned herein, including a display, speaker, haptic device, light, etc. The driver assistance module 102 may utilize the windshield as a display and provide augmented reality via an augmented reality head-up display (ARHUD). The vehicle control module 111 includes the driver assistance module 102.

[0056] The memory 180 includes external input 191 and in-vehicle input 220. External input 191 may include: GPS information; information received via the Internet; and / or information received via V2X communication, WiFi communication, cellular communication, and / or satellite communication. External input 191 may include information received from cloud-based network device 204 and / or back-office 206. In-vehicle input 220 may include the posture of the vehicle occupant, arm, hand, and head positions, eye positions and gaze angles, automated driving system status information, vehicle braking information, steering angle information, object detection information, vehicle acceleration information, etc. This information may be provided via the sensors mentioned herein.

[0057] Cloud-based network device 204 may include control module 230, transceiver 232, and database 234. One or more cloud-based network devices may be included, including one or more edge devices. Back-office office 206 may include control module 240, transceiver 242, and database 244.

[0058] Devices 204 and 206 can create, store, and modify passenger preference profiles, each of which is specific to a particular person (driver and / or vehicle passenger). Profiles may include preferences regarding alarm messages, when and what type of alarm message is provided, whether the person prefers reactive messages, whether the person prefers proactive messages, etc. For example, a profile may indicate whether a person has hearing loss, limited sensation in their legs or hips, etc., so that certain types of alarms should not be provided. A profile may indicate whether the driver typically looks at a particular display and whether an alarm is provided on that display. Profiles can be accessed for the driver using facial recognition, fingerprint recognition, voice recognition, username, password, etc., based on the driver's unique identifier (ID). Profiles are shared with the vehicle and can be accessed by the vehicle (e.g., by...). Figure 1 The driver can create, store, and / or modify preferences using one of the driver assistance modules 102 and / or 103, 104, and 105. Preferences can be used and updated during each driving cycle. Preferences can also be shared with multiple vehicles. The driver can switch between vehicles, and the driver's preferences can be provided to and used by each of the vehicles.

[0059] The driver assistance module 102 can utilize computer vision, machine learning, V2X communication, and cloud computing to highlight objects within a scene to which the host vehicle 100 should yield, including one or more pedestrians, obstructed carriageway objects, oncoming (right-of-way) traffic, and / or other objects. Machine learning algorithms can be used to classify the type and trajectory of objects within the planned path of the host vehicle 100. The driver assistance module 102 can provide a graphical user interface to communicate the presence and intent of detected objects (i.e., the predicted path). The driver assistance module 102 can track the position of the host vehicle 100 and other objects, the speed of the host vehicle 100 and other objects, the trajectory of the host vehicle 100 and other objects, carriageway curvature, vehicle steering, etc., and display images via HMI 212 based on these positions, speeds, trajectories, and carriageway curvatures. The driver assistance module 102 can also display images with selected type, size, shape, and color based on claimed information and occupant position, host vehicle position, and other vehicle-sensed aspects of the driving environment.

[0060] Figure 3 An adaptive driving automation system 300 is shown, which includes an adaptive alert module 103, a multimodal dialogue interface module 104, a context-aware active learning module 105, and, for example, an assistance (e.g., hands-free) driving system 302. The assistance driving system 302 may include steering, braking, and propulsion systems for hands-free driving, such as... Figure 1As shown in Figure 4, modules 103 and 104 receive parameters and / or data associated with precondition checks, and / or perform precondition checks, such as checking time and speed limits, distance to other vehicles, and visibility conditions, designated as 304. Some additional examples of precondition checks are mentioned with reference to Figure 4. Modules 103 and 104 may also receive parameters and / or data associated with driving posture and gaze detection and tracking, and / or perform driving posture and gaze detection and tracking, designated as 306. Posture and gaze detection and tracking can be implemented via cameras 310 and 312 of the DMS 314. Cameras 310 and 312 can be mounted in various locations, such as on the B-pillar, roof, steering wheel, A-pillar, etc.

[0061] The context-aware active learning module 105 can monitor the driver's posture and gaze, designated as 320, and fine-tune the LLM implemented by the multimodal dialogue interface module 104 based on a personalized cloud profile 322. A fine-tuning signal 324 can be sent from the context-aware active learning module 105 to the multimodal dialogue interface module 104. The context-aware active learning module 105 can receive assisted and automated driving system tracking information from the assisted driving system, and can generate fine-tuning signals based on this tracking information. The assisted and automated driving system tracking information may include the state of system 302, including target speed, steering angle, acceleration, deceleration, etc. Context-aware active learning is domain-specific.

[0062] Context-aware proactive learning can include monitoring driver responses, facial expressions, and movements, which can occur in response to reactive responses and proactive messages provided by the multimodal dialogue interface module 104. The context-aware proactive learning module 105 can adjust when to provide alerts, how often to provide alerts, the duration of the provided alerts, and the type of alerts based on context-aware proactive learning. The adaptive alert module 103 adjusts the duration, intensity, and location of alerts, as well as the device(s) providing the alert(s), based on the posture and gaze of the vehicle occupant.

[0063] Figure 4 illustrates the precondition checking method. The following operations can be performed iteratively. Although some example precondition checks are mentioned below, other precondition checks can be performed.

[0064] At 400, the driver assistance module 202 determines whether lane markings and lane boundaries are visible. If so, operation 402 is executed; otherwise, operation 426 can be executed.

[0065] At 402, the driver assistance module 202 determines whether the speed of the host vehicle is within a predetermined range (i.e., between a first threshold EPS1 and a second threshold EPS2). Thresholds EPS1 and EPS2 can be calibrable speed thresholds provided as indicated by arrow 403. If yes, operation 404 is performed; otherwise, operation 426 is performed.

[0066] At 404, the driver assistance module 202 determines whether the vehicle speed is greater than a third threshold EPS3. The third threshold EPS can be calibrated and can be provided as specified by arrow 405. If yes, operation 406 is performed; otherwise, operation 426 is performed.

[0067] At 406, the driver assistance module 202 determines whether the distance(s) between the host vehicle and(s) or more other vehicles(s) in the oncoming lane is less than a fourth threshold EPS4. The threshold EPS4 may be calibrable and may be provided as specified in 407. The distance(s) may be determined based on traffic data, represented as 408. The traffic data may be received from a cloud-based network device and may include front camera data and long-range radar data. If yes, operation 408 is performed; otherwise, operation 426 is performed.

[0068] At 410, the driver assistance module 202 determines whether the distance between the host vehicle and one or more other vehicles and / or one or more obstacles in the same driving direction is less than a fifth threshold EPS5. The distances may be determined based on traffic data 408. If yes, operation 412 is performed; otherwise, operation 426 is performed.

[0069] At 412, the driver assistance module 202 determines whether the road curvature is less than a sixth threshold EPS6 for the next predetermined travel distance (e.g., 200 meters). The threshold EPS6 may be calibrable and may be provided as specified in 413. This determination may be based on map data, as specified in 414. If yes, operation 414 is performed; otherwise, operation 426 is performed.

[0070] At 416, the driver assistance module 202 determines whether a lane change event or lane merging event occurs for the host vehicle. If so, operation 418 is executed; otherwise, operation 426 is executed.

[0071] At 418, the driver assistance module 202 determines whether the host vehicle is in a busy traffic area and / or an accident-prone area. This may be based on received police data, as specified in 419. If so, operation 420 is performed; otherwise, operation 426 is performed.

[0072] At 420, the driver assistance module 202 determines whether low visibility, severe weather, and / or low light conditions exist. This can be based on data from sensors such as a sunlight sensor, a light detection sensor, etc. It can also be based on contextual time constraints of characteristic activity functions, as indicated by box 423 and arrow 422. Characteristic activity functions can be based on the driver's age (indicated by arrow 421), for example, whether the driver is a teenager or an older driver. The driver's age can be determined based on the driver's profile and / or data from the DMS. The DMS can be used to determine the driver's state, including whether the driver is paying attention to the road ahead, is drowsy, or is suffering from a headache, etc. This can be based on posture, head and eye position, gaze angle, facial expressions, etc. These are inputs that can be used to determine and modify alert strategies and for dialogue messages to the driver, as further described herein. If low visibility, severe weather, and / or low light conditions exist, then operation 424 can be performed.

[0073] At 424, the driver assistance module 202 operates in an adaptive alert mode and provides one or more adaptive alerts based on prerequisite checks and other collected and determined information, some of which has been mentioned above. The driver assistance module 202 operates in the adaptive alert mode and provides adaptive alerts when operating in an assisted driving mode (e.g., hands-free assisted driving mode).

[0074] At 426, if the current operation is in adaptive alarm mode, the driver assistance module 202 can terminate operation in adaptive alarm mode and / or operate in driver assistance mode without providing adaptive alarms.

[0075] Figure 5 A method for driver pose and gaze detection and tracking for adaptive alarms and multimodal dialogue is shown. The following operations can be performed iteratively.

[0076] At 500, Figure 1 The driver assistance module 102 can receive data from the DMS and other internal sensors (e.g., internal cameras and / or imaging radar sensors). At 502, the driver assistance module 102 can determine the posture of the driver's (or vehicle occupant's) arms and the position of the driver's hands based on the received data.

[0077] At 504, the driver assistance module 102 can determine the head posture (or the position and orientation of the driver's head) relative to the road ahead (or the centerline of the lane in which the driver's main vehicle is driving) based on the received data.

[0078] At 506, the driver assistance module 102 can determine eye gaze, including eye position and angle of gaze, based on the received data.

[0079] At point 508, the driver assistance module 102 sends information about arm posture, hand position, head posture, and eye gaze to the adaptive alert module 103. This information allows the adaptive alert module 103 to determine how well the driver is paying attention, whether the driver is shifting from left to right, whether the driver's eyes are moving around, where the driver's eyes are focused, and how long the driver is looking in each direction. As described herein, the adaptive alert module 103 adaptively determines and implements alert strategies based on this information, including determining and adjusting the duration, intensity, and type of alerts (or messages).

[0080] At 510, the multimodal dialogue interface module 104 receives at least eye gaze information and determines the type of alarm message to be generated based on the information, including visual, audible and / or tactile alarm messages.

[0081] Figure 6 The method for connecting adaptive alarms and a multimodal dialogue interface is illustrated. The following operations can be performed iteratively.

[0082] At 600, the driver assistance module 102 and / or the adaptive alert module 103 determine whether the prerequisite checks, such as those mentioned herein, are met. If so, operation 608 can be performed; otherwise, operation 612 can be performed.

[0083] At 602, the driver assistance module 102 can detect and track the occupant's posture and gaze, as described herein.

[0084] At position 604, the driver assistance module 102 can obtain the threshold EPS6.

[0085] At position 606, Figure 1 In this context, the active learning module 105 can perform active guidance on the driver's (or passenger's) response time based on the threshold EPS6 and other monitored data.

[0086] At 608, the adaptive alert module 103 customizes the timing and severity of the adaptive alert, including the duration and intensity level of the alert. This can be based on detected and tracked occupant posture and gaze information, as well as response time. The alert can be non-linear in terms of duration and / or intensity, and can be based on how much attention the driver is paying (e.g., how much the driver is looking away from the road ahead of the host vehicle).

[0087] At point 610, the adaptive alert module 103 can use directional ambient light and sound to redirect the driver's (or passenger's) attention to a specific point or area. This point or area could be, for example, on a display screen, on the windshield, or on the road ahead. This can be accomplished using one or more light strips, smart glass, ambient lighting, a center console screen, etc., to provide a multimodal alert. The alert can be provided without the driver requesting information.

[0088] At 612, the driver assistance module 102 can terminate the adaptive alarm mode and / or operate in the driver assistance mode without providing adaptive alarms.

[0089] At 614, when operating in assisted driving mode, assisted driving module 102 can use directional ambient light and directional sound to transmit decision-making and threat direction.

[0090] At 616, when operating in assisted driving mode, the assisted driving module 102 can use enhanced vision in the instrument cluster and / or center console to convey decision-making when the user looks at the instrument cluster or center console.

[0091] At point 618, when operating in assisted driving mode, the multimodal dialogue interface module 104 can indicate why the assisted driving mode is disabled. This can be accomplished via an artificial intelligence (AI) dialogue agent of the assisted driving module 102 to indicate why the system has disengaged from assisted driving mode. The assisted driving module 102 can voluntarily provide information related to situations that the driver is unaware of or does not understand. For example, the assisted driving module 102 may disengage from assisted driving mode if the driver does not understand or know why this is happening, and the assisted driving module 102 can indicate the reason. For example, one or more of the precondition checks may no longer be met. The AI ​​dialogue agent can be reactive and / or proactive in providing information to the driver. The AI ​​dialogue agent can also interact with and maintain a dialogue with the driver, where the AI ​​dialogue agent sends messages to the driver and receives responses from the driver.

[0092] Figure 7A plot illustrating the duration and intensity of alerts based on inattentiveness and imminent threat is shown. Although nine zones are shown, any number of zones may be included. Each zone has a corresponding level of inattentiveness for the driver (or passenger) and a level of imminent threat (e.g., the time until the host vehicle may collide with an approaching object). The level of inattentiveness may be based on and / or a function of driver posture, as well as time and speed limits, and varies from low to high. The level of imminent threat may be based on visibility conditions and distance to other nearby vehicles, and also varies from low to high. IL1-IL3 refer to the level of inattentiveness. IPT1-IPT3 refer to the level of imminent threat. IL1 and IPT1 are the lowest levels, and IL3 and IPT3 are the highest levels.

[0093] Each of these zones has a unique characteristic compared to those formed by... Figure 1 The adaptive alarm module 103 provides alarms associated with corresponding alarm durations and intensity levels. A first zone with IL1 and IPT1 can be associated with long-duration, low-intensity alarms (e.g., low-brightness and / or low-sound-level alarms and / or small-sized alarms). A second zone with IL1 and IPT2 can be associated with short-duration, medium-intensity alarms. A third zone with IL1 and IPT3 can be associated with short-duration, high-intensity alarms. A fourth zone with IL2 and IPT1 can be associated with longer-duration, low-intensity alarms. A fifth zone with IL2 and IPT2 can be associated with medium-duration, medium-intensity alarms. A sixth zone with IL2 and IPT3 can be associated with short-duration, high-intensity alarms. A seventh zone with IL3 and IPT1 can be associated with longer-duration, medium-intensity alarms. An eighth zone with IL3 and IPT2 can be associated with longer-duration, medium-intensity alarms. A ninth zone with IL3 and IPT3 can be associated with short-duration, very high-intensity alarms.

[0094] Zones one, two, and three can be associated with drivers looking forward, and the vehicle's speed is less than, for example, 40 km / h (hr). Zones four, five, and six can be associated with drivers looking to the side, and the vehicle's speed is greater than 40 km / h but less than, for example, 80 km / h. Zones seven, eight, and nine can be associated with drivers looking at the floor or at a mobile phone, where the vehicle's speed is greater than, for example, 100 km / h.

[0095] Zones 1, 4, and 7 can be associated with good visibility and no vehicles nearby the main vehicle. Zones 2, 5, and 8 can be associated with 80% visibility and two vehicles nearby and behind the main vehicle. Zones 3, 6, and 9 can be associated with poor visibility and more than six vehicles nearby and / or in front of and behind the main vehicle.

[0096] Adaptive alerts are provided based on the zone and associated conditions currently being traversed by the host vehicle. This includes the timing (or when) at which the alert is triggered and for how long it is provided. Alerts are provided to address potential threats and compensate for a lack of driver attention to the road. Multimodal interaction with the driver can occur and may include graphic alerts, audio alerts, haptic alerts, lighting-based alerts, voice alerts, etc. One or more modalities may be used to provide each alert. In this embodiment, multiple modalities are used to provide a single alert. In this embodiment, the more intense the alert, the more modalities are used.

[0097] Figure 8 An interior 700 of a host vehicle is shown, providing directional alarms via an interior lighting system. An example light bar 702 is shown and can be used to provide directional lighting to guide the driver to a location. This can be done to direct the driver's attention to an area outside the vehicle and / or to the alarm message being provided. The light bar 702 may include multiple LEDs, for example, illuminated in a sequential mode. As an example, the light bar may be illuminated such that the LED furthest from the area of ​​interest is illuminated first and the LED closest to the area of ​​interest is illuminated last. In this way, the light bar 702 is animated. As an example of the direction of illumination of the LEDs of the light bar, arrows shown near the light bar can guide the driver to the road ahead. The lights may be flashing, illuminated in various patterns, etc. The interior 700 may include other lights, which may be illuminated in various patterns along with the light bar 702. The interior lights can be used to direct the driver's attention to one of the interior displays, the road ahead, or another point of interest and / or area. In embodiments, ambient lighting is also adjusted or alternatively adjusted. This could include adjusting the window's color tone level, closing one or more blackout curtains, etc.

[0098] The interior 700 includes a steering wheel 704, an instrument cluster display 706, and a center console display 708. Displays 706 and 708 can be used to display warnings. The steering wheel 704 may also include illumination that can flash to draw the driver's attention to the instrument cluster display 706.

[0099] Figure 1The multimodal dialogue interface module 104 can have an interactive dialogue with the driver (or passenger) to perform reactive assistance and guidance. Table 1 includes example questions that the driver can ask the dialogue (or AI) agent of the multimodal dialogue interface module 104. The agent can then respond using the answers to the driver's questions. Requests and responses can be provided verbally and / or via one or more modalities. The agent can implement a “freeze” and / or fine-tuned LLM for providing responses. The LLM is domain-specific to provide relevant answers and feedback based on context. The agent can provide responses based on the automatic extraction and / or determination of contextual information, such as precondition checks, driver posture detection and tracking, and / or other driver state information. This information and other information mentioned herein can be stored in memory and / or accessed from one or more network devices detached from the host vehicle.

[0100] Table 1.

[0101] The multimodal dialogue interface module 104 also implements proactive assistance and guidance. Table 2 includes examples of proactive messages provided to the driver.

[0102] Table 2.

[0103] Figure 9 and 10 A multimodal dialogue interface module 104 and a context-based active learning module 105 are illustrated for performing context-based active learning for a domain-specific LLM 900 (such as the LLMs mentioned herein) that is frozen and / or fine-tuned. The context-based active learning module 105 may be implemented as part of the multimodal dialogue interface module 104. The multimodal dialogue interface module 104 may also include a dialogue agent (or AI) agent 902, such as any of the dialogue and / or AI agents mentioned herein. The multimodal dialogue interface module 104 performs context-based active learning based on feedback from the driver, the driver's inaction on messages provided by the agent 902, etc. Feedback may be in response to alarm messages provided to the driver by the agent.

[0104] Feedback can include explicit and / or implicit feedback. Explicit feedback can include a Net Recommendation Score (NPS) rating. For example, feedback can include a rating level provided by the driver regarding previously generated alert messages. The rating level indicates whether the driver likes, is neutral, or dislikes the alert message being provided. Implicit feedback can be provided when the driver takes over control and does not provide an explicit response. Negative scores can be generated when the driver does not respond to a message or ignores it, disengages from alert message delivery, oversteps the recommendation provided in the alert message, and / or prevents the agent from voluntarily providing information to the driver. Scores are recorded, and reward-based actions can be performed by the reward module 1000. For example, certain types of message delivery can be rewarded based on scores, and other types of message delivery can be penalized. Scores and rewards can be stored as part of the driver's preference profile. Future alerts are generated based on rewards. For example, if certain types of message delivery are penalized, the frequency of those types of messages is reduced and / or those types of messages are no longer provided.

[0105] The multimodal dialogue interface module 104 is configured to: i) receive a first score related to the alarm from the passenger; ii) generate a second score based on the response to the alarm and the lack of response to the alarm; and iii) generate a reward based on the first and second scores. The adaptive alarm module 103 is configured to: change the alarm strategy for the passenger based on at least one of the rewards, which is a function of the first and second scores.

[0106] Example score bar 1002 is shown, which can be displayed to the driver when providing feedback. The score bar includes numbers 1-10, where 1 is the lowest score and 10 is the highest score. Faces can be displayed near the numbers, and depending on the score value, the faces can have different facial expressions and be different colors. For example, 1 can be a sad face in red, while 10 can have a smiling face in green. Neutral scores can have a blank face in yellow. As an example, scores 1-6 can be referred to as detractors, scores 7-8 can be referred to as passive scorers, and scores 9-10 can be referred to as promoters.

[0107] The multimodal dialogue interface module 104 can perform state learning based on reactive and proactive templates 910, information provided to and / or received from the assisted (e.g., hands-free) driving system 912, and the learned information represented by box 914. Templates may include contextual information and interpretations, including requests for assisted driving signals and reactive and proactive guidance.

[0108] The learned information can be provided by implementing small-sample (FS) learning (specified as 920), one-sample (1S) learning where K=1 (specified as 922), zero-sample (OS) learning with natural language description (specified as 924), and fine-tuning (FT) training on thousands of examples (specified as 926). FS, 1S, and 0S learning focus on customizing the LLM without changing the parameters, hence the name "frozen" LLM. FT training involves fine-tuning the parameters using a small number of examples (e.g., thousands of examples). The LLM is a context-based, domain-specific LLM. The natural language description of 0S learning is an interpretation without instructing the LLM what response is expected from the LLM for each context. The stated learning / training can be done without changing the parameters of the LLM, making it possible to train the LLM without requiring large amounts of data. A summer 928 is shown and can be used to combine the information generated by implementing one or more of 920, 922, 924, and 926.

[0109] Modules 104 and 105 can perform the declared learning / training based on information received from the cloud-based network device 204 via the telematics module 106 and the distributed communication system 202. The learned information can also be provided from modules 104 and 105 to the cloud-based network device 204.

[0110] Examples disclosed in this paper include: 1) providing adaptive alerts based on eye gaze, head posture, and arm posture detection and tracking signals for an enhanced assisted / automated driving experience that differs from traditional fixed-time alert systems, which do not consider the level of inattention of vehicle occupants; 2) providing adaptive / multi-level alert durations and modalities (e.g., multi-level audio / ambient light / graphics / haptic devices / light strips / smart glass / center console screens, etc.) to enhance the communication experience with occupants compared to traditional single-mode / level alert systems; 3) using multimodal communication to indicate the sensed threat, urgency level, and why the assisted driving mode should be disengaged; 4) implementing an AI conversational agent to provide responsive and proactive assistance for assisted and automated driving with guidance and familiarity in the vehicle; and 5) implementing contextual learning to learn driver preferences and update a personalized profile, which can be stored locally in the host vehicle and / or remotely in a cloud-based network device. Examples provided in this paper include: increasing the enjoyment of driving; reducing stress levels and enabling more relaxed driving; and enhancing the driver experience and satisfaction with assisted driving.

[0111] Examples described in this paper include systems and methods for adaptive alerting, multimodal dialogue HMIs, and context-aware active learning after checking multiple preconditions and taking into account driving posture.

[0112] Examples further include: performing precondition checks, such as checking and using time and speed limits, distance to other vehicles, and visibility conditions, to enable adaptive alert strategies and multimodal dialogue HMIs. These checks include: using visibility confidence levels based on lane markings and / or shoulders to ensure minimum thresholds; when to revert to assisted driving modes in the absence of adaptive alerts; and using forward sensing, and optionally cloud traffic, to determine the degree of isolation / distance / traffic density to determine when driving in the recommended mode is safe.

[0113] Examples further include adaptive alert strategies and multimodal dialogue HMIs that also consider driving posture detected by driver monitoring systems. This includes using eye gaze, head posture, and arm posture to adjust alert timing and alert level relative to inattention levels (e.g., a slight deviation – a gentle alert with ambient light; a very large deviation beyond the adaptive alert posture – a strong alert with haptic seat and audio). The adaptive alert strategy also weights the difference between the primary and secondary hand postures in the adaptive alert strategy, making the distance of the primary hand from the steering wheel more important than the secondary distance.

[0114] Examples further include: enhanced HMIs that redirect the driver's focus to the center of the road ahead and re-engage in assisted driving modes and adaptive alarm modes by using directional ambient light and sound; light strips and / or smart glass for redirecting the driver's attention to the road ahead; headlight flashing in a remote area in low light when no other vehicles are around; a center console display that redirects the driver's focus when the driver is looking in the direction of the center console module (CSM); and the use of enhanced graphic illustrations in cockpit displays (e.g., CSM, instrument cluster (IPC)) to indicate spatial clusters relevant to decision-making.

[0115] The example further includes an AI conversational agent for providing responsive and proactive assistance in guided and familiar driving vehicles. This agent can reactively and proactively answer what / how / where / when and why questions are relevant to assisted driving and the current situation, such as: when an alarm is activated; why the assisted driving mode is disengaged (e.g., due to distracted driving); and why the assisted driving mode changes lanes. Assisted driving can be activated when the user misunderstands or does not know how to perform it after multiple attempts.

[0116] Examples further include In-Context Learning (ICL) to learn user preferences and update personalized cloud profiles. ICL uses explicit / implicit user feedback (ratings, overriding) as the reward function. Context-specific ICL can involve utilizing a static LLM with a small number of samples (FS), one sample (1S), or zero samples (0S) configuration, or alternatively, fine-tuning the model. In the FS scheme, a small number of task demonstrations are presented to the model during inference, acting as conditions without updating any weights. These demonstrations consist of a context and K instances of completion followed by a single context example, where the model is expected to generate the corresponding completion. The 1S setting is similar to FS, but where K is set to 1. The 0S setting mimics FS but replaces the examples, relying instead on a natural language description of the task. Fine-tuning involves updating the weights of a pre-trained model by training on a large number of supervised labels specific to the expected task. Fine-tuning requires adjusting the weights of the pre-trained model by exposing it to reinforcement training with many supervised labels tailored to the target task.

[0117] Examples include: adaptive alerts, conversational agents, contextual learning, and personalized cloud profiles; using directional ambient light and sound to redirect the driver's focus to the center of the road ahead and re-engage in driver assistance mode, and using light strips and / or smart glass to redirect the driver's attention to the road ahead; methods for using location and police databases to determine when it is safe to allow automated driving; logic that adapts alert levels relative to inattention levels and weights the difference between the primary and non-primary hand gestures; and using visibility confidence levels of lane markings and / or shoulders to ensure a minimum threshold, and when to switch back to driver assistance mode in the absence of adaptive alerts; and providing adaptive alerts with enhanced graphical representations in the cockpit display.

[0118] The above description is illustrative in nature and is in no way intended to limit this disclosure, its application, or use. The broad teachings of this disclosure can be implemented in many forms. Therefore, although this disclosure includes specific examples, its true scope should not be so limited, as other modifications will become apparent upon study of the drawings, specification, and appended claims. It should be understood that one or more steps within the method can be performed in a different order (or simultaneously) without altering the principles of this disclosure. Furthermore, although each of the embodiments is described above as having certain features, any one or more of those features described with respect to any embodiment of this disclosure may be implemented in any other embodiment and / or combined with features of any other embodiment, even if such combination is not explicitly described. In other words, the described embodiments are not mutually exclusive, and the arrangement of one or more embodiments with respect to each other remains within the scope of this disclosure.

[0119] Various terms are used to describe spatial and functional relationships between elements (e.g., between modules, circuit elements, semiconductor layers, etc.), including “connection,” “joint,” “coupled,” “adjacent,” “immediately next to,” “on top of,” “above,” “below,” and “set.” Unless explicitly described as “direct,” when describing a relationship between first and second elements in the above disclosure, the relationship can be a direct relationship in which no other intermediary element exists between the first and second elements, or an indirect relationship in which one or more intermediary elements exist (spatially or functionally) between the first and second elements. As used herein, the phrase “at least one of A, B, and C” should be understood to mean logically (A or B or C) using the non-exclusive logic “OR,” and should not be understood to mean “at least one of A, at least one of B, and at least one of C.”

[0120] In the accompanying drawings, as indicated by the arrows, the direction of the arrows generally illustrates the flow of information of interest to the illustration (such as data or instructions). For example, when components A and B exchange various types of information, but the information transmitted from component A to component B is relevant to the illustration, the arrow may point from component A to component B. This unidirectional arrow does not imply that no other information is transmitted from component B to component A. Furthermore, for information sent from component A to component B, component B may send a request for that information or a positive response to receive that information to component A.

[0121] In this application, which includes the definitions below, the term "circuit" may be used instead of the terms "module" or "controller". The term "module" may refer to, be part of, or include the following: application-specific integrated circuit (ASIC); digital, analog, or mixed-signal analog / digital discrete circuit; digital, analog, or mixed-signal analog / digital integrated circuit; combinational logic circuit; field-programmable gate array (FPGA); processor circuitry (shared, dedicated, or group) that executes code; memory circuitry (shared, dedicated, or group) that stores code executed by the processor circuitry; other suitable hardware components that provide the described functionality; or combinations of some or all of the above, such as in a system-on-a-chip.

[0122] A module may include one or more interface circuits. In some examples, the interface circuits may include wired or wireless interfaces connected to a local area network (LAN), the Internet, a wide area network (WAN), or a combination thereof. The functionality of any given module of this disclosure may be distributed among multiple modules connected via the interface circuits. For example, multiple modules may allow for load balancing. In a further example, a server (also referred to as a remote or cloud) module may perform a function on behalf of a client module.

[0123] As used above, the term "code" can include software, firmware, and / or microcode, and can refer to programs, routines, functions, classes, data structures, and / or objects. The term "shared processor circuitry" covers a single processor circuitry that executes some or all of the code from multiple modules. The term "group processor circuitry" covers a processor circuitry that, in conjunction with additional processor circuitry, executes some or all of the code from one or more modules. References to multiple processor circuitry cover multiple processor circuitry on discrete dies, multiple processor circuitry on a single die, multiple cores of a single processor circuitry, multiple threads of a single processor circuitry, or a combination of the above. The term "shared memory circuitry" covers a single memory circuitry that stores some or all of the code from multiple modules. The term "group memory circuitry" covers a memory circuitry that, in conjunction with additional memory, stores some or all of the code from one or more modules.

[0124] The term "memory circuit" is a subset of the term "computer-readable medium." As used herein, the term "computer-readable medium" does not cover transient electrical or electromagnetic signals propagating through a medium (such as on a carrier wave); the term "computer-readable medium" can therefore be considered tangible and non-transient. Non-limiting examples of non-transient tangible computer-readable media are non-volatile memory circuits (such as flash memory circuits, erasable programmable read-only memory circuits, or mask read-only memory circuits), volatile memory circuits (such as static random access memory circuits or dynamic random access memory circuits), magnetic storage media (such as analog or digital magnetic tape or hard disk drives), and optical storage media (such as CDs, DVDs, or Blu-ray discs).

[0125] The apparatus and methods described in this application can be implemented, in part or in whole, by a special-purpose computer created by configuring a general-purpose computer to perform one or more specific functions embodied in a computer program. The function blocks, flowchart components, and other elements described above serve as software specifications that can be translated into computer programs by the routine work of a skilled technician or programmer.

[0126] A computer program includes processor-executable instructions stored on at least one non-transient tangible computer-readable medium. A computer program may also include or depend on the stored data. A computer program may encompass a basic input / output system (BIOS) for interacting with the hardware of a special-purpose computer, device drivers for interacting with specific devices of the special-purpose computer, one or more operating systems, user applications, background services, background applications, etc.

[0127] Computer programs may include: (i) descriptive text to be parsed, such as HTML (Hypertext Markup Language), XML (Extensible Markup Language), or JSON (JavaScript Object Notation); (ii) assembly code; (iii) object code generated from source code by a compiler; (iv) source code for execution by an interpreter; (v) source code for compilation and execution by a just-in-time (JIT) compiler; and so on. As an example only, source code can be written using syntax from languages ​​including: C, C++, C#, Objective-C, Swift, Haskell, Go, SQL, R, Lisp, Fortran, Perl, Pascal, Curl, OCaml, HTML5 (Hypertext Markup Language, 5th Revision), Ada, ASP (Dynamic Server Pages), PHP (PHP: Hypertext Preprocessor), Scala, Eiffel, Smalltalk, Erlang, Ruby, Visual Lua and

Claims

1. A driver assistance system for a main-engine vehicle, the driver assistance system comprising: An adaptive alarm module is configured to adaptively generate alarms with variable duration and intensity; A multimodal dialogue interface module is configured to implement a large language model to use multiple modalities to have dialogue with the passengers of the host vehicle; as well as The driver assistance module is configured to operate in a driver assistance mode and, while in the driver assistance mode, perform a plurality of precondition checks to achieve at least one of the following: i) the adaptive alarm module is to operate in an adaptive alarm mode and provide the alarm via the human-machine interface (HMI) of the host vehicle; And ii) the multimodal dialogue interface module shall perform multimodal dialogue with the passenger via the HMI.

2. The driver assistance system as described in claim 1, wherein the driver assistance mode includes: The driver assistance module controls the steering, acceleration, and deceleration of the main vehicle.

3. The driver assistance system as claimed in claim 1, further comprising: The context-based active learning module is configured to fine-tune the large language model of the multimodal dialogue interface module.

4. The driver assistance system of claim 1, wherein the adaptive alarm module is configured to adaptively change the duration and intensity of the alarm provided to the occupant based on a plurality of parameters.

5. The driver assistance system of claim 1, wherein the adaptive alarm module is configured to adaptively change the duration and intensity of the alarm based on the occupant's level of inattention and the level of imminent potential threat.

6. The driver assistance system of claim 1, wherein the adaptive alarm module is configured to adaptively change the duration and intensity of the alarm based on the detected posture and gaze of the occupant.

7. The driver assistance system as claimed in claim 1, wherein: The multimodal dialogue interface module is configured to perform at least one of the following: using the multiple modalities to send and provide message alerts to the passenger; and The multiple modalities include audio devices, displays, haptic devices, and one or more lights.

8. The driver assistance system of claim 1, wherein the multimodal dialogue interface module is configured to control at least one of ambient lighting, interior lighting, and an audio system to direct the occupant's focus to a point of interest or area.

9. The driver assistance system of claim 8, wherein a point of interest or area refers to the area in front of the display or the host vehicle.

10. The driver assistance system of claim 1, wherein the multimodal dialogue interface module is configured to: control the illumination of a plurality of light strips inside the host vehicle to guide the occupant to a point of interest or area.

Images