Vehicle active interaction system for occupant awareness including timing and interactive style selection

Abstract

An assistance system includes a perception module configured to: collect sensor data, the sensor data including data that tracks a behavior of a vehicle occupant in a host vehicle; and determining perceptual information describing a current situation that ensures initiation of an interactive conversation with the vehicle occupant; an interaction target module configured to determine an interaction target based on the determined perceptual information; an interaction timing module configured to determine a timing to initiate the interaction session based on the interaction target; a conversation module configured to initiate the interactive conversation to provide at least one of a suggestion and a proposal to the vehicle occupant based on the perception information and the timing; and a vehicle control module configured to control operation of at least one of a device and a system of the master vehicle in response to the vehicle occupant accepting the at least one of the suggestion and the proposal.

Description

[0001] introduce

[0002] The information provided in this section is for the purpose of generally presenting the context of the present disclosure. The work of the inventors currently listed (to the extent that it is described in this section) and aspects of the description that may not have been prior art at the time of filing are neither explicitly nor implicitly admitted to be prior art against the present disclosure.

[0003] The present disclosure relates to a vehicle occupant interaction system.

[0004] The master vehicle may include various systems for assisting the driver, for performing autonomous operations, and / or for indicating information about the environment of the master vehicle to vehicle occupants. For example, the master system may include a navigation system that provides map information indicating lane boundaries, street locations, speed limits, the geographic location of a selected destination, etc. The master system may provide instructions to the driver for driving to the selected destination and / or may perform autonomous operations such as braking, steering, and acceleration operations based on the map information to drive the vehicle to the destination.

[0005] As another example, the master vehicle may include an object detection and collision warning system for detecting approaching objects and executing countermeasures and / or taking evasive actions to prevent collisions. The vehicle may include various sensors for detecting objects (such as other vehicles, pedestrians, cyclists, etc.). The controller determines the position of the object relative to the master vehicle and the trajectory of the object and the master vehicle. If it is determined that the master vehicle may collide with one of the objects, one or more alarm signals may be generated to indicate a potential collision to the driver and / or the object of interest. The controller may additionally or alternatively perform one or more other countermeasures (e.g., applying brakes to slow down the master vehicle, changing the steering angle of the master vehicle, etc.) to prevent collisions. Summary of the invention

[0006] An assistance system is disclosed and includes: a perception module, configured to: collect sensor data, the sensor data including data tracking the behavior of a vehicle occupant in a master vehicle; and determine perception information, the perception information describing a current situation that warrants initiating an interactive dialogue with the vehicle occupant; an interaction target module, configured to determine an interaction target based on the determined perception information; an interaction timing module, configured to determine a timing for initiating the interactive dialogue based on the interaction target; a dialogue module, configured to initiate the interactive dialogue based on the perception information and the timing to provide at least one of a suggestion and a proposal to the vehicle occupant; and a vehicle control module, configured to control the operation of at least one of the devices and systems of the master vehicle in response to the vehicle occupant accepting at least one of the suggestion and the proposal.

[0007] In other features, the assistance system further includes: an external analysis module configured to collect data from external sensors; an occupant module configured to collect data from internal sensors; and a vehicle state module configured to collect data from vehicle state sensors. The perception module is configured to determine the perception information based on the data collected from the external sensors, the data collected from the internal sensors, and the data collected from the vehicle state sensors.

[0008] In other features, the perception module determines the perception information based on navigation information received from a navigation module.

[0009] In other features, the assistance system further comprises: an interaction style module configured to select an interaction style from the group consisting of speech, text, and haptic vibration to communicate with the vehicle occupant during the interaction session. The conversation module is configured to communicate with the vehicle occupant using the interaction style during the interaction session.

[0010] In other features, the interaction style module is configured to select an interaction style based on historical data indicating which interaction styles are likely to be accepted by the vehicle occupant.

[0011] In other features, the interaction timing module is configured to determine at least one event trigger and gap pair based on which to initiate the interaction dialog, the at least one event trigger and gap pair being determined based on the interaction target.

[0012] In other features, the interaction timing module is configured to determine an event trigger and a gap pair based on which to initiate the interaction dialog, the event trigger and the gap pair being determined based on the interaction target.

[0013] In other features, the perception module is configured to determine a situation category and a corresponding confidence level. The interaction timing module is configured to determine a timing for initiating the interaction dialog based on the situation category and the confidence level.

[0014] In other features, the interaction timing module is configured to: i) determine a selection criterion based on which at least one event trigger and gap pair is selected, and ii) select the at least one event trigger and gap from a set of event trigger and gap pairs, wherein the selection criterion includes determining at least one of: i) whether a percentage of at least one of acceptance and interaction dialog completion for the selected at least one event trigger and gap pair is greater than a threshold, and ii) the percentage of at least one of acceptance and interaction dialog completion for the selected at least one event trigger and gap pair is greater than a percentage of acceptance and interaction dialog completion for other event trigger and gap pairs in the set of event trigger and gap pairs. At least one of the interaction timing module and the dialog module initiates the interaction dialog based on the selected at least one event trigger and gap pair.

[0015] In other features, the timed interaction module is configured to generate a histogram based on historical data including gaps triggered for at least one event, and select at least one of the gaps based on the histogram. At least one of the timing module and the conversation module initiates the interactive conversation based on the selected at least one of the gaps.

[0016] In other features, the perception module is configured to track gaze and head patterns of the vehicle occupant to determine the perception information based on the gaze and head patterns.

[0017] In other features, the perception module is configured to determine whether the gaze and head patterns indicate whether the vehicle occupant is looking for a refueling station or a charging station. The interaction target module is configured to generate the interaction target to refuel the master vehicle or recharge a battery or battery pack of the master vehicle.

[0018] In other features, the vehicle control module is configured to perform an autonomous driving operation in response to the vehicle occupant accepting the at least one of a suggestion and a proposal.

[0019] In other features, the vehicle control module is configured to activate at least one of a defog system and an air recirculation feature of a heating, ventilation, and air conditioning system in response to the vehicle occupant accepting the at least one of a suggestion and an offer.

[0020] In other features, an auxiliary method is provided and includes: collecting sensor data, the sensor data including data tracking the behavior of a vehicle occupant in a host vehicle; determining perception information, the perception information describing a current situation that warrants initiation of an interactive dialogue with the vehicle occupant; determining an interaction target based on the determined perception information; determining a timing for initiating the interactive dialogue based on the interaction target; initiating the interactive dialogue to provide at least one of a suggestion and an offer to the vehicle occupant based on the perception information and the timing; and controlling the operation of at least one of a device and a system of the host vehicle in response to the vehicle occupant accepting the at least one of the suggestion and the offer.

[0021] In other features, the auxiliary method also includes: collecting navigation information; collecting data from external sensors; collecting data from internal sensors; collecting data from vehicle status sensors; and determining the perception information based on the navigation information, the data collected from the external sensors, the data collected from the internal sensors, and the data collected from the vehicle status sensors.

[0022] In other features, the assistance method also includes: selecting an interaction style from a group including voice, text, and tactile vibration and based on historical data indicating which interaction style is likely to be accepted by the vehicle occupant to communicate with the vehicle occupant during the interaction conversation; and communicating with the vehicle occupant using the selected interaction style during the interaction conversation.

[0023] In other features, the assistance method further comprises determining at least one event trigger and gap pair based on which the interactive dialog is initiated, the at least one event trigger and gap pair being determined based on the interactive target.

[0024] In other features, the assistance method further comprises: determining a situation category and a corresponding confidence level; and determining a timing for initiating the interactive dialog based on the situation category and the confidence level.

[0025] In other features, the assistance method further includes: generating a histogram based on historical data including gaps triggered for at least one event, and selecting at least one of the gaps based on the histogram; and initiating the interactive dialog based on the selected at least one of the gaps.

[0026] A first aspect of the present disclosure provides an assistance system, comprising:

[0027] a perception module configured to: collect sensor data, the sensor data including data tracking behavior of a vehicle occupant in a host vehicle; and determine perception information describing a current situation that warrants initiation of an interactive dialogue with the vehicle occupant;

[0028] an interaction target module, configured to determine an interaction target based on the determined perception information;

[0029] An interaction timing module, configured to determine a timing for initiating the interaction dialogue based on the interaction target;

[0030] a dialogue module configured to initiate the interactive dialogue based on the perception information and the timing to provide at least one of a suggestion and a proposal to the vehicle occupant; and

[0031] A vehicle control module is configured to control operation of at least one of a device and a system of the host vehicle in response to the vehicle occupant accepting the at least one of a suggestion and a proposal.

[0032] The auxiliary system according to the first aspect of the present disclosure further includes:

[0033] an external analysis module configured to collect data from a plurality of external sensors;

[0034] an occupant module configured to collect data from a plurality of internal sensors; and

[0035] a vehicle status module configured to collect data from a plurality of vehicle status sensors,

[0036] The perception module is configured to determine the perception information based on data collected from the multiple external sensors, data collected from the multiple internal sensors, and data collected from the multiple vehicle status sensors.

[0037] According to the assistance system of the first aspect of the present disclosure, the perception module determines the perception information based on navigation information received from a navigation module.

[0038] The assistance system according to the first aspect of the present disclosure further comprises: an interaction style module configured to select an interaction style from the group consisting of voice, text, and tactile vibration to communicate with the vehicle occupant during the interaction dialogue,

[0039] The dialog module is configured to communicate with the vehicle occupant using the interaction style during the interactive dialog.

[0040] The assistance system according to the first aspect of the present disclosure, wherein the interaction style module is configured to select an interaction style based on historical data indicating which interaction style is likely to be accepted by the vehicle occupant.

[0041] According to the assistance system described in the first aspect of the present disclosure, the interaction timing module is configured to determine at least one event trigger and gap pair based on which the interaction dialog is initiated, and the at least one event trigger and gap pair is determined based on the interaction target.

[0042] According to the assistance system of the first aspect of the present disclosure, the interaction timing module is configured to determine a plurality of event triggers and gap pairs based on which the interaction dialog is initiated, and the plurality of event triggers and gap pairs are determined based on the interaction target.

[0043] According to the auxiliary system of the first aspect of the present disclosure, wherein:

[0044] The perception module is configured to determine a situation category and a corresponding confidence level; and

[0045] The interaction timing module is configured to determine a timing for initiating the interaction dialog based on the situation category and the confidence level.

[0046] According to the auxiliary system of the first aspect of the present disclosure, wherein:

[0047] The interaction timing module is configured to: i) determine a selection criterion, select at least one event trigger and gap pair based on the selection criterion, and ii) select the at least one event trigger and gap from a set of event trigger and gap pairs, wherein the selection criterion includes determining at least one of the following: i) whether a percentage of at least one of acceptance and interaction dialog completion for the selected at least one event trigger and gap pair is greater than a threshold, and ii) the percentage of at least one of acceptance and interaction dialog completion for the selected at least one event trigger and gap pair is greater than a percentage of acceptance and interaction dialog completion for other event trigger and gap pairs in the set of event trigger and gap pairs; and

[0048] At least one of the interaction timing module and the dialog module initiates the interaction dialog based on the selected at least one event trigger and gap pair.

[0049] According to the auxiliary system of the first aspect of the present disclosure, wherein:

[0050] The timing interaction module is configured to generate a histogram based on historical data including a plurality of slots triggered for at least one event, and select at least one of the plurality of slots based on the histogram; and

[0051] At least one of the timing module and the dialog module initiates the interactive dialog based on the selected at least one of the plurality of slots.

[0052] According to the assistance system of the first aspect of the present disclosure, the perception module is configured to track the gaze and head pattern of the vehicle occupant to determine the perception information based on the gaze and head pattern.

[0053] According to the auxiliary system of the first aspect of the present disclosure, wherein:

[0054] The perception module is configured to determine whether the gaze and head patterns indicate whether the vehicle occupant is looking for a refueling station or a charging station; and

[0055] The interactive target module is configured to generate the interactive target to refuel the master vehicle or recharge a battery or battery pack of the master vehicle.

[0056] According to the assistance system of the first aspect of the present disclosure, the vehicle control module is configured to perform an autonomous driving operation in response to the vehicle occupant accepting the at least one of a suggestion and a proposal.

[0057] According to the assistance system of the first aspect of the present disclosure, the vehicle control module is configured to activate at least one of a defog system and an air recirculation feature of a heating, ventilation and air conditioning system in response to the vehicle occupant accepting the at least one of a suggestion and a proposal.

[0058] A second aspect of the present disclosure provides an auxiliary method, comprising:

[0059] collecting sensor data, the sensor data including data tracking behavior of vehicle occupants in a host vehicle;

[0060] determining perception information describing a current situation that warrants initiation of an interactive dialog with the vehicle occupant;

[0061] Determine the interaction target based on the determined perceptual information;

[0062] determining a timing for initiating the interactive dialog based on the interactive goal;

[0063] initiating the interactive dialog based on the sensory information and the timing to provide at least one of a suggestion and an offer to the vehicle occupant; and

[0064] Controlling the operation of at least one of the devices and systems of the host vehicle in response to the vehicle occupant accepting the at least one of the suggestion and the offer

[0065] The auxiliary method according to the second aspect of the present disclosure further includes:

[0066] Collect navigation information;

[0067] Collect data from multiple external sensors;

[0068] Collect data from multiple internal sensors;

[0069] Collecting data from multiple vehicle status sensors; and

[0070] The perception information is determined based on the navigation information, data collected from the plurality of external sensors, data collected from the plurality of internal sensors, and data collected from the plurality of vehicle status sensors.

[0071] The auxiliary method according to the second aspect of the present disclosure further includes:

[0072] selecting an interaction style to communicate with the vehicle occupant during the interaction session from the group consisting of speech, text, and haptic vibration and based on historical data indicating which interaction styles are likely to be accepted by the vehicle occupant; and

[0073] Communicating with the vehicle occupant during the interactive session using the selected interaction style.

[0074] The assistance method according to the second aspect of the present disclosure further includes: determining at least one event trigger and gap pair based on which the interactive dialogue is initiated, wherein the at least one event trigger and gap pair is determined based on the interactive target.

[0075] The auxiliary method according to the second aspect of the present disclosure further includes:

[0076] Identify the situation categories and corresponding confidence levels; and

[0077] A timing for initiating the interactive dialog is determined based on the situation category and the confidence level.

[0078] The auxiliary method according to the second aspect of the present disclosure further includes:

[0079] generating a histogram based on historical data including a plurality of gaps triggered for at least one event, and selecting at least one of the plurality of gaps based on the histogram; and

[0080] The interaction session is initiated based on the selected at least one of the plurality of slots.

[0081] Further areas of applicability of the present disclosure will become apparent from the detailed description, claims and drawings.The detailed description and specific examples are intended for purposes of illustration only and are not intended to limit the scope of the present disclosure. BRIEF DESCRIPTION OF THE DRAWINGS

[0082] The present disclosure will be more fully understood from the detailed description and accompanying drawings, in which:

[0083] Figure 1 is a functional block diagram of a master vehicle according to the present disclosure, including an example vehicle control module implementing an active module;

[0084] Figure 2 According to an embodiment of the present disclosure, an interactive timing module is included Figure 1 Functional block diagram of active module and vehicle control module;

[0085] Figure 3 is a single trigger and gap pair selected according to the present disclosure Figure 2 Functional block diagram of the interactive timing module;

[0086] Figure 4 is a functional block diagram of another example interactive timing module and histogram module for selecting a single trigger and gap pair according to the present disclosure;

[0087] Figure 5 is a functional block diagram of another example interactive timing module for selecting a plurality of trigger and gap pairs according to the present disclosure;

[0088] Figure 6 is a functional block diagram of another example interactive timing module and histogram module for selecting a plurality of trigger and gap pairs according to the present disclosure;

[0089] Figure 7 is an example machine learning graph associated with monitoring gaze patterns of vehicle occupants in accordance with the present disclosure;

[0090] Figure 8 is an example view of a vehicle instrument panel including a display showing example interaction buttons for initiating a conversation in a low confidence situation according to the present disclosure; and

[0091] Fig. 9 is a logic flow diagram illustrating an active interaction assistance method according to the present disclosure.

[0092] Among the drawings, reference numerals may be repeated to identify similar and / or identical elements. DETAILED DESCRIPTION

[0093] The virtual assistant of the host vehicle can provide suggestions and / or offers to the vehicle occupants (such as the driver and / or passengers) for certain services and / or to perform certain tasks. For example, the virtual assistant can suggest when to refuel and the location of the gas station relative to the host vehicle. As another example, the virtual assistant can indicate when certain music services are available or when the host vehicle is approaching a restaurant or store that the vehicle occupants are interested in. It is possible for the virtual assistant to act proactively, but in order to be user-friendly, the virtual assistant should minimize interruptions and provide interruptions that are likely to be acceptable to the vehicle occupants. If interruptions are provided too frequently and / or at inappropriate times, the vehicle occupants may consider such interruptions to be an annoyance.

[0094] Examples described herein include a virtual assistance system and a method for active interaction between the virtual assistance system and a vehicle occupant. Active interaction is initiated in a timely manner based on information from multiple sources. The information is combined and used to minimize distractions and provide distractions when there is a high probability that the vehicle occupant will consider the situation appropriate and require such distractions. The virtual assistance system determines when to make distractions and what type of distractions to make so that the vehicle occupant considers the distraction to be timely and acceptable.

[0095] The virtual assistance system monitors the real-time behavior of the vehicle occupant, including speech, gaze patterns, head position, posture (e.g., hand posture, finger posture, facial posture, etc.). The gaze pattern includes the direction of the occupant's head and eyes. Determining the gaze pattern includes determining what the occupant is looking at, how long the occupant looks in a certain direction, how often the occupant changes what the occupant is looking at, etc. For example, and during a specific time period, the vehicle occupant may look straight ahead through the windshield 60% of the time, look at the instrument cluster 20% of the time, and look at the center console 20% of the time. Similarly, the vehicle occupant may look at the fuel gauge in the instrument cluster, then look at the gas station sign, and then look at the navigation map. This may indicate that the occupant is looking for a gas station or other business. Based on this information, an interactive dialogue with the vehicle occupant can be initiated. As another example, a user may show a series of glances and / or a series of user actions when looking for something (e.g., a specific business and / or a type of business), based on which an interactive dialogue with the vehicle occupant can be initiated.

[0096] The virtual assistance system collects vehicle context data, such as vehicle network data, from a vehicle bus or interface (such as from a controller area network (CAN) bus including vehicle activity features). The virtual assistance system also collects data from various sensors and navigation systems. The sensors include external monitoring sensors, internal monitoring sensors, and other sensors such as vehicle status sensors. The virtual assistance system can monitor the environment of the vehicle, including performing external scene analysis and monitoring traffic, road and weather information. The virtual assistance system also collects historical data, including occupant historical behavior data (e.g., driving mode, parking mode, exit mode, refueling mode, shopping preferences, occupant use of the infotainment system, including the use of specific applications, the use time of the application, and the actions performed in the application, etc.) and other historical data, including dialogue interaction history. The interaction history includes previous interactive dialogues between the virtual assistance system and the vehicle occupant. The interaction history includes i) the probability that the vehicle occupant quickly accepts the suggestions and proposals provided, and ii) the duration of the interaction between the virtual assistance system and the vehicle occupant. The interaction history can be occupant-specific, vehicle-specific, geographic location-specific, associated with multiple and / or a group of vehicle occupants, associated with multiple and / or a group of vehicles, etc.

[0097] The virtual assistant system determines when to initiate interference and interaction based on the above information and other information mentioned herein. As further described below, the information can be stored, generated, collected, compared and / or analyzed. The virtual assistant system also selects the style and form of interaction, such as audio (or voice), displayed text, tactile vibration, displayed objects, etc.

[0098] The virtual assistance system implements an algorithm to determine the timing of initiating an interactive proactive dialog with the vehicle occupant. The timing is selected to minimize the false alarm rate so that the number of rejected suggestions and / or offers is minimized and / or zero. Sensor data including the vehicle occupant's gaze pattern is used as input to the algorithm. The interaction style is selected and / or changed to maximize the success rate (i.e., the percentage of vehicle occupants accepting suggestions and / or offers and the percentage of vehicle occupants completing the corresponding interactive dialog).

[0099] As an example, the virtual assistance system can determine that the fuel level indicated by the fuel gauge of the host vehicle is low and / or within the range when the driver usually refuels. The fuel level may not be low enough to light the low fuel light and / or indicate to the driver that refueling is needed. The driver may be looking for a fuel station sign and / or a fuel station along the roadside. This can be detected based on data from one or more internal cameras, one or more external cameras, and a navigation system. The virtual assistance system can determine the appropriate time to initiate a conversation that suggests a suitable fuel station to the driver. The interactive conversation is initiated at the best distance (e.g., the amount of time to initiate an interactive conversation or the amount of time to arrive at a fuel station) at which the driver accepts the refueling suggestion and continues the interactive conversation. In an embodiment, when the possibility of the driver accepting the suggestion and continuing the interactive conversation is higher than a set threshold, an interactive conversation is initiated. Select an interactive style to maximize the success rate. As an example, the virtual assistance system can display text and / or audibly play a message such as "Do you want me to find a suitable fuel station for you along your route?".

[0100] Figure 1 A vehicle 100 is shown including an active virtual assistance system 102 having a vehicle control module 103, which, as shown, includes an active module 104. As further described below, the active module 104 performs perception (or situation) determination operations, historical data lookup and data collection operations, interaction timing operations, interaction style operations, and dialogue operations, including providing voice, text, and / or tactile messages. The vehicle control module 103 can perform various operations based on interactions with the user and messages generated as further described below. The vehicle control module 103 can perform autonomous operations based on interactions including responses received from the user.

[0101] The vehicle 100 and active virtual assistance system 102 also include one or more power sources 105, a telematics module 106, an infotainment module 107, other control modules 108, and a propulsion system 110. The vehicle control module 103 may control the operation of the vehicle 100 and the modules 104, 106, 107, 108, and the propulsion system 110. The power source 105 may include one or more battery packs, generators, converters, control circuits, terminals for high voltage loads and low voltage loads, etc., and one or more battery sensors 111 for detecting the status of the power source 105 (including voltage, current level, charging status, etc.).

[0102] The telematics module 106 provides wireless communication services within the vehicle 100 and communicates wirelessly with service providers and devices outside the vehicle 100. The telematics module 106 may support Bluetooth Low Energy (BLE), Near Field Communication (NFC), Cellular, Traditional (LG) Transmission Control Protocol (TCP), Long Term Evolution (LTE), and / or other wireless communications and / or based on The telematics module 106 may include one or more transceivers 112 and a navigation module 114 having a global positioning system (GPS) and a GNSS (or global navigation satellite system) receiver 116. The transceiver 112 communicates wirelessly with network devices inside and outside the vehicle 100 (including cloud-based network devices, central stations, backends, and portable network devices). The transceiver 112 can perform pattern recognition, channel addressing, channel access control, and filtering operations.

[0103] The navigation module 114 executes a navigation application to provide navigation services. The navigation service may include a location identification service for identifying the location of the vehicle 100. The navigation service may also include guiding the driver and / or guiding the vehicle 100 to a selected location. The navigation module 114 may communicate with a central station to collect map information indicating traffic levels, transport object identification and location (e.g., location and type of signs), path information, where rest areas are located, where gas stations are located, where restaurants are located, etc. As an example, if the vehicle 100 is an autonomous vehicle, the navigation module 114 may guide the vehicle control module 103 to arrive at a selected destination along a selected route. The GPS and GNSS receiver 116 may provide vehicle speed and / or direction (or heading) and / or global clock timing information of the vehicle 100 and other vehicles and objects (e.g., pedestrians and cyclists).

[0104] The infotainment module 107 may include and / or be connected to an audio system 122 and / or a video system including one or more displays (one display 120 is shown). The display 120 and the audio system 122 may be part of a human-machine interface. The display may include a cluster and / or center console display, a head-up display, etc. In addition to the display and audio system 122, a haptic device 124 (e.g., a steering wheel and / or seat vibration device) may also be used to interact with a vehicle occupant (such as a driver or passenger). This interaction is further described below. Messages may be displayed, audibly played, and / or indicated via the display 120, the audio system 122, the haptic device 124, and / or via one or more other output devices.

[0105] The infotainment module 107 may provide a variety of proactive messages and information, including information regarding upcoming and / or nearby gas stations, upcoming and / or nearby restaurants, music services, upcoming and / or nearby stores, vehicle status information, diagnostic information, precursor information, entertainment features, etc. The infotainment module 107 may be used to direct the vehicle operator to a particular location, indicate a trip estimate (e.g., distance to a selected destination), and other information.

[0106] The propulsion system 110 may include one or more torque sources, such as one or more electric motors and / or one or more engines (eg, internal combustion engines). Figure 1 In the example shown in , the vehicle 100 includes an engine 130 and one or more motors 132. The torque sources are independently controlled. The propulsion system 110 includes a motor control system 134, which includes one or more motors 132 and a motor control module 136, which can control the operation of the one or more motors 132 based on signals from the vehicle control module 103.

[0107] The modules 103 , 104 , 107 , 108 may communicate with each other via one or more buses 140 , such as a controller area network (CAN) bus, and / or other suitable interfaces. The vehicle control module 103 may control the operation of vehicle modules, devices, and systems based on feedback from sensors 150 .

[0108] The sensors 150 may include external sensors 152, internal sensors 154, and other sensors 156. The external sensors 152 may include radio radar and / or lidar sensors 158 and imaging and audio devices (e.g., cameras and microphones or microphone arrays) 160. The external sensors 152 may be used to detect objects outside the vehicle 100 and / or in the path of the vehicle 100. The internal sensors 154 may include internal imaging sensors (e.g., cameras) 162 and microphones or microphone arrays 164. The internal sensors 154 may be used to monitor vehicle occupants. As an example, the internal sensors 154 may track the driver's eyes and eye gaze direction, detect gestures made by the driver, detect the orientation of the driver's body, detect the driver's voice, etc. This monitoring is used to determine whether the driver is looking for a gas station, a restaurant, a rest area, or other stores and / or locations. This monitoring can be used to determine whether the driver is looking at a specific sign or type of sign along the path of the vehicle 100. The internal sensors 154 can be used to detect, for example, when the driver is looking at a sign indicating an upcoming gas station. This indicates that the driver wants to stop and refuel. This type of monitoring is also applicable to other types of businesses, rest areas, and / or locations. Other sensors 156 may include a vehicle speed sensor 166 , acceleration sensors (eg, longitudinal and lateral acceleration sensors) 168 , and a fuel level sensor 170 , as shown, as well as other sensors such as an engine temperature sensor and an engine oil pressure sensor.

[0109] The vehicle control module 103 may also include a mode selection module 172 and a parameter adjustment module 174. The mode selection module 172 may select a vehicle operating mode. The parameter adjustment module 174 may be used to adjust parameters of the vehicle 100. As an example, the vehicle control module 103 may operate in a fully or partially autonomous mode and may control the propulsion system 110, the braking system 176, and the steering system 178. In one embodiment, the vehicle control module 103 controls the operation of the systems 110, 176, and 178 based on interaction with the vehicle occupants via the active module 104. The vehicle control module 103 may: i) perform autonomous operations, such as steering, braking, accelerating, etc., and / or ii) display and / or audibly play messages, perform tactile operations via the tactile device 124, and / or output messages and / or corresponding signals via other output devices.

[0110] The vehicle 100 may also include a memory 180. The memory 180 may store sensor data 182, parameters 184, applications 186, algorithms 188, historical data 190, and other data 192. The parameters may include sensor parameters such as vehicle speed, vehicle acceleration, battery state of charge, fuel level, etc. Applications 186 (e.g., trip energy estimation applications). Applications 186 may include applications executed by modules 103, 104, 107, 108.

[0111] Although the memory 180 and the vehicle control module 103 are shown as separate devices, the memory 180 and the vehicle control module 103 may be implemented as a single device. The memory 180 may also store historical data 190 and other data 192, such as driver driving patterns, driver refueling patterns, driver parking patterns, driver pickup patterns, other driver patterns, data collected and / or generated by the active module 104, traffic data, navigation data, map data, GPS data, path data, speed data, acceleration data, etc.

[0112] The vehicle control module 103 can control the operation of the following: the propulsion system 110, the video system including the display 120, the audio system 122, the haptic device 124, the braking system 176, the steering system 178, the heating ventilation and air conditioning (HVAC) system 193, the lighting system 194, the seat system 196, the rearview mirror system 198 and / or other devices and systems according to the parameters set by the modules 103, 104, 107, 108. The HVAC system 193 can include a defog system and / or have an air recirculation feature. Additional defog systems can also be included. The vehicle control module 103 can set at least some of the parameters based on the signals received from the sensor 150.

[0113] The vehicle control module 103 may receive power from the power source 105, which may be provided to the propulsion system 110, the braking system 176, the steering system 178, the HVAC system 193, the lighting system 194, the seat system 196, the mirror system 198, etc. The power supplied to the motor 132, the braking system 176, the steering system 178, the HVAC system 193, the lighting system 194, the seat system 196, the mirror system 198, and / or their actuators may be controlled by the vehicle control module 103 to adjust, for example: motor speed, torque, and / or acceleration; brake pressure; steering wheel angle; pedal position; state of the haptic device 124; etc. This control may be based on the output of the sensors 150, the navigation module 114, the GPS and GNSS receiver 116, and the data and information stored in the memory 180.

[0114] The vehicle control module 103 can determine various parameters, including vehicle speed, motor speed, gear state, accelerator position, brake pedal position, regeneration (charging) power, automatic start / stop discharge power, and / or other information. The power supply 105 and / or its control circuit can determine other parameters, such as: the charge power at each power terminal; the discharge power at each power terminal; the maximum and minimum voltage at the power terminal; the maximum and minimum voltage at the power rail, battery cell, battery block, battery pack and / or battery pack; the SOX value of the battery cell, battery block, battery pack and / or battery pack; the temperature of the battery cell, battery block, battery pack and / or battery pack; the current value of the battery cell, battery block, battery pack and / or battery pack; the power value of the battery cell, battery block, battery pack and / or battery pack; etc. The acronym "SOX" refers to the state of charge (SOC), the state of health (SOH), the state of power (SOP) and / or the state of function (SOF). The power, voltage and / or current sensor can be independent of the power supply 105 and / or included in the power supply 105 for SOX determination. The SOC of a battery cell, battery pack, and / or battery group may refer to the voltage, current, and / or available power stored in the battery cell, battery pack, and / or battery group. The SOH of a battery cell, battery pack, and / or battery group may refer to: life (or operating hours); whether there is a short circuit; whether there is a loose wire or poor contact; the temperature, voltage, power level, and / or current level supplied to or derived from the battery cell, battery pack, and / or battery group under certain operating conditions; and / or other parameters describing the health of the battery cell, battery pack, and / or battery group. The SOF of a battery cell, battery pack, and / or battery group may refer to the current temperature, voltage, and / or current level supplied to or derived from the battery cell, battery pack, and / or battery group, and / or other parameters describing the current functional state of the battery cell, battery pack, and / or battery group. The power supply 105 may determine the connection configuration of the battery cells described herein and the corresponding switch state based on parameters determined by the vehicle control module 103 and / or the control circuit (or module) of the power supply 105.

[0115] Figure 2 Shows Figure 1The vehicle control module 103 and the active module 104. The active module 104 can receive data and information from the navigation module 114, sensors 152, 154, 156 and / or other devices and sensors, such as information from a wireless network, information from nearby vehicles, information from the background, etc. The active module 104 can include an external analysis module 200, an occupant module 202, a vehicle state module 204, a perception module 206, a data collection and interaction target module 208, a history module 210, an interaction style module 212, an interaction timing module 214, and a dialogue module 216. In one embodiment, the modules 208 and 216 are integrated to form a single module.

[0116] Figure 1 102 Implementation of an Active Virtual Assistance System Figure 2 2 and 206. The active level 218 and the action level 220 shown in FIG. The active level 218 includes the navigation module 114, the sensors 152, 154, 156, and the modules 200, 202, 204, and 206. The action level 220 includes the modules 208, 210, 212, 214, and 216. The active level or a portion thereof can collect data and / or information from other systems. For example, the external analysis module 200 can collect data and information about traffic objects from an autonomous or semi-autonomous driving module or system, rather than directly from the external sensor 152.

[0117] The external analysis module 200 is configured to collect data from the navigation module 114 and the external sensors 152 to detect the external environment of the master vehicle and the presence and location of nearby businesses. The external analysis module 200 detects traffic objects, such as: vehicles; pedestrians; cyclists; road signs; traffic lights; signs indicating upcoming businesses (such as gas stations, restaurants, stores, etc.); and other objects. The external analysis module 200 detects lanes, roads, road curvature, etc. The external analysis module 200 provides the detected and determined information to the perception module 206. In an embodiment, the external analysis module 200 receives a raw data stream from the external sensors 152. Based on the raw data stream, the external analysis module 200 creates a list of objects around the master vehicle, location types, road curvature and type, etc. The raw data can originate from other devices and / or systems and then from the external sensors 152. The external analysis module 200 sends the list of objects to the perception module 206.

[0118] The passenger module 202 collects data from the internal sensors 154. The passenger module 202 tracks, monitors and / or detects the passenger's head position, eye movement, direction and position, posture, body positioning, passenger voice, etc., and provides the information to the perception module 206. The posture can include waving, finger swiping, finger tapping, finger pointing, nodding, arm movement, etc. The posture can be provided on the display, on the steering wheel, on the center console, on the armrest, on another vehicle component, on the shifter (or transmission), and / or without touching the vehicle component. The passenger module 202 can receive the raw data stream of the internal sensor 154 located in the internal cockpit of the main control vehicle. This information is provided to the perception module 206 to extract passenger and driver related information. The extracted information includes the gaze pattern and gaze direction determined based on the raw data stream from the internal sensor 154.

[0119] The vehicle state module 204 collects data from vehicle state sensors, such as vehicle speed, vehicle longitudinal and lateral acceleration, fuel level, battery state of charge, etc. The vehicle state module 204 determines the vehicle state and provides this information to the perception module 206. The vehicle state module 204 may receive raw data from the sensors 156. The vehicle state module 204 may send the raw data and / or information determined based on the raw data to the perception module 206.

[0120] The perception module 206 determines the current situation based on the information collected from the modules 200, 202, 204 and optionally based on the information from the navigation module 114. The perception module 206 processes and analyzes the collected information to determine the current situation. In addition to the above information provided by the navigation module 114, the navigation module 114 can also provide information such as where the master vehicle is going, the possible path of the master vehicle, the indication of where the master vehicle is, and the driver's history. The driver's history can include, for example, the indication of the typical destination of the driver, the typical parking point of the driver, the typical likes and dislikes of the driver. The perception module 206 determines the situation and indicates the situation to the data collection and interactive target module 208. As an example, the situation can include the following indications: whether the fuel level is suitable for suggesting refueling, whether the vehicle is within the appropriate range of the fuel station to suggest refueling at the gas station, and whether the driver's behavior is suitable for suggesting refueling. Information for various different situations can be provided.

[0121] In one embodiment, the perception module 206 anticipates the driver's needs and sends a message to initiate a relevant conversation to the data collection and interaction target module 208 and / or the dialogue module 216. For example, if the host vehicle has a low fuel level and the driver keeps looking at the fuel station signs, locations, and / or prices, the perception module 206 sends a message to initiate a conversation about finding and driving to the relevant fuel station.

[0122] The data collection and interaction goal module 208 collects information from modules 206 and 210 and provides the information to modules 212, 214. The data collection and interaction goal module 208 receives past interaction data from the history module 210 and enhances the information with other collected information, which is then provided to modules 212, 214. The data collection and interaction goal module 208 can determine interaction goals based on the collected data. The interaction goal can be, for example, to obtain approval to: go to a fuel station to refuel the master vehicle; stop at a restaurant; go to a charging station to recharge one or more batteries and / or battery packs of the master vehicle; pick up a passenger; or perform another task. The interaction goal can be indicated to the modules 212, 214.

[0123] The history module 210 may collect previous active interaction data to determine when a vehicle occupant typically accepts a particular suggestion and / or offer. The history module 210 may provide probabilities associated with the acceptance of suggestions and offers. The probabilities may be associated with different interaction styles, specific to geographic locations, associated with the timing of suggestions and offers, etc. The timing refers to: i) the amount of time until the suggestion and / or offer is presented to the user, and / or ii) the amount of time until the host vehicle arrives at the corresponding destination of the suggestion and / or offer (e.g., a fuel station or restaurant). The timing may refer to the distance between the current host vehicle location and i) the location when the suggestion and / or offer is to be provided or ii) the location when the host vehicle arrives at the corresponding destination of the suggestion and / or offer.

[0124] The interaction style module 212 determines the interaction style based on the data collection and information provided by the interaction target module 208. The interaction style may include speech played via an audio system, text displayed on a display, haptic vibrations, icons (or buttons) displayed on a display, and the like.

[0125] The interaction timing module 214 determines when it is appropriate to initiate an interactive session with the vehicle occupant based on the data collection and information provided by the interaction target module 208. The interaction timing module 214 can implement one or more methods to determine the appropriate timing. Some example methods refer to Figure 3-7 Give a description.

[0126] The interaction timing module 214 can determine the appropriate timing based on the triggering event. As an example, the timing can be based on the state of charge of one or more batteries (or one or more battery packs). If the battery state of charge level is low, and the driver is looking for a charging station and / or looking at a charging station, and the current driver destination is outside the range of the master vehicle for the current charging state, a conversation can be initiated. In this case, the interaction timing module 214 initiates a conversation at the appropriate time and suggests a charging station to park. The conversation is initiated at the optimal distance from the charging station and / or at the optimal amount of time before the charging station. The timing (distance and amount of time) has the highest corresponding probability for the driver to accept and continue the corresponding interactive conversation. The interaction style (or optimal style) is selected by the interaction style module 212 to maximize success (i.e., the driver accepts and continues the interaction until the station is reached). Other example triggering events are described below.

[0127] The dialogue module 216 may be referred to as a "virtual assistant" and proactively initiates a dialogue with the vehicle occupant at the appropriate time. The dialogue begins with the dialogue module 216 providing advice and / or suggestions to the vehicle occupant. The advice and / or suggestions may be presented in the form of a question, a statement, an icon (or button) to be tapped, or in some other manner that is not considered annoying by the vehicle occupant. The dialogue module 216 may send signals 222 to and from, for example, a display, an audio system, and a haptic device (such as the display 120, the audio system 122, the haptic device 124).

[0128] As another example of a triggering event, an interactive dialog is initiated when the fuel gauge of the host vehicle indicates that the fuel is low, but not low enough for the low fuel light to turn on, and i) the driver repeatedly looks at the fuel station sign, and / or ii) at the fuel station or the driver is distracted. The dialog module 216 initiates a dialog to suggest a suitable fuel station to the driver. The dialog is initiated at an optimal distance and / or amount of time from the fuel station where the driver has the highest likelihood of continuing the interactive dialog. The timing can be determined and set by the interactive timing module 214. The style of the interaction is selected by the interactive style module 212.

[0129] As another example of a triggering event, the dialogue module 216 can initiate a dialogue to perform partial or fully autonomous driving (e.g., Level 2 driving). This may occur when certain conditions exist, such as: the road ahead is open (i.e., few or no other vehicles ahead); the weather is good; and no premature disengagement is detected (e.g., the driver takes over driving control). The interaction timing module 214 determines the best time to start the interaction dialogue for autonomous driving.

[0130] An interactive dialog may be initiated to: find a fuel station while the driver of the master vehicle is having a phone conversation; activate the defog system; activate the air recirculation feature of the master vehicle's HVAC system; make a scheduled call; play the occupant's favorite music or podcast; and pick up a passenger along the route. The interactive timing module 214 may determine the best time to remind the driver to pick up the passenger. This may be determined based on historical data such as past driver habits and / or passenger pick-up times, and / or may be based on a stored passenger pick-up schedule. As an example, an interactive dialog may be initiated to turn on the air recirculation feature when the master vehicle is traveling in an area that is known to be typically "stinky." For example, when driving past a garbage dump. In addition, the interactive dialog may result in a setting change that determines the level of automation for future situations, such that recirculation will always be automatically activated in future situations.

[0131] In an embodiment, the perception module 206 determines the situation category of a particular detected situation. This information can be used by the interaction timing module 214 to determine the timing of the interactive dialogue. The method for classifying the situation is performed by the perception module 206. The classification can be based on data from sensors 152, 154 and 156, including data indicating the state of the environment, the behavior of the driver (or vehicle occupant), and the components and systems of the main control vehicle. The perception module 206 formalizes the situation of the given input and outputs a classification and a confidence level in the classification. The confidence level can be a number between 0 and 1. A low confidence level indicates a low probability that the classification is accurate. A high confidence level indicates a high probability that the classification is accurate.

[0132] The confidence level refers to the confidence that the perception module 206 has in its own predictions, including, for example, how accurate the classification of the situation is. The likelihood of a user accepting an offer can be based on a model learned about the user. In one embodiment, when the confidence level is high and when the classification of the situation refers to the likelihood of the user accepting an offer, the likelihood that the user will accept the suggestion and / or offer is high. In another embodiment, the confidence level is based on the timing of the intervention rather than the content of the intervention.

[0133] The situation categories may include: an instant communication category, which requires immediate interaction with the user (e.g., immediate voice interaction with the user); a timed communication category, which requires interaction with the user within X seconds, where X is a positive real number; a question category, which requires clarification from the user; and an explanation category, where the user requires an explanation. In an embodiment, the situation categories are determined based on a set of rules and / or based on a machine learning classifier. The machine learning classifier can learn from expert annotated data or from research that reveals what users prefer to get in certain driving situations.

[0134] As an example, if the user is looking at the fuel gauge and around the road with some frequency, the dialogue module 216 can initiate a dialogue with the user to clarify whether the user is looking for a gas station. As another example, if the user is looking at the display of the navigation system and the signs along the road, the dialogue module 216 can provide information for the next turn or exit ramp. As yet another example, if the user keeps engaging and disengaging Level 2 autonomous driving, the dialogue module 216 can provide an explanation of the driving behavior of the host vehicle.

[0135] When implementing the rule-based method, the interactive timing module 214 can initiate an interactive dialogue when a predetermined set of rules is met and / or a condition of a predetermined pattern is detected. The rule may include logical terms, such as AND, OR and NOT that associate a condition with a specific event. As an example, when the fuel level is low and the driver is looking at a fuel station sign and / or a fuel station, a specific event may exist. As another example, when the distance (e.g., the distance from the current position or the navigation distance) is less than a threshold value (e.g., 4 kilometers) and the driver is looking for a parking space, a specific event may exist. As another example, when the workload based on the driver's gaze pattern is low and this occurs after a predetermined time period (e.g., 5 seconds) after the driver looks at the rearview mirror of the master vehicle, a specific event may exist. The driver's gaze pattern can be detected, and when the gaze pattern matches a predetermined pattern and / or meets a predetermined criterion, an interactive dialogue is initiated.

[0136] In an embodiment, the interactive timing module 214 determines the timing of initiating an interactive dialogue based on the situation category and the corresponding confidence level. For example, the timing i) may occur now, ii) may occur within x seconds, where x is a positive real number, or iii) may not occur. The dialogue module 216 may play or display the message now, within x seconds, or not play or display the message at all, even if the situation may be suitable for making suggestions and / or proposals. The timing may be rule-based and / or based on the output of a machine learning classifier.

[0137] In another embodiment, the timing is based on event triggers and gaps. Event triggers refer to one or more conditions that exist to warrant an interactive dialog. Gap refers to the distance between the host vehicle's current location and i) the location when the interactive dialog is initiated or ii) the recommended destination associated with the suggestion and / or offer. The gap can be defined in terms of an amount of time, such as i) the amount of time until the interactive dialog is initiated or ii) the amount of time until the host vehicle reaches the recommended destination.

[0138] One or more event triggers and gap pairs can be selected and used as the criteria based on which the interactive dialogue is initiated. The selection of a single event trigger and gap pair can be referred to as a hard selection. The selection of multiple event triggers and gap pairs can be referred to as a soft selection. Each event trigger and gap pair defines the time point at which the interaction starts. The optimal timing is learned through an iterative learning process that maximizes the optimality criterion. The optimality criterion refers to one or more event triggers and gap pairs that have the highest probability of causing the corresponding suggestions and / or proposals to be accepted by the user and / or the interactive dialogue to be completed (i.e., the user accepts and follows the suggestions and / or proposals) when used to initiate a dialogue. An example of timing includes starting an interactive dialogue X meters before the master vehicle leaves the highway to go to a fuel station. As another example, timing can include starting an interactive dialogue T seconds before the master vehicle may arrive at a specific restaurant. As another example, timing can include starting an interactive dialogue K kilowatt hours (kWh) before the master vehicle may arrive at a charging station, where X, T, and K are positive real numbers. This is an example of defining timing or more specifically defining gaps in kilowatt hours (or battery charge state) rather than time or distance. The gap may also be defined based on the fuel level of the host vehicle (ie, the amount of fuel in the fuel tank).

[0139] Figure 3 A method of selecting a single event trigger (or trigger) and gap pair from one or more sets of event triggers, gaps, and / or event trigger and gap pairs that may be stored in a memory is shown. Figure 2 The interaction timing module 214 may include a trigger type selection module 300, a gap type selection module 302, and a criteria selection module 304. The trigger type selection module 300 selects an event trigger 305 based on an interaction goal 306, which may be provided by the data collection and interaction goal module 208. The event trigger selection may be based on the location of the host vehicle and a recommended destination (e.g., the location of a fuel station or a charging station), an engine temperature greater than a first threshold, an engine oil pressure less than a second threshold, and / or based on another event trigger.

[0140] The gap type selection module 302 selects a gap 307 based on the interaction goal. The gap type can be measured in terms of distance (e.g., miles or kilometers), time (e.g., seconds), fuel status (e.g., liters or gallons), battery or battery pack status (e.g., kWh), and / or via other measurement parameters.

[0141] Criteria selection module 304 evaluates event triggers and gap pairs and determines criteria that provide event triggers and gap pairs with a high interactive dialog completion percentage. The optimality criteria are selected and used to control the selection of event triggers and gap pairs selected by modules 300 and 302.

[0142] Figure 4An interactive timing module 400 and a histogram module 402 for a single trigger and gap pair are shown. The interactive timing module 400 may be substituted for Figure 2 The interactive timing module 400 may include a trigger type selection module 404, a gap type selection module 406, and a criterion selection module 408, which may be used together with Figure 3 Modules 300, 302, 304 operate similarly and can communicate with the histogram module 402. Modules 404, 406 operate based on the interaction goal 405 and select event triggers 407 and gaps 409. The histogram module 402 can generate a histogram based on historical data such as the percentage of completion of the interactive dialogue and the gaps. The histogram includes the percentage of completion for different gaps. The gap value with the highest percentage of completion can be selected by the gap type selection module 406. Therefore, the gap is selected from a group of gaps that may be associated with one or more event triggers. In an embodiment, a histogram is generated based on historical data of event triggers, gaps, and optimality criteria. The histogram module 402 or other modules can periodically perform exploration to verify that the histogram has not changed. Figure 4 An example histogram curve 410 is shown where the maximum completion percentage of an interaction is at 4, which may be measured in seconds.

[0143] Figure 5 Another example of another interaction timing module 500 is shown, which can be used instead of Figure 2 The interactive timing module 214 is configured to select a plurality of trigger and gap pairs. The interactive timing module 500 may include a plurality of trigger type selection modules 502, 504, a plurality of gap type selection modules 506, 508, and a criteria selection module 510. The modules 502, 504, 506, 508, 510 may be associated with Figure 3 Modules 300, 302, 304 of the present invention operate similarly. Based on the same interaction goal 509, the trigger type selection modules 502, 504 select event triggers 505, and the gap type selection modules 506, 508 select gaps 507. Each event trigger 505 is associated with a corresponding one of the selected gaps 507. When determining whether to initiate an interaction dialog and when determining the timing of an interaction dialog, multiple selected event trigger and gap pairs can be combined and used. The criteria selection module 510 can review historical data to determine the event trigger and event pair combination with the highest associated acceptance and completion percentages.

[0144] Figure 6 An interactive timing module 600 and a histogram module 601 for multiple trigger and gap pairs are shown. The interactive timing module 600 can replace Figure 2The interactive timing module 600 may include a plurality of trigger type selection modules 602, 604, a plurality of gap type selection modules 606, 608, and a criteria selection module 610. The modules 602, 604, 606, 608, 610 may be associated with Figure 3 Modules 300, 302, 304 and Figure 5 The modules 502, 504, 506, 508, 510 operate similarly. Based on the same interaction target 609, the trigger type selection modules 602, 604 select event triggers 605, and the gap type selection modules 606, 608 select gaps 607. Each event trigger 605 is associated with a corresponding one of the selected gaps 607.

[0145] The histogram module 601 generates a histogram for different gap sets A and B. A histogram is a two-dimensional feature space with a combination of gaps and their associated event triggers. Each of the gap sets A and B is associated with a corresponding one of the event triggers 605. In the example shown, a histogram and a threshold ring 620 are shown. In an embodiment, two gaps within the threshold ring 620 are selected, one from the gap set A and the other from the gap set B. In another embodiment, a gap at the center of the histogram and / or at the center of the threshold ring 620 is selected. The threshold ring 620 can be centered on the gap with the highest associated acceptance and completion percentage. A threshold can be set and used in relation to an optimality criterion so that a gap combination with an associated acceptance and completion percentage greater than a threshold is selected. If there are more than two gaps with associated acceptance and completion percentages, the two gaps with the highest associated acceptance and completion percentages are selected.

[0146] although Figure 5-6 It is described with respect to two event trigger and gap pairs, but more than two event trigger and gap pairs can be used. Other features can be combined using other related representations of the feature space. The optimal gap can be predicted based on criteria created using logical operators or machine learning algorithms.

[0147] Figure 7An example machine learning graph 700 associated with monitoring the gaze patterns of vehicle occupants is shown. The machine learning graph 700 and / or corresponding algorithms can be used by any interaction timing module mentioned herein. Machine learning occurs based on examples. Modeling of locations in a space where interactions have correct timing and other locations where interactions have incorrect timing is performed. In this space, models such as k-means, probabilistic models, or deep learning are used to estimate the effectiveness of the timing of interactions. The machine learning graph 700 is a graph related to the following: i) the likelihood that the driver is looking at the windshield (represented by arrow 702), ii) the likelihood that the driver is looking at the center console (represented by arrow 704), and iii) the likelihood that the driver is looking at the instrument cluster (represented by arrow 706). An example point 708 for correct timing and an example point 710 for incorrect timing are shown. Points 708 and 710 are within a triangle 712, which is located on and / or centered at the intersection 714 of arrows 702, 704, and 706. The gaze pattern associated with point 708 may be used as an event trigger.

[0148] When machine learning is used to determine the appropriate timing for initiating an interactive dialog, learning examples can be used to train the machine learning algorithm. This may include learning the driver's comfortable acceleration range and jerk range. Acceleration is the derivative (or change) of velocity, and jerk is the derivative (or change) of acceleration. The driver's comfort range of longitudinal and lateral acceleration and longitudinal and lateral jerk can be learned. Rules for determining timing can be used based on these parameters and / or other parameters that define the driver's typical patterns and / or characteristics.

[0149] Figure 8 A dashboard 800 of a vehicle is shown that includes a display 802 that displays an example interaction button 804 for initiating an interaction dialog in a low confidence situation (confidence is too low to start a voice interaction; however, confidence is high enough to start a display interaction). The interaction button 804 is shown as an example of a refueling example. For other situations, other buttons and / or icons may be displayed. The interaction button 804 may be displayed when it is determined that the confidence level that the user will accept and / or complete an interaction dialog associated with finding, locating, and driving to a gas station is low. Figure 8 In FIG. 8 , a steering wheel 806, a dashboard display 808, and a center console (or center console) 810 are shown. Figure 2 In the case where the perception module 206 has a low driver goal confidence, the perception module 206 can initiate a dialogue by displaying icons, buttons, and / or other items and / or forms. The driver can then touch the displayed items to respond and continue the dialogue. The dialogue module 216 can then continue with voice, text, and / or other interaction methods.

[0150] Fig. 9 A logical flow chart illustrating the active interaction assistance method is shown. Figure 2 The method is described in the embodiment of the present invention, but the method is applicable to Figure 1-8 The operations of the method may be performed iteratively.

[0151] At 900, information from the navigation module 114 and sensor data from the sensor 150 are collected, as described above. The following operations 902, 904, 906 may be performed simultaneously (or in parallel).

[0152] At 902, the external analysis module 200 detects traffic objects and determines road information based on data received from the external sensors 152. At 904, the occupant module 202 tracks and monitors occupant behavior and receives occupant input, such as voice input, based on data received from the internal sensors 154. At 906, the vehicle state module 204 collects data from vehicle state (or state) sensors, such as from the sensors 156.

[0153] At 908 , the perception module 206 determines perception information and optionally a situation category for a potential interactive session with the occupant based on the information and data determined and collected during operations 900 , 902 , 904 , and 906 .

[0154] At 910 , the data collection and interaction target module 208 collects the sensory data and the related historical data and determines the interaction target based thereon.

[0155] At 912 , the perception data, the historical data, and the interaction goals can be received at the interaction style module 212 , based on which the interaction style module 212 can determine an interaction style.

[0156] At 914, an interaction timing module (such as any interaction timing module mentioned herein) receives the sensory data, historical data, and interaction goals, and determines interaction timing based thereon. The interaction timing may also be determined based on: i) a situation category, ii) one or more event trigger and gap pairs, iii) rules, and / or iv) a machine learning algorithm. As described above, one or more trigger and gap pairs may be determined based on the sensory data, historical data, and interaction goals.

[0157] At 916 , the timed interaction module may also determine whether the timing for the interactive session with the vehicle occupant is appropriate.

[0158] At 918 , the timed interaction module may determine that the timing is inappropriate and avoid interacting with the vehicle occupant.

[0159] At 920, the timed interaction module may determine that the timing is appropriate, and the dialog module 216 may initiate an interactive dialog with the vehicle occupant, including providing suggestions and / or offers to the vehicle occupant. This may include conducting a voice, text, and / or haptic-based dialog and / or displaying icons, buttons, or other objects for the vehicle occupant to select.

[0160] At 922 , the dialogue module 216 can determine whether the vehicle occupant has accepted the suggestion and / or offer. If so, operation 924 can be performed, otherwise the method can end.

[0161] At 924, the dialogue module 216 may perform assistance operations, including optionally performing operations associated with the service, optionally performing navigation assistance, optionally performing autonomous driving assistance, and / or providing other assistance as described above. If navigation assistance is to be provided, the vehicle control module 103 may provide instructions to the driver to drive the master vehicle to a refueling station, a charging station, and / or other selected destinations. If autonomous driving assistance is to be provided, the vehicle control module 103 may control the vehicle system to drive the master vehicle to a refueling station, a charging station, and / or other selected destinations.

[0162] The examples described herein include systems for timing and style selection of active voice interactions. The system includes microphones and / or microphone arrays, an interior camera as part of a driver monitoring system (DMS), an exterior camera, other external sensors, and a module for reading information from a vehicle bus (such as a CAN bus). A perception module is included for the purpose of situation understanding, which combines information collected from input sources to create a coherent situation and context description. The dialogue module selects the style, type, and timing of the interactive dialogue, initiates the interactive dialogue at the appropriate time, and completes the interactive dialogue.

[0163] Examples include adaptation and personalization. These examples form a closed loop and understand whether the proactive conversation timing and style is successful. False alarms are minimized and / or prevented. A false alarm refers to a proactive conversation initiated by the virtual assistance system at a specific timing and cancelled by the vehicle occupant. This cancellation can be done immediately after initiation, or at some time after initiation. False detections are also minimized and / or prevented. False detections refer to conversations initiated by the occupant. The virtual assistant system did not predict that a conversation should occur when the occupant initiated the conversation.

[0164] Adaptation and personalization are defined in the framework of reinforcement learning (RL). In this case, success is defined based on whether the driver ignored the entire interaction, whether the driver performed the suggested action, whether the driver accepted the suggestion and / or proposal, whether the driver completed the interactive dialogue, etc. Such information can be aggregated and specific to a certain driver, a certain driver performing a certain task, a collection of drivers (called crowdsourcing), a collection of drivers performing a certain task, etc. Different levels of aggregation can be implemented. Rules can be generated for a specific driver or a collection of drivers. Rules can be generated for a specific geographic location and / or for some other aspect. Event triggers and gap pairs can be selected based on the aggregation used and / or the rules generated.

[0165] In one embodiment, the operations described herein may be implemented so that the driver perceives that control of the primary vehicle is maintained. Some drivers prefer to maintain control of the vehicle. Therefore, the more operations the vehicle performs autonomously, the less control the driver feels over the vehicle. Performing active operations may be perceived as taking over some control from the driver. Therefore, active interference and active interactive dialogues may be limited. For example, the number of interactive dialogues per vehicle trip, per hour, per day, etc. may be limited to a predetermined number (e.g., 1-5). Therefore, the examples herein address the gap between an active system that starts on its own and the concept of maintaining driver control of the primary vehicle. In order to address the problem of taking over control from the driver using an active system, a signal that a voice interaction is about to begin and / or a signal similar to Figure 8 An override button 804 of the button can be displayed on one of the displays mentioned herein. The location of the override button allows the driver to easily touch it to terminate the current interaction. In addition, the vehicle can be configured to request a certain amount of desired active voice interaction from the driver. For example, the virtual assistant system can ask the driver whether the driver prefers no active interaction, no more than once a day, X times a day or a driving event, and / or whether the driver wants to participate in the test version evaluation.

[0166] The foregoing description is essentially merely illustrative and is in no way intended to limit the present disclosure, its application or use. The broad teachings of the present disclosure can be implemented in a variety of forms. Therefore, although the present disclosure includes specific examples, the true scope of the present disclosure should not be so limited, because other modifications will become apparent after studying the drawings, descriptions and appended claims. It should be understood that one or more steps in the method can be performed in different orders (or simultaneously) without changing the principles of the present disclosure. In addition, although each embodiment is described as having specific features above, any one or more of those features described with respect to any embodiment of the present disclosure can be implemented in any other embodiment and / or combined with the features of any other embodiment, even if the combination is not explicitly described. In other words, the described embodiments are not mutually exclusive, and the arrangement of one or more embodiments to each other is still within the scope of the present disclosure.

[0167] The spatial and functional relationships between elements (e.g., between modules, circuit elements, semiconductor layers, etc.) are described using various terms, including "connected," "engaged," "coupled," "adjacent," "next to," "on top of," "above," "below," and "disposed." Unless explicitly described as "directly," when describing the relationship between a first element and a second element in the above disclosure, the relationship can be a direct relationship, in which there are no other intermediate elements between the first element and the second element, but can also be an indirect relationship, in which there are one or more intermediate elements (spatially or functionally) between the first and second elements. As used herein, the phrase "at least one of A, B, and C" should be interpreted to mean a logical (A or B or C), using a non-exclusive logical "or," and should not be interpreted to mean "at least one of A, at least one of B, and at least one of C."

[0168] In the drawings, the direction of the arrows, as indicated by the arrows, generally demonstrates the flow of information (e.g., data or instructions) of interest to the illustration. For example, when component A and component B exchange various information, but the information transmitted from component A to component B is relevant to the illustration, the arrow may be directed from component A to component B. This unidirectional arrow does not mean that no other information is transmitted from component B to component A. In addition, for information sent from component A to component B, component B may send a request or a receipt confirmation of the information to component A.

[0169] In this application, including the definitions below, the term "module" or the term "controller" may be replaced with the term "circuit". The term "module" may refer to, be part of, or include: an application specific integrated circuit (ASIC); a digital, analog, or mixed analog / digital discrete circuit; a digital, analog, or mixed analog / digital integrated circuit; a combinational logic circuit; a field programmable gate array (FPGA); a processor circuit (shared, dedicated, or group) that executes code; a memory circuit (shared, dedicated, or group) that stores code executed by the processor circuit; other suitable hardware components that provide the described functionality; or a combination of some or all of the above, such as in a system on a chip.

[0170] The module may include one or more interface circuits. In some examples, the interface circuit may include a wired or wireless interface connected to a local area network (LAN), the Internet, a wide area network (WAN), or a combination thereof. The functions of any given module of the present disclosure may be distributed between multiple modules connected via the interface circuit. For example, multiple modules may allow load balancing. In a further example, a server (also referred to as a remote or cloud) module may perform some functions on behalf of a client module.

[0171] The term "code" as used above may include software, firmware, and / or microcode, and may refer to a program, a routine, a function, a class, a data structure, and / or an object. The term "shared processor circuit" encompasses a single processor circuit that executes some or all of the code from multiple modules. The term "group processor circuit" encompasses a processor circuit that executes some or all of the code from one or more modules in combination with an additional processor circuit. References to multiple processor circuits encompass multiple processor circuits on discrete dies, multiple processor circuits on a single die, multiple cores of a single processor circuit, multiple threads of a single processor circuit, or a combination of the above. The term "shared memory circuit" encompasses a single memory circuit that stores some or all of the code from multiple modules. The term "group memory circuit" encompasses a memory circuit that stores some or all of the code from one or more modules in combination with additional memory.

[0172] The term "memory circuit" is a subset of the term "computer-readable medium". As used herein, the term "computer-readable medium" does not cover transient electrical or electromagnetic signals propagated through a medium (such as on a carrier wave); therefore, the term "computer-readable medium" may be considered to be tangible and non-transitory. Non-limiting examples of non-transitory, tangible computer-readable media are non-volatile memory circuits (such as flash memory circuits, erasable programmable read-only memory circuits, or mask read-only memory circuits), volatile memory circuits (such as static random access memory circuits or dynamic random access memory circuits), magnetic storage media (such as analog or digital magnetic tape or hard disk drives), and optical storage media (such as CDs, DVDs, or Blu-ray discs).

[0173] The apparatus and methods described in this application may be implemented in part or in whole by a special purpose computer created by configuring a general purpose computer to perform one or more specific functions embodied in a computer program. The above-mentioned function blocks, flow chart components and other elements serve as software specifications, which can be translated into a computer program by routine work of a skilled technician or programmer.

[0174] The computer program includes processor executable instructions stored on at least one non-transitory tangible computer readable medium. The computer program may also include or rely on stored data. The computer program may include a basic input / output system (BIOS) that interacts with the hardware of the special-purpose computer, device drivers that interact with specific devices of the special-purpose computer, one or more operating systems, user applications, background services, background applications, etc.

[0175] A computer program may include: (i) descriptive text to be parsed, such as HTML (Hypertext Markup Language), XML (Extensible Markup Language), or JSON (JavaScript Object Notation), (ii) assembly code, (iii) object code generated by a compiler from source code, (iv) source code for execution by an interpreter, (v) source code for compilation and execution by a just-in-time compiler, etc. By way of example only, source code may be written using syntax from a language including: C, C++, C#, Objective C, Swift, Haskell, Go, SQL, R, Lisp, Fortran, Perl, Pascal, Curl, OCaml, HTML5 (Hypertext Markup Language Fifth Edition), Ada, ASP (Active Server Pages), PHP (PHP: Hypertext Preprocessor), Scala, Eiffel, Smalltalk, Erlang, Ruby, Lua, MATLAB, SIMULINK, and

Claims

1. An auxiliary system, comprising: a perception module configured to: collect sensor data, the sensor data including data tracking behavior of vehicle occupants in a host vehicle; and determining perception information describing a current situation that warrants initiation of an interactive dialog with an occupant of the vehicle; an interaction target module, configured to determine an interaction target based on the determined perception information; An interaction timing module, configured to determine a timing for initiating the interaction dialogue based on the interaction target; a dialogue module configured to initiate the interactive dialogue based on the perception information and the timing to provide at least one of a suggestion and a proposal to the vehicle occupant; as well as A vehicle control module is configured to control operation of at least one of a device and a system of the host vehicle in response to the vehicle occupant accepting the at least one of a suggestion and a proposal.

2. The assistance system according to claim 1, further comprising: an external analysis module configured to collect data from a plurality of external sensors; an occupant module configured to collect data from a plurality of internal sensors; and a vehicle status module configured to collect data from a plurality of vehicle status sensors, The perception module is configured to determine the perception information based on data collected from the multiple external sensors, data collected from the multiple internal sensors, and data collected from the multiple vehicle status sensors. 3 . The assistance system of claim 1 , wherein the perception module determines the perception information based on navigation information received from a navigation module.

4. The assistance system according to claim 1, further comprising: an interaction style module configured to select an interaction style from the group consisting of speech, text, and haptic vibration to communicate with the vehicle occupant during the interaction session, The dialog module is configured to communicate with the vehicle occupant using the interaction style during the interactive dialog. 5 . The assistance system of claim 4 , wherein the interaction style module is configured to select an interaction style based on historical data indicating which interaction styles are likely to be accepted by the vehicle occupant. 6 . The assistance system according to claim 1 , wherein the interaction timing module is configured to determine at least one event trigger and gap pair based on which the interaction dialog is initiated, and the at least one event trigger and gap pair is determined based on the interaction target. 7 . The assistance system according to claim 1 , wherein the interaction timing module is configured to determine a plurality of event triggers and gap pairs based on which the interaction dialog is initiated, the plurality of event triggers and gap pairs being determined based on the interaction target.

8. The assistance system according to claim 1, wherein: The perception module is configured to determine a situation category and a corresponding confidence level; and The interaction timing module is configured to determine a timing for initiating the interaction dialog based on the situation category and the confidence level.

9. The assistance system according to claim 1, wherein: The interaction timing module is configured to: i) determine a selection criterion, select at least one event trigger and gap pair based on the selection criterion, and ii) select the at least one event trigger and gap from a set of event trigger and gap pairs, wherein the selection criterion includes determining at least one of the following: i) whether a percentage of at least one of acceptance and interaction dialog completion for the selected at least one event trigger and gap pair is greater than a threshold, and ii) the percentage of at least one of acceptance and interaction dialog completion for the selected at least one event trigger and gap pair is greater than a percentage of acceptance and interaction dialog completion for other event trigger and gap pairs in the set of event trigger and gap pairs; and At least one of the interaction timing module and the dialog module initiates the interaction dialog based on the selected at least one event trigger and gap pair.

10. The assistance system according to claim 1, wherein: The timed interaction module is configured to generate a histogram based on historical data including a plurality of slots triggered for at least one event, and select at least one of the plurality of slots based on the histogram; and At least one of the timing module and the dialog module initiates the interactive dialog based on the selected at least one of the plurality of slots.

Images