Dynamic voice assistant system for vehicle

Abstract

A voice assistant system for a vehicle includes a microphone configured to detect an audio signal from a user of the vehicle; a speaker configured to output a conversation in response to the audio signal; and a processor programmed to, in response to detecting a session involved by the user, reduce a length setting of the voice assistant system to reduce a length of the session, and increase an independence setting of the voice assistant system to prevent a confirmation problem from the voice assistant system.

Description

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS

[0002] This application claims priority to U.S. Application No. 17 / 848,598, filed on June 24, 2022, the disclosure of which is incorporated herein by reference in its entirety. Technical Field

[0003] The present disclosure generally relates to a dynamic voice assistant system for a vehicle. More specifically, the present disclosure relates to a vehicle voice assistant system that can be dynamically adjusted and customized. Background Art

[0004] Voice assistant systems have been increasingly used in vehicles to perform various tasks by interacting with one or more vehicle users. Different vehicle users may prefer different human-like personalities during interactions. Personality assignments can affect not only the sound and tone of the system's voice, but also the content of the speed at which the system interacts with the user. For example, the system can adjust decision making, what information is provided, and how the information is presented to the user based on different personality assignments. Summary of the invention

[0005] In one or more illustrative examples of the present disclosure, a voice assistant system for a vehicle includes: a microphone configured to detect an audio signal from a user of the vehicle; a speaker configured to output a conversation in response to the audio signal; and a processor programmed to, in response to detecting a conversation involving the user, reduce a length setting of the voice assistant system to reduce the length of the conversation and increase an independence setting of the voice assistant system to prevent confirmation issues from the voice assistant system.

[0006] In one or more illustrative examples of the present disclosure, a method for a voice assistant system for a vehicle includes: detecting an utterance from one of multiple users of the vehicle via a microphone; identifying one of the multiple users who made the utterance; determining a user characteristic associated with the one user; and adjusting multiple voice assistant settings between at least one lower boundary and at least one upper boundary based on the user characteristic.

[0007] In one or more illustrative examples of the present disclosure, a non-transitory computer-readable medium includes instructions that, when executed by a processor of a vehicle, cause the vehicle to: detect, via a microphone, speech from one of a plurality of users of the vehicle;

[0008] A processor identifies a user among multiple users who makes an utterance; the processor determines a user characteristic associated with the user; the processor adjusts multiple voice assistant settings between at least one lower boundary and at least one upper boundary based on the user characteristic; and outputs a conversation based on the multiple voice assistant settings via a speaker. BRIEF DESCRIPTION OF THE DRAWINGS

[0009] FIG. 1 illustrates a block diagram of an in-vehicle audio and voice assistant system in an automotive application having a processing system according to one embodiment;

[0010] FIG2 shows a block architecture diagram for a customizable audio and voice assistant system according to one embodiment;

[0011] 3A and 3B show schematic diagrams of a voice assistant persona interface according to one embodiment;

[0012] 4A and 4B show schematic diagrams of a voice assistant role interface according to another embodiment;

[0013] FIG5 illustrates a flow chart of an example process for adjusting a role of a vehicle assistant system, according to one embodiment.

[0014] FIG6 illustrates a flow chart of an example process for adjusting a role of a vehicle assistant system according to another embodiment.

[0015] FIG. 7 illustrates a flow chart of an example process for adjusting a role of a vehicle assistant system according to yet another embodiment. DETAILED DESCRIPTION

[0016] As required, detailed embodiments of the present invention are disclosed herein; however, it should be understood that the disclosed embodiments are merely examples of the present invention that can be implemented in various and alternative forms. The drawings are not necessarily drawn to scale; some features may be exaggerated or minimized to show details of particular components. Therefore, the specific structural and functional details disclosed herein should not be interpreted as limiting, but merely as a representative basis for teaching those skilled in the art to use the present invention in various ways.

[0017] FIG1 shows a block diagram of a car voice assistant system 100 with a multi-modal input processing system according to one embodiment. The car voice assistant system 100 may be designed for a vehicle 104 configured to transport passengers. The vehicle 104 may include various types of passenger vehicles, such as a crossover utility vehicle (CUV), a sport utility vehicle (SUV), a truck, a recreational vehicle (RV), a boat, an airplane, or other mobile machines for transporting people or goods. In addition, the vehicle 104 may be an autonomous, partially autonomous, self-driving, unmanned, or driver-assisted vehicle. The vehicle 104 may be an electric vehicle (EV), such as a battery electric vehicle (BEV), a plug-in hybrid electric vehicle (PHEV), a hybrid electric vehicle (HEV), and the like.

[0018] The vehicle 104 may be configured to include various types of components, processors, and memory, and may communicate with a communication network 110. The communication network 110 may be referred to as a "cloud" and may involve data transmission via a wide area network and / or a local area network such as the Internet, a global positioning system (GPS), a cellular network, Wi-Fi, Bluetooth, etc. The communication network 110 may provide communication between the vehicle 104 and external or remote servers 112 and / or databases 114 and other external applications, systems, vehicles, etc. The communication network 110 may provide navigation, music or other audio, program content, marketing content, Internet access, speech recognition, cognitive computing, artificial intelligence, etc. to the vehicle 104.

[0019] The remote server 112 and database 114 may include one or more computer hardware processors coupled to one or more computer storage devices for performing the steps of one or more methods as described herein, and may enable the vehicle 104 to communicate and exchange information and data with systems and subsystems external to the vehicle 104 and local or onboard the vehicle 104. The vehicle 104 may include one or more processors 120 configured to execute certain instructions, commands, and other routines as described herein. An internal vehicle network 122 may also be included, such as a vehicle controller area network (CAN), an Ethernet network, and a media-oriented systems transport (MOST). The internal vehicle network 122 may allow the processor 120 to communicate with other vehicle 104 systems (such as a vehicle modem, a GPS module and / or a global system for mobile communications (GSM) module configured to provide current vehicle position and heading information, and various vehicle electronic control units (ECUs) configured to cooperate with the processor 120).

[0020] The processor 120 may execute instructions for certain vehicle applications, including navigation, infotainment, climate control, etc. Various types of computer-readable storage media 124 may be used to maintain instructions for the respective vehicle systems in a non-volatile manner. The computer-readable storage media 124 (also referred to herein as memory 124 or storage) includes any non-transitory medium (e.g., tangible medium) that participates in providing instructions or other data that may be read by the processor 120. Computer-executable instructions may be compiled or interpreted from a computer program created using a variety of programming languages ​​and / or techniques, including, alone or in combination, but not limited to Java, C, C++, C#, Objective C, Fortran, Pascal, JavaScript, Python, Perl, and Structured Query Language (SQL).

[0021] The processor 120 may also be part of a multi-mode processing system 130. The multi-mode processing system 130 may include various vehicle components, such as the processor 120, memory 124, sensors 132, input / output devices 134, displays 136, and the like. The multi-mode processing system 130 may include one or more input and output devices 134 for exchanging data processed by the multi-mode processing system 130 with other elements shown in FIG. 1 via a human-machine interface (HMI) control (not shown). Some examples of these processes may include navigation system output (e.g., time-sensitive directions for the driver), incoming text messages converted to output speech, vehicle status output, and the like, such as output from a local or onboard storage medium or system. In some embodiments, the multi-mode processing system 130 provides input / output control functionality with respect to one or more electronic devices, such as a heads-up display (HUD) 136, a vehicle display 136, and / or a driver or passenger's mobile device 138, sensors 132, cameras 132, and the like. The multi-modal processing system 130 includes a dynamic adjustment function that is configured to adjust the personality of the automotive voice assistant system 100 based on user behavior, vehicle occupancy, and / or external environment detected by the vehicle sensors 132, as described in more detail below.

[0022] The vehicle 104 may include a wireless transceiver 140 configured to be compatible with various user devices and to communicate with the communication network 110, such as a Bluetooth module, a ZIGBEE transceiver, a Wi-Fi transceiver, an IrDA transceiver, a radio frequency identification (RFID) transceiver, an ultra-wideband (UWB), etc.

[0023] The vehicle 104 may include various sensors and input devices as part of the multi-mode processing system 130. For example, the vehicle 104 may include at least one microphone 142. The microphone 142 may be configured to receive audio signals from within the vehicle cabin, such as acoustic utterances including spoken words, phrases, or commands from a user. The microphone 142 may include an audio input configured to provide audio signal processing functions to the processor 120, including amplification, conversion, data processing, etc. The vehicle 104 may include at least one microphone 142 arranged throughout the vehicle 104. Although the microphone 142 is described herein as being used for the purpose of the multi-mode processing system 130, the microphone 142 may be used for other vehicle functions, such as active noise cancellation, hands-free interface, etc. The microphone 142 may facilitate speech recognition from the audio received via the microphone 142 based on grammatical analysis associated with available commands and voice prompt generation. The microphone 142 may include multiple microphones 142 arranged throughout the vehicle cabin.

[0024] Microphone 142 can be configured to receive audio signals from the vehicle cabin. These audio signals can include occupant speech, sound, etc. Processor 120 can receive these audio signals to determine the number of occupants in the vehicle. For example, processor 120 can detect various voices via timbre, pitch, frequency, etc., and determine that more than one occupant is in the vehicle at a given time. Based on the audio signal and various frequencies, etc., processor 120 can determine the number of occupants. Based on this, processor 120 can adjust certain settings related to the car voice assistant system 100. This is described in more detail below. Additionally or alternatively, in the case where multiple microphones 142 are provided throughout the vehicle cabin, multiple microphones 142 can achieve omnidirectional measurement of voice commands issued by occupants at different seat positions in the vehicle cabin, so that the identity of the occupant who issued the voice command can be more accurately determined based on the identity of the user at different seat positions.

[0025] The microphone 142 may also be used to identify occupants via direct recognition (eg, spoken name) or through voice recognition performed by the processor 120. The microphone may also be configured to receive non-occupancy related data (such as spoken utterances, etc.).

[0026] The sensor 132 may include at least one camera configured to provide facial recognition of (one or more) occupants. The camera 132 may also be configured to detect non-verbal cues about the driver's behavior (such as the user's gaze direction, user gestures, etc.). The camera 132 may monitor the driver's head position, as well as detect any other movement of the user (such as the movement of the user's arm or hand, the shaking of the user's head, etc.). In the example of the camera 132, the camera may provide imaging data taken of the user to indicate certain movements made by the user. The camera 132 may be a camera capable of taking still images as well as videos and detecting the movement of the user's head, eyes, and body. The camera 132 may include multiple cameras, and the imaging data may be used for qualitative analysis. For example, the imaging data may be used to determine whether the user is looking at a certain location or vehicle display 136. Additionally or alternatively, the imaging data may also be supplemented with timing information because it is related to user movement or gestures. In addition to the voice data detected via the microphone 142, the imaging data may also be used to determine the user's emotions. Based on the user's emotions, the processor 120 may adjust certain settings associated with the car voice assistant system 100.

[0027] The vehicle 104 may include an audio system having audio playback capabilities through the vehicle speakers 144 or headphones. The audio playback may include audio from sources such as a vehicle radio, including satellite radio, decoded amplitude modulation (AM) or frequency modulation (FM) radio signals, audio playback from a compact disc (CD) or digital versatile disc (DVD), streaming audio from a mobile device, commands from a navigation system, voice output from the car voice assistant system 100, etc.

[0028] As explained, the vehicle 104 may include various displays and user interfaces, including a HUD, a center console display, steering wheel buttons, etc. A touch screen may be configured to receive user input. A visual display 136 may be configured to provide visual output to a user.

[0029] The sensors 132 may also include other sensors configured to detect and measure data not described above. The sensors 152 may include one or more sensors in addition to the microphone 132, the data provided by which may be used to help detect occupancy, such as pressure sensors in the vehicle seat, door sensors, cameras, etc. The occupant data from these sensors may be used in conjunction with the audio signal to determine occupancy, including the number of occupants.

[0030] Although not specifically shown herein, the vehicle 104 may include many other systems, such as a GPS system, human-machine interface (HMI) controls, a video system, etc. The multi-mode processing system 130 may use inputs from various vehicle systems, including the speaker 144 and the sensor 132. For example, the multi-mode processing system 130 may determine whether a user's utterance is system-directed (SD) or non-system-directed (NSD). SD utterances may be made by a user that are intended to affect outputs within the vehicle 104, such as a spoken command to "turn on the music." NSD utterances may be utterances spoken during a conversation with another occupant, on the phone, or when speaking to someone outside the vehicle. These NSDs are not intended to affect vehicle outputs or systems. NSDs may be person-to-person conversations.

[0031] Although automotive systems are discussed in detail here, other applications are contemplated. For example, similar functionality can also be applied to other non-automotive use cases, such as for augmented reality or virtual reality use cases with smart glasses, phones, eye trackers, etc. in a living environment. Although the term "user" is used throughout, this term is interchangeable with other terms such as speaker, occupant, driver, etc.

[0032] Referring to FIG. 2 , an example block architecture diagram of a voice assistant customization system 200 is illustrated according to one embodiment. The system 200 may allow a vehicle designer / engineer to customize and adjust voice assistant functionality before the functionality is implemented on the vehicle 104 of FIG. 1 . Additionally, the system 200 may allow a designer to modify the global role / personality configuration of the voice assistant functionality after installation on the vehicle 104 via one or more communication components (such as via the wireless transceiver 140 of the multi-mode system 130 of the vehicle 104 ). As an example, the voice assistant customization system 200 may be implemented via the remote server 112 in conjunction with the database 114 .

[0033] The voice assistant customization system 200 may include a dialogue machine 202 configured to manage the structure of the conversation. The occupants of the vehicle 104 may participate in certain conversations with the vehicle 104 and each other. The dialogue machine 202 may be configured to control the structure of the conversation in a manner that ignores location or customer information. That is, the dialogue machine 202 may be configured to assemble fragments of the dialogue structure to facilitate prompt construction, regardless of the specific environment and user identity. The voice assistant customization system 200 may also include a natural language generation (NLG) library 204, which is configured to store and list possible expressions of variables and / or concepts. The NLG library 204 may also be configured to map possible expressions to different roles in various areas. The voice assistant customization system 200 may also include a global role setting component 206, which is configured to allow vehicle designers to customize and adjust the global role settings (also referred to as personality settings) of one or more vehicle models based on the desired personality in each corresponding place and use case. The global role setting component 206 can be implemented in various ways. For example, the global role setting component 206 can be implemented using a computing hardware device provided with processing power in combination with software. The details of the global role setup component 206 will be described in detail below. The voice assistant customization system 200 may also include a prompt generator 208, which is configured to generate one or more prompts as voice output based on the configuration from the dialogue machine 202, the NLG library 204, and the global role setup component 206 to allow the multimodal system 130 of the vehicle 104 to participate in a conversation with one or more vehicle users. The prompt generator 208 can use artificial intelligence (AI) and historical system setting data to generate prompts / dialogues. The vehicle designer can adjust the configuration of the global role setup component 206 based on the prompt output from the prompt generator 208.

[0034] Referring to FIG. 3A , a schematic diagram of an interface of a global character setting component 206 according to one embodiment is shown. Continuing to refer to FIG. 1 and FIG. 2 , the global character setting component 206 in this example can be configured to allow a vehicle designer to customize and adjust character settings, including formality 302, length 304, and independence 306. As shown in FIG. 3A , each setting can correspond to a value that can be adjusted via a slider with a predefined range, and can affect a command or output from the vehicle speaker 144 in response to a spoken word from a user. The formality setting 302 can correspond to a formality slider 312 that is adjustable between a minimum amplitude value of zero indicating a casual character and a maximum amplitude value of ten indicating a formal character. The formality setting 302 can be associated with a conversational formality style of the voice assistant function of the vehicle multi-modal system 130. The vehicle designer can adjust the formality value by sliding the slider 312 left and right. When the formality value is low, the voice assistant can be configured to perform speech in a more casual style. For example, in response to the vehicle user setting a navigation destination, the voice assistant can output “Sure! I’ll get you there as soon as possible.” When the formality value is high, the voice assistant can be configured to perform the speech in a more formal style. Taking the same example above, the voice assistant can confirm the destination by outputting "Got it. I will navigate you to your destination by the fastest route."

[0035] The length setting 304 may correspond to a length slider 314 that is adjustable between a minimum value of zero, indicating a brief speech, and a maximum value of ten, indicating a lengthy speech. The length setting 304 may be associated with a text style of speech for the voice assistant function of the vehicle multi-modal system 130. When the length value is low, the voice assistant may be configured to perform a shorter speech. Continuing with the above navigation example, the voice assistant may simply output "OK!" to confirm the navigation destination. When the length value is high, the voice assistant may be configured to perform a longer speech. For example, the voice assistant may output "OK! The navigation destination has been successfully set" in the confirmation.

[0036] The independence setting 306 may correspond to an independence slider 316 that is adjustable between a minimum value of zero indicating a dependent role and a maximum value of ten indicating an independent role. The independence setting 306 may be associated with a degree of independence with which the voice assistant can perform a task without requiring user input. When the independence value is low, the voice assistant of the multimodal system 130 may be configured to be more dependent on the user before beginning to perform a task. For example, in response to receiving a navigation destination, the voice assistant may request user confirmation before beginning the navigation route, such as “Please confirm that you want me to start navigating.” Conversely, when the independence value is high, the voice assistant may be configured to be more like an agent of the vehicle user who performs tasks more independently. Continuing with the above navigation example, in response to receiving a destination, the voice assistant may output “OK, start navigating.” And automatically begin navigation without seeking confirmation from the user.

[0037] In the embodiment shown in reference figure 3A, a single slider corresponds to a single value as a global setting for one or more vehicles. For example, the formality slider 312 has a value of two tenths, the length slider 314 has a value of three tenths, and the independence slider 316 has a value of eight tenths as a global setting for the vehicle 104. By default, a predefined range can be provided for each setting. In this way, the multi-mode system 130 of the vehicle 104 can be configured to automatically adjust the setting to within the range. For example, the global role setting component 206 can be pre-configured to assign a + / -1 range change based on the value set by the vehicle designer. Taking the formality setting as an example, when the vehicle designer sets the formality value to two via the formality slider 312, the actual value range of the voice assistant of the vehicle multi-mode system 130 can be between one and three adjusted based on the specific usage scenario of the vehicle 104 (to be discussed in detail below). Additionally, the upper and lower limits of the value selected by the designer can be different (e.g., the lower boundary is -1 and the upper boundary is +2). Additionally, different ranges can be assigned to different settings. For example, while the formality setting 302 may be configured to allow a + / 1 variation, the length setting 304 may be configured to allow a + / - 3 variation. This may allow for more customizable and flexible settings for each attribute.

[0038] Although a slider is used to adjust the character settings in this example, the present disclosure is not limited thereto, and other components for adjusting the value of a setting may be used under substantially the same concept. For example, other gradient-like settings (such as pie charts, bars, scatter points, etc.) may be used to allow the user to adjust the settings. In another example, the interface may allow the vehicle designer to manually enter the value corresponding to each setting. It should also be noted that although a magnitude of zero to ten is used for all three character setting values ​​in this example, as shown in reference FIG. 3A , the present disclosure is not limited thereto. Different magnitude quantizations may be assigned to different character settings.

[0039] Referring to FIG3B , a schematic diagram of an interface of a global role setting component 206 according to another embodiment is shown. Different from the example shown in reference FIG3A , the global role setting component 206 in this example can be configured to allow a vehicle designer to manually set a global range for each role setting. The formality setting 302 can correspond to a formality slider 322 that is adjustable between a minimum value of zero indicating a casual role and a maximum value of ten indicating a formal role. The formality slider 322 can be associated with a lower slider 332 that defines a lower boundary of the global formality setting 302 and an upper slider 334 that defines an upper boundary of the global formality setting 302. The lower boundary and the upper boundary define a global formality value range 336, within which the vehicle 104 is allowed to dynamically adjust the formality setting 302. The vehicle designer can manually customize the formality range 336 by adjusting the lower slider 332 and / or the upper slider 334.

[0040] The length setting 304 may correspond to a length slider 322 that is adjustable between a minimum value of zero indicating brief speech and a maximum value of ten indicating lengthy speech. The length slider 324 may be associated with a lower slider 342 that defines a lower boundary of the global length setting 304 and an upper slider 344 that defines an upper boundary of the global length setting 304. The lower boundary and the upper boundary define a global formality value range 346 within which the vehicle 104 is allowed to dynamically adjust the length setting 304. A vehicle designer may manually customize the length range 346 by adjusting the lower slider 342 and / or the upper slider 344.

[0041] The independence setting 306 may correspond to an independence slider 326 that is adjustable between a minimum value of zero indicating a dependent role and a maximum value of ten indicating an independent role. The independence slider 326 may be associated with a lower slider 352 that defines a lower boundary of the global independence setting 304 and an upper slider 354 that defines an upper boundary of the global independence setting 304. The lower boundary and the upper boundary define a global formality value range 356 within which the vehicle 104 is allowed to dynamically adjust the independence setting 306. A vehicle designer may manually customize the independence range 356 by adjusting the lower slider 352 and / or the upper slider 354.

[0042] Referring to FIG. 4A and FIG. 4B , an interface diagram 400 of the global role setting component 206 according to yet another embodiment is shown. This may be an example interface presented to a user via a display 136 or a mobile device, but may actually be an interface provided to a vehicle designer as a visual way to understand settings or example users. The role setting interface 400 may include a personality portion 402 configured to adjust the role of the voice assistant, a parameter portion 404 configured to receive parameters (such as questions), and a prototype portion configured to output a conversation. Similar to the embodiment shown in FIG. 3A and FIG. 3B , a vehicle designer may adjust the role settings of the voice assistant via the personality portion 402 shown as a graphical user interface (GUI). In this example, the role settings may include an independence setting 412 associated with the degree of independence of the voice assistant feature of the vehicle multimodal system 130. The personality settings may also include a formality setting 414 associated with the formality style of the voice of the voice assistant. The personality settings may also include a length setting 416 associated with the conversation length style of the voice assistant. The personality settings may also include an initiative setting 418 associated with the initiative degree of the voice assistant. The personality settings may also include a specificity setting 420 associated with the conversational specificity style of the voice assistant. The personality settings may also include a humor setting 422 associated with the conversational humor style of the voice assistant.

[0043] As shown in Figures 4A and 4B, in this embodiment, the personality section is presented in the form of a radar chart (also known as a spider web chart) to present multiple setting variables at the same time. The radar chart may include multiple spokes, each spoke representing the current setting value of the corresponding setting. The data length of each spoke may be proportional to the amplitude of the data value corresponding to the corresponding setting. In this embodiment, each setting spoke can be adjusted between a minimum amplitude of zero and a maximum amplitude of five via a slider along each corresponding spoke. For example, a vehicle designer can reduce the independence setting 412 of the voice assistant from four as shown in Figure 4A to one as shown in Figure 4B by sliding the slider on the corresponding spoke downward. As discussed above with reference to Figures 3A and 3B, when the independence setting is associated with a higher value (e.g., a value of 4), the voice assistant can act in a more independent manner.

[0044] Continuing with the above navigation example, in response to the vehicle designer inputting “Navigate to the Science Center,” the voice assistant may directly respond via the prototype portion 406 with “OK. I will begin navigating to the Science Center on Atomic Avenue,” without requiring confirmation from the vehicle designer. The prototype portion 406 may be associated with the prompt generator 208 as part of the voice assistant customization system 200. Here, the above answer from the voice assistant may be one of a plurality of candidate variations 430 corresponding to the current global character setting at the personality portion 402, and may be configured via the parameter portion 404. In this example, two candidate variations 430 are presented to the vehicle designer via the parameter portion 404, and the vehicle designer may manually select from the variations 430. Additionally, the parameter portion 404 may also be configured to allow the vehicle designer to add more candidate variations in response to user input corresponding to the global character setting configured at the personality portion 402. For example, the vehicle designer may manually type an answer via the parameter portion 404 as an additional candidate variation. Additionally, the vehicle designer may remove one or more candidate variations from parameters section 404 such that the removed variations may no longer be considered for corresponding global character settings configured via personality section 402 .

[0045] As shown in FIG4B , in response to the independence setting 412 being reduced from four to one, the voice assistant may act in a more dependent manner and seek user confirmation before continuing to perform the assigned task. Continuing with the navigation example above, the voice assistant may output a confirmation dialog such as “Do you want me to navigate you to the Science Center on Atomic Avenue?” Similarly, the vehicle designer may add, remove, and modify candidate variations via the parameters section 404.

[0046] Although zero and five are used in this example to define the range of value settings, it should be noted that the present disclosure is not limited to this, and other amplitude quantization mechanisms can be used for personality portion 402 under substantially the same concept. It should also be noted that the radar chart presented in personality portion 402 may be only an example, and the present disclosure is not limited to this. Role settings can be configured in substantially the same manner via the personality portion in other GUI or non-GUI ways. It should also be noted that although in the embodiments shown in reference figures 4A and 4B, each role setting is associated with only a single value, the present disclosure is not limited to this. Various global role settings can be configured to have a range of minimum and maximum boundaries similar to the examples explained above with reference figures 3A and 3B.

[0047] Since the independence setting 412, the formality setting 414, and the length setting 416 are described above with reference to FIG. 3A and FIG. 3B, the description is not repeated here. The initiative setting 418 can be adjusted between a minimum amplitude of zero indicating that the voice assistant reacts to the user input and a maximum amplitude of five indicating that it is active without waiting for the user input. In other words, when the initiative setting 418 is associated with a low value, the voice assistant system can be configured to behave in a more reactive manner and not initiate a dialogue or task without user input. When the initiative setting 418 is associated with a high value, the voice assistant can act more actively and initiate dialogues and tasks by itself. For example, in response to traffic established on the navigation route on which the vehicle is traveling, a more active voice assistant can automatically notify the user of the traffic and suggest alternative routes, while a less active voice assistant can be prohibited from initiating a dialogue with the user until the user asks if any alternative routes are available. Additionally, in the case where both the initiative setting 418 and the independence setting 412 are associated with high values, the voice assistant can automatically perform new tasks without user confirmation. Continuing with the traffic setup example above, the voice assistant could automatically switch to a new route without asking for user confirmation.

[0048] The specificity setting 420 can be adjusted between a minimum magnitude value of zero, indicating that the dialogue prompt is non-specific, and a maximum value of five, indicating that it is specific. In other words, when the specificity setting 420 is associated with a low value, the voice assistant can output prompts that generally only convey information without specifying the details of the task to enhance the user's privacy. When the specificity setting 420 is associated with a high value, the voice assistant can output more specific dialogue prompts to provide more information to the user. Continuing with the navigation example discussed above, in response to the user setting the science center as a destination, a non-specific confirmation may be "OK. Starting navigation now." Instead of a specific confirmation from the voice assistant, such as "OK. Navigating to the Science Center on 123 Atomic Avenue, State, City. Plan to arrive in XX minutes."

[0049] The humor setting 422 is adjustable between a minimum value of zero, indicating a serious conversation style, and a maximum value of five, indicating a humorous conversation style.

[0050] By referring to the interfaces shown in Figures 3 and 4, a vehicle designer can customize and adjust the global role settings of one or more vehicles via the global role setting component 206. As an example, the same global role of the voice assistant can be assigned to a fleet of vehicles of the same model. The global role of the voice assistant can vary between different vehicle models. The vehicle designer can assign a higher formality setting value 414 and a lower initiative value 418 to commercial vehicles (such as commercial sedans), and assign a lower formality setting value 414 and a higher initiative value 418 to sporty vehicles. Once the global role settings are determined, the voice assistant of the vehicle 104 can operate based on the global settings within the range of each corresponding role setting. The vehicle 104 can be configured to adjust the corresponding role settings within the range set by the global settings.

[0051] Referring to FIG5 , a flow chart of a process 500 for dynamically adjusting the role of a vehicle assistant system based on usage conditions is shown. Continuing to refer to FIGS. 1-4 , the process 500 can be implemented in whole or in part by the multimodal system 130 of the vehicle 104 to adjust the settings of the voice assistant system based on the global settings assigned to the vehicle 104. The process 500 can also be implemented in whole or in part via other components associated with the vehicle 104, such as the mobile device 138. For simplicity, the following description will be made with reference to the multimodal system 130. At operation 502, the multimodal system 130 obtains vehicle operation data from various sensors to determine the vehicle operation conditions. In general, the multimodal system 130 can be configured to adjust various settings of the voice assistant based on the vehicle operation conditions to provide an improved user experience to the user. As described above, the setting value of each voice assistant setting can be associated with a range having an upper boundary and a lower boundary manually set by the vehicle designer or automatically defined. The multimodal system 130 can be allowed to adjust the settings within the range to provide an improved user experience to the vehicle user by taking into account the vehicle operation data.

[0052] As a few non-limiting examples, the vehicle operation data may include traffic data received via the server 112 regarding the route that the vehicle is currently or is scheduled to traverse. The vehicle operation data may also include vehicle occupancy data indicating the number of users in the vehicle 104 and their identities (if applicable) detected via the camera and / or seat sensor 132. The vehicle operation data may also include data indicating conversations between vehicle users and conversations via the mobile device 138 detected via the microphone 142 and the connection to the mobile device 138. The vehicle operation data may also include vehicle powertrain data indicating a driving style (e.g., sport, leisure driving style) received from the powertrain. The vehicle operation data may also include automatic speech recognition (ASR) confidence data indicating the quality and accuracy of recognized speech from the user. The vehicle operation data may also include user emotion data indicating user emotions detected via the camera 132 and / or characteristics of speech from the user (e.g., timbre, voice, pitch). In response to receiving the vehicle operation data, the multimodal system 130 processes the data and determines whether to adjust one or more voice assistant settings.

[0053] At operation 504, the multimodal system 130 verifies whether the traffic on the route indicated by the traffic data is above a predefined threshold. If the answer is yes, indicating high traffic conditions on the route, indicating that the vehicle is traveling at a relatively low speed, the process proceeds to operation 506, and the multimodal system 130 increases the length setting 416 and the specificity setting 420, because the user may have more time to listen to a more detailed conversation from the voice assistant system. Additionally, the multimodal system reduces the aggressiveness setting 506 to allow the user to make a decision about whether to perform an action. If the answer to operation 504 is no, indicating low traffic on the route, which indicates that the vehicle speed is high, the process proceeds to operation 508, and the multimodal system 130 reduces the length setting 416 and the specificity setting 420 to avoid providing too much information to the driver. Additionally, the multimodal system 130 increases the aggressiveness setting 418 and acts in a more proactive manner.

[0054] The voice assistant may interact with multiple users using the vehicle 104 simultaneously. In response to receiving a voice command from a vehicle user, at operation 510, the multimodal system 130 verifies whether there are other occupants in the vehicle. If the answer is no, the process proceeds to operation 512, and the multimodal system 130 reduces the formality setting 414, provided that the user is known and familiar with the vehicle 104, and does not need to act in such a formal manner. If other users are present in the vehicle, the process proceeds to operation 514 to determine whether there are any other users who require special attention, which may require the voice assistant to behave in a formal manner. For example, users who require special attention may include underage users and / or users who are not identified as being associated with the commanding user (e.g., non-family members who imply business travel). If no special attention is required (e.g., the passenger is an adult family member of the driver), the process proceeds to operation 512. In addition, in response to detecting that other occupants may require special attention (implying business travel or carpooling situations), the process proceeds to operation 516, and the multimodal system 130 increases the formality setting 414, so that the voice assistant behaves in a more formal manner when other occupants are present.

[0055] At operation 518, the multimodal system 130 determines whether the vehicle user is having a conversation with another passenger and / or having a conversation via the phone. Typically, when the user is having an ongoing conversation, prompts from the voice assistant may preferably be kept short. Therefore, if the answer is yes, the process proceeds to operation 520, and the multimodal system 130 decreases the length setting 416 and the specificity setting 420, and increases the aggressiveness setting 418 and the independence setting 412 to reduce interruptions to the user's conversation. In addition, if the user is not having a conversation, the process proceeds to operation 522, and the multimodal system 130 increases the length setting 416 and the specificity setting 420, and decreases the aggressiveness setting 418 and the independence setting 412.

[0056] At operation 514, the multi-modal system 130 determines the user's driving style. In response to detecting a sporty driving style (e.g., rapid acceleration), the process proceeds to operation 516 and the multi-modal system 130 decreases the voice assistant's formality setting 414. Otherwise, if the multi-modal system 130 detects a more casual / business driving style, the process proceeds to operation 528 and the multi-modal system 130 increases the formality setting 414.

[0057] The voice assistant may be further adjusted by the ASR confidence of the user utterance. In response to receiving the utterance from the user, at operation 530, the multimodal system 130 determines an ASR confidence score for the user utterance received via the microphone 142. In response to the ASR confidence score being greater than a predefined threshold indicating sufficient confidence, the process proceeds to operation 532, and the multimodal system 532 increases the independence setting 412 to act in a more independent manner. In addition, the process proceeds to operation 534, and the multimodal system 130 decreases the independence setting 412 and requests user confirmation before proceeding to execute the received voice command.

[0058] The voice assistant may also be adjusted by the user's emotional state or mood. In response to receiving a user utterance, the multimodal system 130 may analyze the utterance using factors such as voice, pitch, timbre, etc. to determine the user's emotion. Here, the utterance may include a voice command directed to the voice assistant. Additionally or alternatively, the utterance may include conversational speech and / or non-conversational speech (e.g., sighs) from the user captured by the microphone 142. The user's emotion may be quantified as a value, and in response to an emotion value indicating low user emotion, the process proceeds to operation 538, and the multimodal system 130 may increase the humor setting 422 in an attempt to cheer the user up through the conversation. In addition, the process proceeds to operation 540 to reduce the humor setting 422. Similarly, the multimodal system 130 may increase the humor setting 422 in response to increased traffic to keep the driver relaxed.

[0059] Referring to FIG6 , a flowchart of another process 600 for dynamically adjusting the role of a vehicle assistant system based on usage conditions is shown. For simplicity, the following description will be made with reference to the multimodal system 130 . Continuing to refer to FIGS. 1-5 , the voice assistant can be adjusted by user input / feedback to the output dialogue. At operation 602 , the voice assistant outputs a dialogue via the speaker 144 . The dialogue can last for a period of time (e.g., a few seconds). In response to detecting that the user interrupts the dialogue output via the speaker 144 , the process proceeds from operation 604 to 606 , and the multimodal system 130 reduces the specificity setting 420 and the length setting 416 to reduce the length of the dialogue output. At operation 608 , in response to detecting that the user requires more detailed information after the dialogue output, the process proceeds to operation 610 , and the multimodal system 130 increases the specificity setting 420 and the length setting 416 . In an alternative embodiment, a counter can be used to associate with operations 604 and 608 , and the multimodal system 130 can adjust the settings only if the user interrupts and / or the problem reoccurs more than a predefined number of times.

[0060] Referring to FIG. 7 , a flowchart of another process 700 for dynamically adjusting the role of a vehicle assistant system based on user identity and characteristics is shown. For simplicity, the following description will be made with reference to the multimodal system 130 . Continuing to refer to FIGS. 1-5 , the multimodal system 130 can be configured to adjust the personality settings of the voice assistant based on the identity and characteristics of the user. User characteristics can include various information indicating the user who is interacting with the voice assistant. For example, user characteristics can include the user's seat position indicating the user's role in the vehicle (e.g., driver, front passenger, rear passenger), user identity, user age, etc. At operation 702 , in response to receiving a voice command, the multimodal system 130 identifies which user the voice command comes from via vehicle sensors 132 (e.g., cameras, seat sensors), microphones (e.g., sound orientation), etc. at operation 704 . In response to determining the user, at operation 706 , the multimodal system 130 identifies the user who is speaking via facial recognition technology and a connection to a mobile device 138 associated with the user to determine at operation 708 whether the user has a user profile indicating voice assistant preferences stored in the storage device 124 . If the answer is yes, the process proceeds to operation 710, and the multimodal system 130 loads the user profile and adjusts the voice assistant system using the user profile. Otherwise, if the answer is no, the process proceeds from operation 708 to operation 712, and the multimodal system 130 analyzes user characteristics. For example, the multimodal system 130 can analyze the age of the speaking user using voice analysis, facial recognition, and the user's weight through a seat sensor. At operation 714, the multimodal system 130 uses the user characteristics obtained at operation 712 to adjust the voice assistant settings.

[0061] Although examples are described herein, other vehicle systems may be included and contemplated. Although not specifically shown, the vehicle may include an onboard automotive processing unit, which may include an infotainment system including a head unit and a processor and memory. The infotainment system may interface with a peripheral set including one or more peripherals such as microphones, speakers, tactile elements, cabin lights, cameras, projectors, and indicators, etc. The head unit may execute various applications such as a voice interface and other entertainment applications such as a karaoke application. Other processing includes text to speech, recognition components, etc. These systems and components may respond to user commands and requests.

[0062] The computing devices described herein generally include computer-executable instructions, which can be executed by one or more computing devices such as those listed above. The computer-executable instructions can be compiled or interpreted from a computer program created using a variety of programming languages ​​and / or technologies, including, alone or in combination, Java, TM, C, C++, C#, Visual Basic, Java Script, Perl, etc. Typically, a processor (e.g., a microprocessor) receives instructions, such as from a memory, a computer-readable medium, etc., and executes these instructions to perform one or more processes, including one or more of the processes described herein. Various computer-readable media can be used to store and transmit such instructions and other data.

[0063] Although exemplary embodiments are described above, these embodiments are not intended to describe all possible forms of the present invention. On the contrary, the words used in the specification are descriptive words rather than restrictive words, and it should be understood that various changes can be made without departing from the spirit and scope of the present invention. Additionally, the features of various implementation embodiments can be combined to form other embodiments of the present invention.

Claims

1. A voice assistant system for a vehicle, comprising: a microphone configured to detect an audio signal from a user of the vehicle; a speaker configured to output a conversation in response to the audio signal; as well as A processor programmed to: In response to detecting a conversation involving the user, a length setting of the voice assistant system is reduced to reduce the length of the conversation, and an independence setting of the voice assistant system is increased to prevent confirmation issues from the voice assistant system.

2. The voice assistant system of claim 1, wherein the processor is further programmed to: In response to detecting increased traffic on the route traversed by the vehicle, increasing the specificity setting of the voice assistant system to provide more detailed information via the conversation.

3. The voice assistant system of claim 1, wherein the processor is further programmed to: In response to detecting the presence of an occupant other than the user in the vehicle, increasing the formality setting of the voice assistant system so that the conversation is structured in a more formal manner.

4. The voice assistant system of claim 3, wherein the processor is further programmed to: identifying the occupant; and In response to the occupant not being recognized by the voice assistant system or being underage, the formality setting is increased.

5. The voice assistant system of claim 1, wherein the processor is further programmed to: In response to detecting that the acceleration of the vehicle exceeds a threshold, reducing the formality setting of the voice assistant system so that the conversation is constructed in a less formal manner.

6. The voice assistant system of claim 1, wherein the processor is further programmed to: processing the audio signal to generate a voice command, and determining a confidence score for the voice command; and In response to the confidence score being below a threshold, reducing a dependency setting of the voice assistant system such that the voice assistant system requests user confirmation before beginning to execute the voice command.

7. The voice assistant system of claim 1, wherein the processor is further programmed to: analyzing the audio signal and using at least one of the timbre or pitch of the audio signal to determine a user emotion; and In response to the user emotion being less than a threshold, increasing a humor setting of the voice assistant so that the conversation is structured in a more relaxed manner.

8. The voice assistant system of claim 1, wherein the processor is further programmed to: In response to a user interruption while outputting the dialog via the speaker, reducing a specificity setting of the voice assistant system to output a less specific dialog.

9. A method for a voice assistant system for a vehicle, comprising: detecting, via a microphone, speech from a user among a plurality of users of the vehicle; identifying the one of the plurality of users that made the utterance; determining user characteristics associated with the one user; as well as A plurality of voice assistant settings are adjusted between at least one lower boundary and at least one upper boundary based on the user characteristic.

10. The method according to claim 9, wherein: The user characteristic includes at least one of: a user profile stored in the vehicle or received from a mobile device wirelessly connected to the vehicle, a user seat location, or a user age.

11. The method according to claim 9, wherein: The voice assistant settings include at least three of the following: an independence setting, a formality setting, a length setting, an initiative setting, a specificity setting, or a humor setting.

12. The method according to claim 11, further comprising: identifying a second user other than the one user who made the utterance; as well as In response to the second user not being recognized by the voice assistant system or being underage, increasing the formality setting.

13. The method according to claim 11, further comprising: In response to detecting a conversation between the multiple users of the vehicle, a length setting of the voice assistant system is reduced, and the independence setting and the aggressiveness setting of the voice assistant system are increased.

14. The method according to claim 11, further comprising: In response to detecting increased traffic on a route traversed by the vehicle, the humor setting is increased.

15. The method according to claim 11, further comprising: In response to receiving a user question requesting more information, the specificity setting is increased.

16. A non-transitory computer readable medium comprising instructions that, when executed by a processor of a vehicle, cause the vehicle to: detecting, via a microphone, speech from a user among a plurality of users of the vehicle; identifying, via the processor, the one of the plurality of users that made the utterance; determining, via the processor, a user characteristic associated with the one user; adjusting, via the processor, a plurality of voice assistant settings between at least one lower boundary and at least one upper boundary based on the user characteristic; as well as Conversations based on multiple voice assistant settings are output via the speaker.

17. The non-transitory computer readable medium of claim 16, further comprising instructions that, when executed by the processor of the vehicle, cause the vehicle to: In response to detecting that there is another user among the multiple users who is not recognized by the vehicle or is underage, increasing the formality setting of the voice assistant setting via the processor so that the conversation is structured in a more formal manner.

18. The non-transitory computer readable medium of claim 16, further comprising instructions that, when executed by the processor of the vehicle, cause the vehicle to: In response to a user interruption while outputting the dialog via the speaker, reducing, via the processor, a specificity setting of the voice assistant setting to output a less specific dialog.

19. The non-transitory computer-readable medium of claim 16, wherein the user characteristics include at least one of: a user profile stored in the vehicle or received from a mobile device wirelessly connected to the vehicle, a user seat location, or a user age.

Images