Emotion detection in disruption analysis

Abstract

A method for managing an interaction between a user and a driver interaction system in a vehicle includes presenting a first audio output from an output device of the driver interaction system to the user, and, while presenting the first audio output to the user, receiving a sensing input at the driver interaction system, processing the sensing input, and outputting the processed sensing input to the driver interaction system. And controlling the interaction based at least in part on emotional content of the sensed input, processing the sensed input including determining emotional content of the driver.

Description

[0001] Related Applications

[0002] This application claims the benefit of the priority date of U.S. Application No. 17 / 846,401, filed on June 22, 2022, the contents of which are incorporated herein by reference. Background Art

[0003] The present invention relates to "barge-in" detection and analysis in driver interaction systems.

[0004] The driver interaction system includes an in-vehicle voice assistant that performs tasks or services for the driver (and possibly passengers) based on commands received from the driver. Typically, the driver provides commands to the voice assistant by speaking the command or entering the command through a user interface. For some commands, the voice assistant serves the command and presents a verbal output to the driver.

[0005] An "interrupt" event occurs when the driver begins speaking during the presentation of spoken output. Some conventional voice assistants interrupt the voice assistant's presentation of spoken output when an interrupt event is detected. Summary of the invention

[0006] Some conventional methods of handling interruption events are prone to "false positives" because they can detect speech and interrupt the voice assistant even when the speaker does not intend for their speech to interrupt the voice assistant. In addition, many conventional methods of handling interruption events are unsophisticated because they simply stop the interaction between the driver and the voice assistant without taking any corrective action.

[0007] Aspects described herein relate to a method of handling interruption events that analyzes the content and nature of a driver's speech (and possibly other variables) to infer the driver's emotional state. The emotional state is then used to determine whether the spoken output of a voice assistant should be interrupted, and if so, how to modify the interaction between the driver and the voice assistant based on the inferred emotional state.

[0008] In one aspect, the invention features a method for managing an interaction between a user in a vehicle and a driver interaction system. The method may include: presenting a first audio output to the user from an output device of the driver interaction system. The method may also include: while presenting the first audio output to the user, receiving a sensory input at the driver interaction system, processing the sensory input, and controlling the interaction based at least in part on an emotional content of the sensory input, processing the sensory input including determining an emotional content of the driver.

[0009] In a general aspect, a method for managing interaction between a user in a vehicle and a driver interaction system includes presenting a first audio output to the user from an output device of the driver interaction system, receiving a sensory input at the driver interaction system while presenting the first audio output to the user, processing the sensory input, and controlling the interaction based at least in part on an emotional content of the sensory input, wherein processing the sensory input includes determining an emotional content of the driver.

[0010] Aspects can include one or more of the following features.

[0011] The sensory input may include spoken input. Processing the sensory input may include determining one or more words present in the spoken input. Processing the sensory input may include determining the presence of speech in the spoken input. Determining the presence of speech in the spoken input may be based on one or more of signal energy, frequency content, and periodicity of the spoken output. Determining the emotional content of the sensory input may include classifying features of the sensory input according to an emotion detector. The emotion detector may classify the features of the sensory input into a discrete set of emotion categories. The discrete set of emotion categories may include a happy class, an anxious class, an angry class, and an annoyed class. Classifying the features of the sensory output by the emotion detector may include assigning scores of the sensory output to multiple emotions of the discrete set of emotions. The discrete set of emotions may include one or more of a happy emotion, an anxious emotion, an angry emotion, and an annoyed emotion.

[0012] Determining the emotional content of the sensory input may include processing the sensory input to determine a dimensional representation of the emotional content of the sensory input. The dimensional representation of the emotional content may include a multi-dimensional representation of the emotional content. A first dimension of the multi-dimensional representation of the emotional content may be a valence dimension, and a second dimension of the multi-dimensional representation of the emotional content may be an arousal dimension. The dimensional representation of the emotional content may include a scalar representation of the emotional content in a substantially continuous range of scalar values ​​corresponding to the emotional range. Processing the spoken input may include determining one or more of a pitch and an amplitude of the spoken input.

[0013] Processing the spoken input may also include processing spectral features of the spoken input. The sensory input may include one or more of the following: spoken input, camera input, force sensor input, physiological sensor input, radar sensor input, proximity sensor input, location information, and temperature input. Controlling the interaction may include terminating the presentation of the first audio output according to the processing. Terminating the presentation according to the processing may include determining that the emotional content of the spoken input indicates a negative emotion to the first audio output. Negative emotions may indicate a lack of understanding of the first audio input. Controlling the interaction may include determining a dialogue state according to the processing and presenting subsequent audio outputs based on the determined dialogue state. Controlling the interaction may include: continuing to present the first audio output based on determining that the emotional content of the spoken input indicates a positive emotion to the first audio output. The first audio output may include spoken output. The driver interaction system may continuously receive sensory input.

[0014] In another general aspect, a driver interaction system for interacting with a driver in a vehicle includes a driver sensing component, a voice detector, a voice recognizer, an emotion detector, and an interruption detector, wherein the driver sensing component includes a microphone, wherein the driver sensing component senses a sensing signal including a voice signal, the voice detector processes the sensing signal and generates an output voice signal corresponding to a voice from the driver; the voice recognizer processes the sensing signal to generate a transcript of the voice signal, the emotion detector processes the sensing signal to generate a classified emotion of the driver, and the interruption detector processes any of the sensing signal, the output voice signal, the transcript, and the classified emotion to determine whether an interruption event has occurred.

[0015] In another general aspect, a non-transitory computer-readable medium has instructions stored thereon that, when executed by a processor of a driver interaction system, cause the driver interaction system to perform a first action, a second action, and a third action when the driver interaction system outputs a first audio output to a driver. The first action is an action of receiving a sensory input, the second action is an action of processing the sensory input, the processing the sensory input including determining an emotional content of the driver, and the third action is an action of controlling an interaction with the driver based at least in part on the emotional content of the sensory input.

[0016] Other features and advantages of the invention will be apparent from the following description, and from the claims. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] Figure 1 is a schematic diagram of the driver interaction system.

[0018] Figure 2 yes Figure 1 Schematic diagram of a driver interaction system in a first exemplary scenario.

[0019] Figure 3 yes Figure 1 Schematic diagram of a driver interaction system in a second exemplary scenario.

[0020] Figure 4 yes Figure 1 Schematic diagram of the driver interaction system in a third exemplary scenario.

[0021] Figure 5 yes Figure 1 Schematic diagram of the driver interaction system in a fourth exemplary scenario. DETAILED DESCRIPTION

[0022] 1 Overview

[0023] Reference Figure 1 , the driver interaction system 100 provides a communication interface to a driver 110 in a vehicle 105. For example, by providing spoken output to be heard by the driver 110 and receiving spoken input from the driver 110, at least one mode of communicating with the driver is via an acoustic interface using a speaker 122 and a microphone 124 (or alternatively, a directional microphone array).

[0024] Very commonly, the driver interaction system 100 transmits a message to the driver 110, such as a route instruction or a response to a driver's verbal or other input command. In some examples, the driver 110 speaks during the communication performed by the driver interaction system 100, intending to interrupt the communication, which is referred to as an "interrupt" event in this article. An interrupt event may involve any number of expressions of the driver 110, such as a verbal request to clarify the route instruction by the driver interaction system 100, or a voice notifying the driver interaction system 100 that it has misunderstood the driver 110's past instructions. However, not all voices from the driver 110 (or from another person in the car) are interrupt events intended to interrupt communication. For example, the detected voice from someone other than the driver 110 or from the driver 110 communicating with someone else in the car may not be intended to interrupt the driver interaction system 100. As described in more detail below, the driver interaction system 100 analyzes the voice and other sensors and context data to infer the emotional state of the driver 110. The driver interaction system 100 uses the inferred emotional state to determine whether and how to respond to the driver's voice.

[0025] The driver interaction system 100 includes a driver presentation component 132 and a driver sensing module 134. The driver presentation component 132 provides output to the driver 110, for example, synthesized spoken output for presentation via a speaker 122, or a video output via a screen. The driver sensing module 134 receives sensor data including a signal from a microphone 124. In some examples, the sensor data also includes a signal from a camera 126 in the steering wheel and one or both of a signal from a pressure sensor 128. For example, the camera 126 captures a view of the driver's face that can help determine when the driver 110 speaks (or even what they say), as well as other characteristics as further described below. The pressure sensor 128 measures characteristics such as the force with which the driver 110 holds the steering wheel. In some examples, the driver sensing module 134 also receives a signal provided to the speaker 122 to perform signal elimination to remove as much output audio signal as possible from the microphone signal to provide a signal representing the driver's spoken output.

[0026] In operation where interruptions are not present and the system and driver take “turns” to provide output and speak, respectively, the interaction control module 160 directs the interaction by processing inputs received from the driver via the driver sensing module 134 and providing outputs via the driver presentation module 132 .

[0027] To support the interruption feature of the system, the driver sensing module 134 also outputs the sensed driver signals to a plurality of signal analysis modules 141, which are used to detect and / or characterize interruption events. Typically, the signal analysis modules 141 analyze the sensed signals and provide the analysis results to the interruption logic 150. The interruption logic 150 processes the analysis results to determine whether an interruption event has occurred, and to characterize the interruption event, for example, based on the emotional content of the driver input. The output of the interruption logic 150 is provided to the interaction control module 160, which processes the output of the interruption logic 150 to determine how to control the interaction with the driver based on the driver's input.

[0028] In some examples, the signal analysis module 141 includes a voice detector 146, a voice recognizer 144, and an emotion detector 142. The voice detector 146 processes the sensed signal to determine whether the driver 110 is speaking. For example, based on the periodicity of the signal energy, frequency content, and / or audio signal (i.e., after the signal is eliminated), and the time variation of such signal measurements may be considered, a variety of voice detection methods may be used. In some examples, the camera 126 senses the driver's lip movement and provides information notifying the voice detector 146 when the driver 110 speaks. In an example where the driver's voice characteristics are known, the voice detector 146 can distinguish the driver's input from other passenger inputs based on the voice characteristics. The voice detector 146 thus processes the sensed audio and video signals to determine which audio signals in the sensed audio signals correspond to the driver's voice, and outputs instructions about which audio signals (if any) correspond to the driver's voice to one or more of the voice recognizer 144, the emotion detector 142, and the interruption logic 150. Processing the sensed audio signal may include determining a particular metric (eg, pitch and / or amplitude). Alternatively, processing may include processing the raw audio waveform.

[0029] The speech recognizer 144 processes the sensor data to generate an automatic transcript of what the driver 110 said. In some examples, the speech recognizer 144 runs continuously and produces an output transcript when it determines that speech is present, while in other examples, the speech detector 146 triggers speech recognition. In some versions, the speech recognizer 144 uses instructions from the speech detector 146 to construct a transcript specifically corresponding to the driver's speech. The speech recognizer 144 outputs the transcript to the emotion detector 142, the interruption logic 150, and the interaction control module 160.

[0030] The emotion detector 142 analyzes the characteristics of the sensing signal generated by the driver 110 and characterizes the emotional content of the signal. For example, the emotion detector 142 classifies the signal as belonging to one or more discrete emotional states. For example, these states can be "happy", "anxious", "angry", "irritated", etc. Alternatively, the emotion detector can provide weights of emotional states, or some other categories or non-categorical representations of emotional content. Examples of non-categorical representations of emotional content include digital or dimensional (e.g., scalar or vector) representations of emotional content. An example of a dimensional representation of emotional content includes a two-dimensional valence and arousal representation of emotional content (wherein emotions are represented as vectors in a two-dimensional space). In another example, a continuous range of emotions is represented by a continuous range of scalar values ​​(e.g., 0.0 to 1.0), where each value in the range represents a different emotion on a unit circle in a valence and arousal space (e.g., mapping a unit circle in a valence and arousal space to a line with values ​​of 0.0 to 1.0).

[0031] The emotion detector 142 may use one or more of the sensing signals provided by the driver sensing module 134 and the versions of these signals processed by the analysis module 141. For example, the classifier may use audio of the user's voice, a voice detection signal generated by the voice detector 146, a voice transcript from the voice recognizer 144, a video of the user's face, a physical measurement such as the user's grip on the steering wheel, or a specific subset of these signals in different alternatives. Examples of users of such signals for the emotion detector 142 include: using facial features of the driver to aid in emotion classification, a tight grip indicating that the driver 110 is not "relaxed", and a transcript of spoken words indicating the driver's emotional state.

[0032] In different implementations, the emotion detector 142 may use different methods of emotion classification. Such methods include knowledge-based or rule-based (dictionary-based) methods, machine learning methods (e.g., statistical methods or neural networks), and hybrid methods that combine knowledge-based methods and machine learning methods. The machine learning method can be configured through a "training" process that is performed before the system is deployed and used to determine the values ​​of configurable parameters of the emotion detector. For example, such parameters can be weights of a neural network, or parameters used with statistical and / or probabilistic methods. A training method utilizes a corpus of sensor signals (e.g., audio, video, grip) annotated with emotion labels. For example, the corpus can have multiple discrete inputs (e.g., utterances), each of which is labeled with a corresponding emotional state. In some examples, the emotional state annotation can be a time-varying signal that characterizes changes in emotional states within an utterance or continuous input. In some examples, the emotion detector processes features of the sensed signal to determine its output. In some examples, the features of the sensed signal include the pitch and amplitude of the sensed input. In other examples, the features may include segments of the raw sensory data (eg, audio segments of 10 ms duration) or spectral representations of the raw sensory data (eg, spectrum of short duration such as 10 ms).

[0033] Typically, the emotion detector 142 runs continuously (or at least when the driver provides spoken input) and generates an output classified emotion, which is provided to the interruption logic 150. In some alternative examples, the emotion detector 142 outputs a predicted emotion only when certain triggers are met. The output of the emotion detector 142 is provided to the interruption logic 150.

[0034] The interruption logic 150 receives as input the sensed signal, the classified emotion, the transcript, and the indication from the voice detector 146, and processes the input to determine (1) whether an interruption event has occurred, and (2) what characteristics are associated with the interruption event. Using the sensed signal and the output of the signal analysis module 141, the interruption logic 150 determines when the driver intends to interrupt the communication being conducted by the driver presentation component 132 (i.e., whether an interruption event has occurred). Similarly, if the interruption logic 150 determines the characteristics of the interruption event, such as the emotion expressed by the driver 110 during the interruption, the volume of the driver's voice, the tone of the driver's voice, etc. The interruption logic 150 outputs an indication of whether an interruption event has occurred and at least some characteristics of the interruption event (if it has occurred) to the interaction control module 160.

[0035] When the interaction control module 160 is providing output to the driver (i.e., via the driver presentation module 160) and it receives an indication of an interruption event from the interruption module, it determines how to react to the interruption event. While the interaction control module may have strict rules for interrupting the presentation of output to the driver whenever the interaction control module receives an indication of an interruption event, more generally, the interaction controller implements more nuanced handling of interruption events. A set of non-limiting examples of how the interaction control module may be configured and respond to interruption events are provided below.

[0036] In some examples, information provided to the interaction control module 160 by the interruption logic 150 leads the interaction control module to infer that output to the driver 110 should cease. For example, this may be the case when the user's emotional content indicates a level of urgency, such as when the driver 110 has begun to speak a new command (e.g., to correct a misunderstanding of a previous command).

[0037] In some examples, output from the interruption logic 150 may indicate to the interactive control module that the driver 110 is providing confirmatory feedback, in which case the output should continue.

[0038] In some examples, the output from the interruption logic may have emotional content indicating that the driver is impatient, in which case the interaction control module may determine to stop or abbreviate the output to the driver while maintaining the interaction state with the driver. In some cases, such a determination may affect how the interaction control module provides further output (e.g., in long or abbreviated form).

[0039] In some examples, the interaction control module 160 may also use external sensing data from the external sensing module 170, such as indicating vehicle speed, position, turn or brake status, etc., and the manner in which the interaction control module 160 reacts to the interruption event may depend on the external sensing data. For example, a verbal interruption combined with a grimace on the driver's face may be associated with a vehicle position indicating that a navigation turn has been missed, so the interruption event combined with the vehicle position may affect how the interaction control module further guides the driver.

[0040] 2 Example Scenario

[0041] Typically, the interaction between the system and the driver 110 follows a "turn-taking" sequence. The interactive control module 160 causes the output to be presented to the driver 110 via the driver presentation component 132, typically in audio form, via the speaker 122. After the output is presented, in some cases, the driver 110 can provide a response or command, such as the response or command is captured by the microphone 124, passed through the driver sensing module 134 and provided to the interactive control module 160, and the response or command acts on the driver's response or command.

[0042] As introduced above, in some examples, the driver 110 does not wait for his or her turn, but instead "interrupts" on the system's output (i.e., simultaneous system output and driver input) by talking to the system. A number of example scenarios of driver interruptions and the system determining how to react to the interruptions are provided below.

[0043] 2.1 Example Scenario 1

[0044] refer to Figure 2 , with reference to a plurality of enumerated events depicted on the figure to depict the first example scenario of the driver 110 interruption. These events and similarly described events in subsequent figures can be referred to as "steps", but do not imply that a step is completed before the next step begins. In this example, step 1 represents that the interactive control module 160 enables the driver presentation component 132 to provide navigation instructions by outputting an audio command "turn left, ...". The output of the instruction continues during the steps described below, including in step 2, which represents that the driver 110 talks about the output from the driver presentation component 132 by saying "no left side! I can't turn left!" (e.g., surprised to find that there is no place to make the indicated left turn) interruption. The driver 110 said that this made a loud sound with an aggressive tone while tightly grasping the steering wheel. As described below, this driver input is detected by the driver interaction system 100, and the interactive control module 160 finally receives the notification of the interruption and takes action on it.

[0045] In step 3, when the driver 110 speaks, the microphone 124 captures an audio signal including the driver's voice, the camera 126 captures a video signal including the driver's face when speaking, and the pressure sensor 128 captures a pressure signal representing the driver's grip strength on the steering wheel. The audio signal, the video signal, and the pressure signal are received and processed by the driver sensing module 134. The driver sensing module 134 outputs the sensing signal to the signal analysis module 141 for subsequent processing.

[0046] Steps 4A-4D represent processing of the output of the driver sensing module 134 that begins during input from the driver 110 (i.e., they occur simultaneously with the driver's input and with the output of the driver presentation component 132). In this example and in example scenarios described subsequently, the signal is processed on an ongoing basis in the event that the interactive control module 160 is not expecting input, and the signal analysis module 141 in turn provides its output to the interruption logic 150 (and in some cases, as described below).

[0047] Step 4A represents that the voice detector 146 processes the sensed audio and video signals to output an indication that the driver 110 is speaking to the interruption logic 150. That is, the signal transmitted from the voice detector 146 is an indicator that it has detected a voice signal and possibly and more specifically a voice signal from the driver 110.

[0048] Step 4B represents that the speech recognizer 144 processes the sensed audio signal to output a transcript of the detected speech (i.e., "No left! I can't turn left!") to the interruption logic 150 and the interaction control module 160 on an ongoing basis (e.g., outputting a partial transcript as the driver's speech is processed rather than waiting for the driver 110 to stop speaking). In some alternatives, the speech recognizer 144 processes input only after it receives a speech detection indicator from the speech detector 146.

[0049] Step 4C represents that the emotion detector 142 processes the sensed driver signal from the driver sensing module 134 to generate an output to the interrupt logic 150, which represents the emotional state of the driver determined according to the input. Like the voice detector 146 and the voice recognizer 144, the emotion detector 142 provides an indication of the emotional state of the driver on an ongoing basis, or alternatively, whenever it detects a change in the state. Due to the loud volume, coarse modulation and content of the driver's voice in this example, the emotion detector 142 classifies the driver's emotion as "angry" in this example. In this example, the emotion detector 142 can also access other signals, such as the driver's grip on the steering wheel and the driver's facial expression, which help the emotion detector output the "angry" classification to the interrupt logic 150. In at least some examples, the emotion detector 142 receives a transcript from the voice recognizer 144 and / or a voice detection indicator from the voice detector 146, and uses these inputs in its decision to provide an "angry" classification.

[0050] In step 4D, the interruption logic 150 receives and processes the ongoing outputs of the speech detector 146, the speech recognizer 144, and the emotion detector 142 to determine whether to provide an indication to the interaction control module 160 that an interruption event from the driver 110 may exist, and if so, what attributes of the interruption event are provided in the indication. In this example, the interruption logic 150 determines that an interruption event has occurred, and the interruption logic 150 makes this determination based on the "angry" classification, the transcript, and the sensed driver signal.

[0051] Step 5 represents issuing an interruption indication indicating the occurrence of an interruption event from the interruption logic 150 to the interaction control module 160. In this example, the interruption indication includes characteristics including the volume of the driver's voice and the emotion of the driver 110. In some examples, for example, as more input is transcribed or further emotion classification becomes available, the interruption logic 150 issues subsequent indications during the driver's input.

[0052] Step 6 represents that the interactive control module 160 receives and processes the indication including the volume and emotional characteristics issued by the interruption logic 150. The interactive control module 160 processes the indication and determines in this scenario that its attributes represent a situation in which the output (which is still in progress) should be interrupted and a new communication should be made. In this scenario in which the interactive control module 160 receives the indication in the process of providing navigation commands, the interactive control module 160 infers that the driver 110 does not understand the command or cannot execute the command. In order to address the assessment of the situation in this scenario, step 7 represents that the interactive control module 160 enters an explanation / clarification dialogue instruction, which results in the output of a new prompt "Okay, let me explain: the next turn is to turn left at the next traffic light in 450 feet" via the driver presentation component 132. In this scenario, for example, the interactive dialogue is conducted using the interactive control module 160 providing continuous navigation commands, for example, providing continuous navigation commands when the interactive control module 160 senses (e.g., via the external sensing module 170) that the vehicle is traveling along the planned route.

[0053] 2.2 Example Scenario 2

[0054] refer to Figure 3 In the second example scenario, the driver interaction system 100 captures the passenger's voice and determines that no interruption event has occurred. In this example, step 1 represents that the interactive control module 160 causes the driver presentation component 132 to provide navigation instructions by outputting an audio command "turn left, ...". The output of the instructions continues during the steps described below, including in step 2, which represents that the passenger talks about the output from the driver presentation component 132 by saying "Are we still there?" At the same time, the driver 110 maintains a neutral facial expression and does not change his or her grip on the steering wheel. As described below, the driver input is detected by the driver interaction system 100, and the interactive control module 160 eventually receives notification of the interruption and takes action on it.

[0055] In step 3, when the passenger speaks, the microphone 124 captures an audio signal including the passenger's voice, the camera 126 captures a video signal including the driver's facial expression, and the pressure sensor 128 captures a pressure signal representing the driver's grip strength on the steering wheel. The audio signal, the video signal, and the pressure signal are received by the driver sensing module 134 and may be processed by the driver sensing module 134. The driver sensing module 134 outputs the sensor signal to the signal analysis module 141 for subsequent processing.

[0056] Steps 4A-4D begin during input from the driver 110. Step 4A represents processing of the sensed audio and video signals by the voice detector 146. The voice detector 146 outputs an indication that the sensed audio signal does not correspond to the driver's voice based on its processing of the sensed audio and video signals (as described above).

[0057] Step 4B represents the speech recognizer 144 processing the sensed audio signal and outputting a transcript of the detected speech to the interruption logic 150. In this embodiment, the speech recognizer 144 processes the input only after it receives a (negative) speech detection indicator from the speech detector 146, which causes the speech recognizer 144 to output a blank transcript to the interruption logic 150 and the interaction control module 160.

[0058] Step 4C represents the emotion detector 142 processing the sensed driver signals from the driver sensing module 134 to classify the driver's emotion as “content” based on the driver's neutral facial expression, the driver's lack of voice, and the driver's neutral stress signal.

[0059] Step 4D represents the interruption logic 150 processing the outputs of the speech detector 146, the speech recognizer 144, and the emotion detector 142 to determine that an interruption event has not occurred. In this example, the interruption logic 150 makes this determination based at least in part on the "content" classification and the speech detection indicator.

[0060] Step 5 represents sending an indication that an interruption event did not occur (as a result of determining the lack of an interruption event) from the interruption logic 150 to the interaction control module 160. In this case, since the interruption logic 150 determines that an interruption event did not occur, the interruption indication does not contain further related characteristics to the interaction control module 160.

[0061] Step 6 represents the interactive control module 160 receiving and processing any information received from the interruption logic 150. In this example, the determination of the interruption logic 150 and the associated transmission from the interruption logic 150 to the interactive control module 160 causes the interactive control module 160 to continue operating as if speech had never been detected. Step 7 thus represents the interactive control module 160 inputting a dialog instruction causing the driver presentation component 132 to continue from step 1 by saying "... continue 400 feet before turning right on State Street" without pausing its sentence.

[0062] 2.3 Example Scenario 3

[0063] refer to Figure 4, a third example of a driver 110 interruption is depicted with reference to a plurality of enumerated events depicted on the figure. In this example, the first step represents that the interactive control module 160 causes the driver presentation component 132 to provide navigation instructions by outputting an audio command "Leave the highway at exit 123, previously known as ...". The output of the instruction continues during the steps described below, including at step 2, which represents that the driver 110 talks about the output from the driver presentation component 132 by saying "I know, I know!" (e.g., annoyingly, because the driver 110 knows the updated exit number, and therefore does not want to be informed of its previous exit number). The driver 110 said this in a loud voice with a bad tone. As described below, this driver input is detected by the driver interaction system 100, and the interactive control module 160 eventually receives the notification of the interruption and acts on it.

[0064] In step 3, when the passenger speaks, the microphone 124 captures an audio signal including the passenger's voice, the camera 126 captures a video signal including the driver's facial expression, and the pressure sensor 128 captures a pressure signal representing the driver's grip strength on the steering wheel. The audio signal, the video signal, and the pressure signal are received by the driver sensing module 134 and may be processed by the driver sensing module 134. The driver sensing module 134 outputs the sensor signal to the signal analysis module 141 for subsequent processing.

[0065] Steps 4A-4D represent processing of the output of the driver sensing module 134 , which begins during input from the driver 110 .

[0066] Step 4A represents that the voice detector 146 processes the sensed audio and video signals to output an indication of the presence of the voice signal to the interruption logic 150. Step 4B represents that the voice recognizer 144 processes the sensed audio signal to output a transcript of the detected voice to the interruption logic 150 (i.e., "I know, I know!"). As described above, step 4C represents that the emotion detector 142 processes the sensed driver signal from the driver sensing module 134 to classify the driver's emotion as "annoying", in this example, due to the driver's facial expressions, the driver's repeated statements, and the driver's loud voice and annoying tone. Step 4D represents that the interruption logic 150 processes the ongoing outputs of the voice detector 146, the voice recognizer 144, and the emotion detector 142 to determine that an interruption event has occurred. In this example, the interruption logic 150 makes this determination based on the "annoying" classification, the transcript, and the sensed driver signal.

[0067] Step 5 indicates that an interruption indication of the occurrence of an interruption event is sent from the interruption logic 150 to the interaction control module 160. The interruption indication sent to the interaction control module 160 includes characteristics, including the (loud) volume of the driver's voice and the (annoying) emotion of the driver 110.

[0068] Step 6 represents that the interactive control module 160 receives and processes the instructions and associated attributes sent from the interrupt logic 150. The interactive control module 160 processes the instructions and determines in this scenario that its attributes represent the situation that the output (which is still in progress) should be interrupted. In this scenario where the interactive control module 160 is in the process of providing navigation commands, the interactive control module 160 infers that the driver 110 no longer wants to hear what the previous name of the newly renamed exit is in this and subsequent driver instructions. In order to solve the evaluation of the situation in this scenario, step 7 represents that the interactive control module 160 enters a dialogue instruction to stop, so that the driver presentation component 132 stops its current output (that is, the ongoing message that the driver 110 interrupted in step 2 in step 1). In this scenario, for example, the interactive dialogue is carried out using the interactive control module 160 that provides continuous navigation commands, and the navigation command does not contain the previous exit name, for example.

[0069] 2.4 Example Scenario 4

[0070] refer to Figure 5 , a fourth example scenario of driver 110 interruption is depicted with reference to a plurality of enumerated events depicted on the figure. In this example, the first step represents that the interactive control module 160 causes the driver presentation component 132 to provide navigation instructions by outputting an audio command "turn left, ...". The output of the instruction continues during the steps described below, including in step 2, which represents that the driver 110 talks about the output from the driver presentation component 132 by saying "where? ..." (e.g., confusing the specific location where the driver 110 is instructed to turn). The driver 110 says this with a normal volume of voice and a neutral tone. As described below, this driver input is detected by the driver interaction system 100, and the interactive control module 160 eventually receives the notification of the interruption and takes action on it.

[0071] In step 3, when the passenger speaks, the microphone 124 captures an audio signal including the passenger's voice, the camera 126 captures a video signal including the driver's facial expression, and the pressure sensor 128 captures a pressure signal representing the driver's grip strength on the steering wheel. The audio signal, the video signal, and the pressure signal are received by the driver sensing module 134 and may be processed by the driver sensing module 134. The driver sensing module 134 outputs the sensor signal to the signal analysis module 141 for subsequent processing.

[0072] Steps 4A-4D represent processing of the output of the driver sensing module 134, which begins during input from the driver 110. Step 4A represents that the voice detector 146 processes the sensed audio and video signals. The voice detector 146 outputs an indication of the presence of a voice signal to the interruption logic 150. Step 4B represents that the voice recognizer 144 processes the sensed audio signal to output a transcript of the detected voice to the interruption logic 150 and the interactive control module 160 on an ongoing basis. Step 4C represents that the emotion detector 142 processes the sensed driver signal from the driver sensing module 134 to produce an output to the interruption logic 150, which represents the emotional state of the driver determined based on the input. In this example, the emotion detector 142 classifies the driver's emotion as "confused" due to the volume, tone and content of the driver's voice in this example. Step 4D represents that the interruption logic 150 processes the ongoing output of the voice detector 146, the voice recognizer 144 and the emotion detector 142 to determine that an interruption event has occurred. In this example, the interruption logic 150 makes this determination based on the “confusion” classification, the transcript, and the sensed driver signal.

[0073] Step 5 represents sending an interruption indication of the interruption event from the interruption logic 150 to the interaction control module 160. The interruption indication sent to the interaction control module 160 contains characteristics including the (normal) volume of the driver's voice and the (confused) emotion of the driver 110.

[0074] Step 6 represents the interaction control module 160 receiving and processing the indication and associated attributes sent from the interruption logic 150. The interaction control module 160 processes the indication and determines in this scenario that its attributes indicate a situation where the output (which is still in progress) should be interrupted. In this scenario where the interaction control module 160 is in the process of providing navigation commands, the interaction control module 160 infers that the driver 110 needs more information about the next turn than previously communicated.

[0075] To address the assessment of the situation in this scenario, step 7 represents that the interactive control logic 160 inputs an explanation / clarification dialogue instruction, causing a new prompt "Let me explain - continue 600 feet and turn left at the next stop sign" to be output via the driver presentation component 132. In this scenario, for example, the interactive dialogue is conducted using the interactive control module 160 that provides continuous navigation commands, which, for example, include the name of the previous exit. In a further example, where the emotion detector 142 determines that the driver 110 is, for example, panicking, the interactive control module 160 may instead provide, for example, continuous navigation commands designed to calm the driver 110.

[0076] 3 Alternatives

[0077] In the above examples, the emotion detector is described as generating a single predicted emotion as an output. In other examples, the emotion detector may instead generate a weighted set of emotions (e.g., [("angry", 0.5), ("happy, 0.0"), ("confused", 0.2), ...]). Such a weighted set may provide a more detailed picture of the driver's emotional state, resulting in improved interruption detection. Similarly, in some examples, the interruption logic 150 may generate a binary output (i.e., yes or no) indicating whether an interruption event has occurred, but in other examples, the interruption logic 150 may also generate a weighted output indicating a confidence interval for whether an interruption event has occurred.

[0078] In other examples, other dimensional models of emotions (e.g., PANA (Positive and Negative Affect, positive and negative effects) model or PAD emotional state model) are used to represent emotional content, such as happiness, dominance, positive and negative, etc. In some examples, scoring systems corresponding to other dimensional models of emotions or other methods or models of emotion classification can be used to determine the emotional state of the driver. A plurality of different types of sensors are described above, but it should be noted that any other type of sensor suitable for the purpose of interruption detection can be used. For example, motion sensors, temperature sensors, vibration sensors, humidity sensors, photoelectric sensors, etc. can all be used to sense the driver for use by the interruption logic. In some examples, physiological sensors are used to measure physiological input signals, such as force sensors, pressure sensors, skin conductance sensors, heart rate or electrocardiogram sensors, brain wave sensors, and any other type of sensors suitable for detecting physiological processes.

[0079] In all of the above examples, the driver speaking is part of causing the interruption event. However, in some examples, sensor data (such as camera signals and pressure signals from steering wheel sensors) can be used to cause the interruption event without any voice input. For example, if the driver makes a face and grips the steering wheel tightly, there may be an interruption event.

[0080] In some examples, the system may utilize external information beyond just sensing the driver and may instead involve the context or context of the interaction with the driver, such as information derived from the driver's calendar, weather information, traffic information, etc.

[0081] A number of embodiments of the present invention have been described. However, it should be understood that the foregoing description is intended to illustrate rather than limit the scope of the present invention, which is defined by the scope of the appended claims. Therefore, other embodiments are also within the scope of the appended claims. For example, various modifications may be made without departing from the scope of the present invention. In addition, some of the steps described above may be sequence independent and may therefore be performed in an order different from that described.

Claims

1. A method for managing interaction between a user in a vehicle and a driver interaction system, the method comprising: presenting a first audio output to a user from an output device of the driver interaction system; while presenting the first audio output to the user, receiving a sensory input at the driver interaction system; processing the sensory input, including determining the emotional content of the driver, and The interaction is controlled based at least in part on the emotional content of the sensory input.

2. The method according to claim 1, wherein: The sensory input includes spoken input.

3. The method according to claim 1 or 2, wherein: Processing the sensory input further includes determining one or more words present in the spoken input.

4. The method according to claim 1 or 2, wherein: Processing the sensory input further includes determining the presence of speech in the spoken input.

5. The method according to any one of the preceding claims, further comprising: The presence of speech in the spoken input is determined based on one or more of signal energy, frequency content, and periodicity of the spoken output.

6. A method according to any one of the preceding claims, wherein: Determining the emotional content of the sensory input includes classifying features of the sensory input according to an emotion detector.

7. The method according to claim 6, wherein: The emotion detector classifies features of the sensory input into a set of discrete emotion categories.

8. The method according to claim 7, wherein: The discrete emotion category set includes: a happy category, an anxious category, an angry category, and an annoyed category.

9. The method according to claim 6, 7 or 8, wherein: Classifying, by the emotion detector, features of the sensory output includes assigning scores to the sensory output to a plurality of emotions in a set of discrete emotions.

10. The method according to claim 7, 8 or 9, wherein: The discrete emotion set includes one or more of: a happy emotion, an anxious emotion, an angry emotion, and an annoyed emotion.

11. The method according to claim 1, wherein: Determining the emotional content of the sensory input includes processing the sensory input to determine a dimensional representation of the emotional content of the sensory input.

12. The method according to claim 11, wherein: The dimensional representation of the emotional content includes: a multi-dimensional representation of the emotional content.

13. The method according to claim 12, wherein: The first dimension of the multidimensional representation of the emotional content is the valence dimension, and the second dimension of the multidimensional representation of the emotional content is the arousal dimension.

14. The method according to claim 11, wherein: The dimensional representation of the emotional content includes a scalar representation of the emotional content in a substantially continuous range of scalar values ​​corresponding to a range of emotions.

15. The method according to any one of claims 2 to 14, wherein: Processing the spoken input further includes determining one or more of a pitch and an amplitude of the spoken input.

16. The method according to any one of claims 2 to 15, wherein: Processing the spoken input further includes: processing spectral features of the spoken input.

17. A method according to any one of the preceding claims, wherein: The sensory input includes one or more of: spoken input, camera input, force sensor input, physiological sensor input, radar sensor input, proximity sensor input, location information, and temperature input.

18. A method according to any one of the preceding claims, wherein: Controlling the interaction includes discontinuing presentation of the first audio output in accordance with the processing.

19. The method according to claim 18, wherein: Ceasing the presentation in accordance with the processing includes determining that emotional content of the spoken input indicates a negative emotion toward the first audio output.

20. The method according to claim 19, wherein: The negative emotion indicates a lack of understanding of the first audio input.

21. The method of claim 18, 19 or 20, wherein: Controlling the interaction further includes determining a dialog state based on the processing, and presenting subsequent audio output based on the determined dialog state.

22. A method according to any one of the preceding claims, wherein: Controlling the interaction includes continuing to present the first audio output based on determining that the emotional content of the spoken input indicates a positive emotion toward the first audio output.

23. A method according to any one of the preceding claims, wherein: The first audio output comprises spoken output.

24. A method according to any one of the preceding claims, wherein: The driver interaction system continuously senses sensory inputs.

25. A driver interaction system for interacting with a driver in a vehicle, the system comprising: A driver sensing component, comprising a microphone, wherein the driver sensing component senses a sensing signal, wherein the sensing signal comprises a voice signal; a voice detector, wherein the voice detector processes the sensing signal and generates an output voice signal corresponding to a voice from the driver; a speech recognizer, wherein the speech recognizer processes the sensed signal to generate a transcript of the speech signal; an emotion detector, wherein the emotion detector processes the sensed signal to generate a classified emotion of the driver, and An interruption detector, wherein the interruption detector processes any of the sensed signal, the output voice signal, the transcript, and the classified emotion to determine whether an interruption event occurs.

26. A non-transitory computer readable medium having instructions stored thereon that, when executed by a processor of a driver interaction system, cause the driver interaction system to perform a first action, a second action, and a third action when the driver interaction system outputs a first audio output to a driver, wherein: The first action is an action of receiving a sensing input; The second action is an action of processing the sensory input, wherein the processing the sensory input includes: determining the emotional content of the driver, and The third action is an action of controlling an interaction with the driver based at least in part on the emotional content of the sensory input.

Images