Emotion-aware voice assistant

Abstract

An interface customized for a detected emotional state of a user is provided. An audio signal indicative of a spoken word, phrase, or command is received from at least one microphone. A wakeup word (WuW) in the audio signal is detected. An emotion in the WuW-containing audio signal is also detected. And configuring an emotion perception processing system according to the detected emotion. A voice control session is performed using an emotion-aware processing system configured according to the detected emotion.

Description

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS

[0002] This application claims priority to U.S. patent application serial number 17 / 833,102, filed on June 6, 2022, the disclosure of which is incorporated herein by reference in its entirety. Technical Field

[0003] Described herein is a voice assistant system that provides a voice user interface that is customized to a detected emotional state of a user. Background Art

[0004] Some speech-enabled systems allow a voice trigger to be spoken to start interacting with the system. A voice trigger may include a specified word or phrase that indicates to the system that the user intends to interact with the system (e.g., issue one or more commands to the system). Such voice triggers are referred to herein as "wake-up words" or "WuW" (wake-up word), and refer to both single-word triggers and multi-word trigger phrases. Typically, once the wake-up word is detected, the system begins to recognize subsequent speech spoken by the user. In most cases, unless and until the system detects a wake-up word, the system will assume that the acoustic input received from the environment is not directed to the system or intended for the system, and will not further process the acoustic input. Summary of the invention

[0005] In one or more illustrative examples, a system for providing an interface customized for a detected emotional state of a user is provided. At least one microphone is configured to generate an audio signal indicative of a spoken word, phrase, or command. A processor is programmed to receive an audio signal from at least one microphone, detect a wake-up word (WuW) in the audio signal, detect an emotion in the audio signal containing the WuW, configure an emotion-aware processing system based on the detected emotion, and perform a voice control session using the emotion-aware processing system configured based on the detected emotion.

[0006] In one or more illustrative examples, a method for providing an interface customized for a detected emotional state of a user is provided. An audio signal is received from at least one microphone, the audio signal indicating a spoken word, phrase, or command. WuWs are detected in the audio signal. Emotions in the audio signal containing WuWs are detected. An emotion-aware processing system is configured based on the detected emotions. A voice control session is performed using the emotion-aware processing system configured based on the detected emotions. BRIEF DESCRIPTION OF THE DRAWINGS

[0007] Embodiments of the present disclosure are particularly pointed out in the appended claims. However, other features of the various embodiments will become more apparent and will be better understood by referring to the following detailed description in conjunction with the accompanying drawings, wherein:

[0008] Figure 1 A block diagram of a voice assistant system according to one embodiment is shown, the voice assistant system providing an interface customized for a detected emotional state of a user;

[0009] Figure 2 Example details showing aspects of a mood customization application; and

[0010] Figure 3 An example process for providing an interface tailored to a detected emotional state of a user is shown. DETAILED DESCRIPTION

[0011] Embodiments of the present disclosure are described herein. However, it should be understood that the disclosed embodiments are merely examples, and other embodiments may take various and alternative forms. The drawings are not necessarily drawn to scale; some features may be exaggerated or minimized to show the details of a particular component. Therefore, the specific structural and functional details disclosed herein should not be interpreted as restrictive, but only as a representative basis for teaching those skilled in the art to adopt the embodiments in various ways. As will be understood by those of ordinary skill in the art, the various features shown and described with reference to any one of the figures may be combined with the features shown in one or more other figures to produce embodiments that are not explicitly shown or described. The combination of the features shown provides representative embodiments for typical applications. However, various combinations and modifications of features consistent with the teachings of the present disclosure may be desired for specific applications.

[0012] WuW can convey information about the user's underlying emotional state. This emotional information can be used to improve the operation of the voice assistant system. To this end, the emotion of the WuW spoken by the user can be detected in parallel with the detection of WuW. The detected emotion can be used to customize the user interface of the voice assistant system. Therefore, in addition to triggering an automatic speech recognition (ASR) / natural language understanding (NLU) session to identify what the user said after WuW, the user's underlying emotional state can also be derived from WuW to customize the voice control session.

[0013] Customization of the user interface can be used for various purposes. In one example, the emotional state can be used to signal the empathy capabilities of the speech system through visual feedback or acoustic feedback (referred to herein as ear tags). In another example, the emotional state can be used to load a specific ASR / NLU model or configuration corresponding to the detected emotional context to improve recognition. For example, the execution of the ASR / NLU system can be adjusted according to the emotional state through an acoustic model, a language model weight set, a specific language model, etc. corresponding to the detected emotional state. In yet another example, the emotional state can be used to modify the manner and style in which the user interface (UI) or virtual assistant (VA) responds, such as its verbosity or its empathic response (e.g., submissive behavior in the face of aggressive user input). Other aspects of the present disclosure are discussed in detail herein.

[0014] Figure 1 A block diagram of a voice assistant system 100 for providing an interface customized for a detected emotional state of a user according to an embodiment of the present disclosure is shown. In one example, the voice assistant system 100 may be designed for a vehicle 104 configured to transport passengers. The vehicle 104 may include various types of passenger vehicles, such as a crossover utility vehicle (CUV), a sport utility vehicle (SUV), a truck, a recreational vehicle (RV), a boat, an airplane, or other mobile machines for transporting people or goods. In addition, the vehicle 104 may be an autonomous, partially autonomous, self-driving, unmanned, or driver-assisted vehicle. The vehicle 104 may be an electric vehicle (EV), such as a battery electric vehicle (BEV), a plug-in hybrid electric vehicle (PHEV), a hybrid electric vehicle (HEV), etc. Although Figure 1 , but other environments may be applied in other embodiments. For example, similar functionality may be applied to non-automatic environments, such as control of smart speakers or other home devices, control of augmented reality or virtual reality sessions, control of phones or other mobile devices, etc. Although the term "occupant" is used throughout, this term may be interchangeable with speaker, user, or another term for a user interacting with the voice assistant system 100 via speech.

[0015] The voice assistant system 100 may include one or more processors 106 configured to execute certain instructions, commands, and other routines as described herein. The processor 106 may include various general purpose devices, custom circuits such as an application specific integrated circuit (ASIC) or a field programmable gate array (FPGA), and / or a combination thereof, which may be operated to perform one or more of the processes described below to provide an interface customized for a detected emotional state of a user.

[0016] The processor 106 may run instructions for certain applications including navigation, infotainment, climate control, etc. Various types of computer-readable storage media 108 may be used to maintain instructions for controlling the processor 106 in a non-volatile manner. The computer-readable storage media 108 (also referred to herein as memory or storage devices) includes any non-transitory media (e.g., tangible media) that participates in providing instructions or other data that can be read by the processor 106. Computer-executable instructions may be compiled or interpreted from a computer program created using various programming languages ​​and / or techniques including, but not limited to, Java, C, C++, C#, Objective C, Fortran, Pascal, JavaScript, Python, and Perl, alone or in combination.

[0017] The voice assistant system 100 may be configured to communicate with a communication network 110. The communication network 110 may be referred to as a "cloud" and may involve data transmission via various wide area networks and / or local area networks such as the Internet, cellular networks, Wi-Fi, Bluetooth, etc. The communication network 110 may provide various services to the vehicle 104, such as navigation, music or other audio program content, marketing content, Internet access, speech recognition, cognitive computing, artificial intelligence computing, etc.

[0018] The voice assistant system 100 may include a wireless transceiver 112 configured to communicate over the communication network 110, such as a Bluetooth module, a ZIGBEE transceiver, a Wi-Fi transceiver, an IrDA transceiver, a radio frequency identification (RFID) transceiver, etc. The wireless transceiver 112 may also be used to allow the voice assistant system 100 to communicate with compatible transceivers of various user devices or other vehicles 104 as well.

[0019] The communication network 110 can provide communication between the vehicle 104 and an external or remote server 114 and / or database 116, as well as communication between other external applications, systems and / or vehicles 104. The remote server 114 and / or database 116 may include one or more computer hardware processors coupled to one or more computer storage devices for performing the steps of one or more methods as described herein. The remote server 114 and / or database 116 can thus enable the vehicle 104 to communicate and exchange information and data with systems and subsystems external to the vehicle 104 and local or onboard the vehicle 104.

[0020] The voice assistant system 100 may include one or more internal networks 118, such as a vehicle controller area network (CAN), an Ethernet network, and a media oriented systems transport (MOST), etc. The internal network 118 may allow the processor 106 to communicate with other vehicle 104 systems, such as a modem, a global navigation satellite system (GNSS) controller (not shown) configured to provide current position and heading information, and various electronic control units (ECUs) configured to cooperate with the processor 106.

[0021] The voice assistant system 100 may further include various sensors and input devices to support the operation of an interface customized for the detected emotional state of the user. For example, the cabin of the vehicle 104 may include at least one audio sensor 120. In one example, the audio sensor 120 may include an array of microphone elements arranged throughout the cabin. The audio sensor 120 may be configured to receive audio signals from within the environment, such as acoustic utterances of words, phrases, or commands spoken by the occupants. The audio sensor 120 may include an audio preprocessor configured to provide audio signal processing features (including amplification, conversion, data processing, etc.) to the processor 106. The audio sensor 120 may also be used for other environmental features, such as active noise cancellation, hands-free interfaces, etc. The signal from the audio sensor 120 may facilitate speech recognition from the audio received via the audio sensor 120 according to the grammar associated with the available commands and voice prompt generation.

[0022] The voice assistant system 100 may optionally include at least one image sensor 122 within the environment. The image sensor 122 may be a position sensor and / or a camera configured to detect the user's gaze direction, occupant posture, etc. The image sensor 122 may monitor the driver's head position, as well as detect any other movement of the occupant, such as the movement of the user's arm or hand, etc. In the example of a camera, the camera may provide imaging data captured of the vehicle 104 occupant to indicate certain movements made by a particular occupant. The image sensor 122 may be able to capture still images, videos, and / or detect user head, eye, and body movements. In some examples, the image sensor 122 may include multiple imaging elements, and the imaging data may be used for qualitative analysis. For example, imaging data from the image sensor 122 may be used to determine whether the user is looking at a certain location or vehicle display 126. Additionally or alternatively, the imaging data may also be supplemented with timing information because it is related to driver movement.

[0023] The voice assistant system 100 may also include an audio system that provides audio playback functionality through speakers 124 or headphones located within the environment. Audio playback may include audio from a variety of sources such as radio (including satellite radio), decoded amplitude modulation (AM) or frequency modulation (FM) radio signals, and audio signals from compact disk (CD) or digital versatile disk (DVD) audio playback, streaming audio from a mobile device, commands from a navigation system, etc. Audio playback may also include sounds or text-to-speech (TTS) generated by the voice assistant system 100 to be heard by occupants of the environment.

[0024] The voice assistant system 100 may also include various displays 126 configured to provide visual output to occupants of the environment. The displays 126 may include user interfaces, including a head-up display (HUD), a center console display, steering wheel buttons, etc. In some examples, the displays 126 may include input features, such as a touch screen, which is configured to receive user input in addition to displaying information.

[0025] The voice assistant system 100 may also include various physiological sensors 128. In one example, the physiological sensors 128 may include one or more pressure sensors, for example, to measure the firmness of the occupant's grip on the steering wheel. In another example, the physiological sensors 128 may include one or more skin conductance sensors. For example, the skin conductance sensor may be located on a steering wheel and configured to capture the skin conductance of an occupant holding the steering wheel. In another possibility, the skin conductance sensor may be located on a smart watch coupled to the voice assistant system 100 (e.g., wirelessly) to provide skin conductance information to the voice assistant system 100 of the wearer of the smart watch. In yet another example, the temperature of an occupant of the vehicle 104 may be measured using one or more temperature sensors, such as via an infrared or red channel of a camera (e.g., image sensor 122, another sensor, etc.).

[0026] At least a subset of the components of the voice assistant system 100 may operate together to perform the operations of the emotion sensing processing system 130. The emotion sensing processing system 130 may include various components of the voice assistant system 100, such as the processor 106, the computer-readable storage medium 108, the audio sensor 120, the image sensor 122, the speaker 124, the display 126, the physiological sensor 128, etc. The emotion sensing processing system 130 may include one or more input and output devices for communicating with the emotion sensing processing system 130. Figure 1 The other elements shown exchange data that is processed by the emotion sensing processing system 130 .

[0027] Figure 2Other aspects of elements of the emotion-aware processing system 130 are shown. As shown, the emotion-aware processing system 130 may include a wake word detector 202, one or more ASR models 204, an emotion detector 206, an emotion customization engine 208, and a voice control session recognizer 210.

[0028] The wake-up word detector 202 may receive a signal from the audio sensor 120 indicating a sound captured in the environment. The wake-up word detector 202 may include any suitable system capable of receiving a signal representing a sound received from the audio sensor 120, and performing wake-up word recognition using the received signal. The wake-up word detector 202 may perform speech recognition on the acoustic signal received by the audio sensor 120 to identify whether the wake-up word is spoken. For example, the wake-up word detector 202 may include an ASR model 204 that is configured to analyze the sound signal to determine whether the sound includes the utterance of the wake-up word. In an example, the ASR model 204 may generate a confidence value that the analyzed acoustic signal includes the utterance of the wake-up word. This confidence value may be compared to a threshold, above which detection of the wake-up word may be considered to have occurred.

[0029] The emotion detector 206 may also receive a signal indicating the sound captured in the environment from the audio sensor 120. In one example, the emotion detector 206 may analyze the manner in which the speech is spoken. This may include extracting a set of features (e.g., prosody, spectrum, cepstrum, voice quality, etc.) and learning a mapping between these features and corresponding target emotions. These target emotion representations may be categorical (e.g., neutral, happy, angry, fearful, angry, surprised, disgusted, another target emotion category, etc.) and / or dimensional (e.g., valence indicating the pleasantness of the stimulus, arousal indicating the intensity of the emotion caused by the stimulus, dominance indicating the degree of control exerted by the stimulus, etc.). In another example, the emotion detector 206 may learn features inherent in the mapping directly from the raw waveform data of the signal received from the audio sensor 120. The emotion detector 206 may generate a confidence value that the analyzed acoustic signal matches the target emotion for each corresponding target emotion. The most likely prediction of the emotion detector 206 may be considered as the predicted emotion of the received signal.

[0030] In some examples, the emotion detector 206 may be configured to detect the user's emotions additionally or alternatively based on information received from the image sensor 122. In one example, the raw data or processed feature data of the user detected from the image sensor 122 may be applied to a machine learning model to help predict the user's possible emotional state. For example, the data from the image sensor 122 may include information such as the state of the user's facial features (smile, frown, tears, red eyes, etc.), the user's voice changes, etc., which may be useful clues to the user's emotional state.

[0031] In some examples, the emotion detector 206 can be configured to detect the user's emotions based on information received from the physiological sensor 128 in addition or alternatively. In one example, the raw data or processed feature data of the user detected from the physiological sensor 128 can be applied to the machine learning model to help predict the user's possible emotional state. For example, the data from the physiological sensor 128 may include information such as how firmly the steering wheel is gripped, skin conductance, etc., which can be useful clues to the user's emotional state.

[0032] In some examples, emotion detector 206 may use a fused representation of audio, video, and / or psychological features to generate a confidence value for each respective target emotion to increase the confidence of the detected emotion compared to using only audio, only video, or only psychological features.

[0033] The emotion detector 206 may be configured to operate in parallel with the operation of the wake word detector 202 to allow the emotion-aware processing system 130 to derive the user's emotional state from WuW. Since most emotions are conveyed in the first point of contact with the emotion-aware processing system 130, this may allow the emotional state to be ready after the user's first interaction with the emotion-aware processing system 130. In addition, the emotion detector 206 may be relatively easier to train because the emotion detector 206 may be trained to detect emotional states based on a limited corpus of WuW, rather than having to be trained to recognize emotions from general speech input.

[0034] The emotion customization engine 208 can be configured to utilize the detected emotional state to customize further interactions of the user with the emotion sensing processing system 130. In one example, the emotion customization engine 208 can utilize the detected emotional state to signal the empathy capabilities of the speech system through visual feedback or acoustic feedback (referred to herein as ear tags).

[0035] In another example, the emotion customization engine 208 can utilize the detected emotion state to load a specific ASR / NLU model or configuration corresponding to the detected emotion context to improve recognition. For example, the execution of the ASR / NLU system can be adjusted according to the emotion state through an acoustic model, a language model weight set, a specific language model, etc. corresponding to the detected emotion state.

[0036] In yet another example, the emotion customization engine 208 can utilize the detected emotional state to modify the manner and style of the UI or VA response, such as its verbosity or its empathic response (e.g., submissive behavior in the face of aggressive user input). In addition, the TTS model used by the emotion-aware processing system 130 can also be configured. For example, if the user is in a good mood, a first TTS model that provides a lively response can be used, but if the user is in a bad mood, a sad TTS model that provides a simple response can be used instead.

[0037] The voice control session recognizer 210 may be configured to perform a voice control session triggered by the wake word detector 202 as configured by the emotion customization engine 208. For example, using the voice control session recognizer 210, a user may ask for directions, locate a business, order food, or any of a variety of speech conversation tasks.

[0038] Figure 3 An example process 300 for providing an interface customized for a detected emotional state of a user is shown. In an example, the process 300 can be performed by components of the emotion perception processing system 130 in the context of the voice assistant system 100.

[0039] At operation 302, the emotion sensing processing system 130 receives signals from the audio sensor 120. In one example, the audio sensor 120 may generate audio signals indicative of audio waves in an environment and may provide those signals to the processor 106. In some cases, the environment is a cabin of a vehicle 104. In other cases, the environment is a room of a building or an outdoor venue. In other cases, the environment is an augmented reality or virtual reality living environment.

[0040] At operation 304 , the emotion-aware processing system 130 detects WuW in the audio signal. In an example, the emotion-aware processing system 130 may utilize the WuW detector 202 to identify whether a WuW is located in the same audio signal captured by the audio sensor 120 .

[0041] At operation 306, the emotion-aware processing system 130 detects emotions in the audio signal. In one example, the emotion-aware processing system 130 may utilize the emotion detector 206 to identify the emotions of the same audio signal detected as including WuW. In another example, the emotion detector 206 may further utilize additional information sources, such as image data from the image sensor 122 and / or physiological data from the physiological sensor 128, to enhance the recognition of the user's emotions.

[0042] At operation 308, the emotion-aware processing system 130 configures the emotion-aware processing system 130 according to the emotion detected at operation 306. For example, the audio user interface can play an ear tag of a short melodic tone to confirm that the voice control session recognizer 210 is now listening to the user's speech. The specific melody or chord played can be changed based on the emotion detected in the WuW. This can allow the emotion-aware processing system 130 to provide initial emotion customization in the first response returned to the user from the emotion-aware processing system 130. In one example, in response to the detected emotion indicating that sadness is detected, the ear tag can be customized to play a longer main chord to make the user happy. In another example, in response to the detected emotion indicating that happiness is detected, the ear tag can be customized to play shorter major chords distributed over more octaves.

[0043] As another possibility, instead of the indication provided via the audio user interface or in addition to the indication provided via the audio user interface, the visual user interface can provide an alternative interface interface of complementary visual feedback. In an example, different colors and / or symbols can be displayed to display 126 based on the detected emotion. In a more specific example, in response to the detected emotion indication detecting anger or aggressiveness, red visual feedback can be displayed. As another possibility, in the case where the detected emotion is more resolute, the ear tag can be replaced with verbal confirmation to solve the user's resolute condition.

[0044] In one example, the emotion-aware processing system 130 may dynamically load a context-specific acoustic model (AM) based on the detected emotion. For example, the emotion-aware processing system 130 may use an AM specific to anger in response to a detected emotion indicating anger. In another example, the emotion-aware processing system 130 may replace a specific layer or transformation in a deep neural network (DNN) based on the detected emotion.

[0045] In another example, the emotion-aware processing system 130 may dynamically load a context-specific language model (LM) and / or LM weight set based on the detected emotion. In one example, if an emotion of excitement or anger is detected, the emotion-aware processing system 130 may load a LM to facilitate recognition of curse words that may not be recognized using a LM loaded in a happy emotional state.

[0046] In yet another example, the emotion-aware processing system 130 may dynamically load an NLU model based on the detected emotion. In one example, the emotion-aware processing system 130 may load a specific NLU model to facilitate and / or support the emotion-aware NLU model for recognition of user input after WuW.

[0047] As another possibility, the verbosity of the prompt of the emotion-aware processing system 130 can be changed based on the detected emotion. In one example, the verbosity can be changed based on the detected emotion to select from a no answer prompt for user input, a few answer prompts for user input, a medium answer prompt for user input, or a long answer prompt for user input. In another example, the TTS style of the answer prompt can be changed using an emotional TTS specific to the detected emotion (e.g., a first TTS for happy users, a different second TTS for unhappy users, etc.). In yet another example, in response to a detected emotion indicating anger, a prompt can be played in an apologetic TTS style. In another example, the wording and tone of the response can be adjusted based on the detected emotion, for example, in response to detecting an aggressive user input, or an aggressive compliance under an aggressive input.

[0048] In yet another example, a modality may be initiated based on detected emotions to support answer prompts. As one possibility, in response to emotions detected in WuW, the emotion-aware processing system 130 may be adapted to use one or more of visual feedback, haptic or tactile feedback, olfactory feedback, taste feedback, etc.

[0049] At operation 310, the emotion-aware processing system 130 provides an initial confirmation to the user. This initial confirmation to the user may have been configured at operation 308 based on the emotional state detected at operation 306. In an example, the emotion-aware processing system 130 may provide a confirmation that WuW has been detected and a voice control session has been initiated. The confirmation may include one or more of a visual confirmation (e.g., via the display 126), an ear tag (e.g., provided to the speaker 124), and / or a speech confirming the session (e.g., provided to the speaker 124).

[0050] At operation 312, the emotion-aware processing system 130 executes a voice control session configured according to the detected emotion. In one example, the voice control session recognizer 210 may execute a voice control session triggered by the wake word detector 202 as detected at operation 306, which is configured by the emotion customization engine 208 at operation 308. After operation 312, the process 300 ends.

[0051] Thus, in parallel with triggering the ASR / NLU instance to recognize what the user said after WuW, the user's underlying emotional state can be derived, at least in part, from WuW. Since most emotions are conveyed in the first point of contact with the emotion-aware processing system 130, this can allow the emotional state to be ready after the user's first interaction with the emotion-aware processing system 130. In addition, the emotion detector 206 can be relatively easier to train because the emotion detector 206 can be trained to detect emotional states based on a more limited corpus of WuW, rather than any general speech input. The emotion customization engine 208 can be configured to utilize the detected emotional state to customize the user's further interactions with the emotion-aware processing system 130. Therefore, the detection of emotions of WuW spoken by the user can be used to implement and build empathic user interfaces.

[0052] The description of the various embodiments has been presented for purposes of illustration, but is not intended to be exhaustive or limited to the disclosed embodiments. Many modifications and variations will be apparent to those of ordinary skill in the art without departing from the scope and spirit of the described embodiments.

[0053] The various aspects of the present embodiment may be embodied as a system, method or computer program product. Therefore, the various aspects of the present disclosure may take the form of a complete hardware embodiment, a complete software embodiment (including firmware, resident software, microcode, etc.) or an embodiment of a combination of software and hardware, which may generally be referred to as a "module" or "system" in this article. In addition, the various aspects of the present disclosure may take the form of a computer program product embodied in one or more computer-readable media, and the one or more computer-readable media have a computer-readable program code embodied therein.

[0054] Any combination of one or more computer-readable media may be utilized. A computer-readable medium may be a computer-readable signal medium or a computer-readable storage medium. A computer-readable storage medium may be, for example, but not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, device, or apparatus, or any suitable combination of the foregoing. More specific examples of computer-readable storage media (a non-exhaustive list) include the following: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing. In the context of this document, a computer-readable storage medium may be any tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, device, or apparatus.

[0055] Aspects of the present disclosure are described above with reference to flowchart illustrations and / or block diagrams of methods, devices (systems) and computer program products according to embodiments of the present disclosure. It should be understood that each frame of the flowchart illustration and / or block diagram and the combination of frames in the flowchart illustration and / or block diagram can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device enable the function / action specified in one or more frames of the flowchart and / or block diagram to be implemented. Such a processor can be, but is not limited to, a general-purpose processor, a special-purpose processor, a special-purpose processor or a field programmable processor.

[0056] The flowcharts and block diagrams in the accompanying drawings illustrate the possible implementation architectures, functions and operations of the systems, methods and computer program products according to various embodiments of the present disclosure. In this regard, each box in the flowchart or block diagram may represent a module, fragment or portion of a code, which includes one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative embodiments, the functions annotated in the box may not occur in the order annotated in the figure. For example, two boxes shown in succession can actually be executed substantially simultaneously, or these boxes can sometimes be executed in reverse order, depending on the functions involved. It will also be noted that each box illustrated in the block diagram and / or flowchart and the combination of boxes in the block diagram and / or flowchart can be implemented by a combination of a dedicated hardware-based system or dedicated hardware and computer instructions that performs a specified function or action.

[0057] Although exemplary embodiments are described above, these embodiments are not intended to describe all possible forms of the present invention. On the contrary, the words used in the specification are descriptive rather than restrictive, and it should be understood that various changes may be made without departing from the spirit and scope of the present invention. In addition, the features of various implementation embodiments may be combined to form further embodiments of the present invention.

Claims

1. A system for providing an interface customized for a detected emotional state of a user, comprising: at least one microphone configured to generate an audio signal indicative of a spoken word, phrase, or command; and A processor programmed to: receiving the audio signal from the at least one microphone, Detecting a wake-up word (WuW) in the audio signal; detecting emotions in the audio signal containing the WuW, configuring an emotion-aware processing system based on the detected emotion; A voice control session is performed using the emotion-aware processing system configured according to the detected emotion.

2. The system according to claim 1, wherein: The processor is also programmed to: Analyzing the audio signal using an automatic speech recognition (ASR) model to generate a confidence value that the audio signal includes the utterance of WuW; Comparing the confidence value with a threshold, where if the confidence value is higher than the threshold, the detection of the WuW is considered to have occurred; In response to the confidence value exceeding the threshold, the WuW and the emotion are detected in the audio signal.

3. The system according to claim 1, wherein: The processor is also programmed to: For each of a plurality of target emotions, generating a confidence value that the audio signal matches the corresponding target emotion; and The emotion is identified as a target emotion among the plurality of target emotions corresponding to a most probable confidence value among the confidence values.

4. The system according to claim 3, wherein: The processor is also programmed to: extracting a feature set from the audio signal including the WuW using an emotion detector, and comparing the feature set with the plurality of target emotions to determine the confidence value, The feature set includes one or more of prosodic features, spectral features, cepstrum features and / or speech quality features.

5. The system according to claim 4, wherein: The processor is further programmed to include raw data and / or processed feature data of the user detected from an image sensor as a further input to the emotion detector to enhance the detection of the emotion in the audio signal.

6. The system according to claim 4, wherein: The processor is further programmed to include raw data and / or processed feature data of the user detected from a physiological sensor as a further input to the emotion detector to enhance the detection of the emotion in the audio signal.

7. The system according to claim 3, wherein: The plurality of target emotions include categorized emotions including one or more of neutral emotion, happy emotion, angry emotion, or another target emotion category.

8. The system according to claim 3, wherein: The plurality of target emotions include dimensional emotions including components of one or more of valence, arousal, and dominance.

9. The system according to claim 1, wherein: Configuring the emotion-aware processing system based on the detected emotion includes one or more of the following: Playing ear tags based on detected emotions; Providing visual feedback based on detected emotions; or A modality of feedback to the user is adjusted based on the detected emotion.

10. The system according to claim 1, wherein: Configuring the emotion-aware processing system based on the detected emotion includes one or more of the following: Dynamically load context-specific acoustic models (AMs) based on detected emotions; Dynamically loading a context-specific language model (LM) and / or LM weight set based on the detected emotion; or Dynamically load a natural language understanding (NLU) model based on detected sentiment.

11. The system according to claim 1, wherein: Configuring the emotion-aware processing system based on the detected emotion includes one or more of the following: adjusting the verbosity of prompts of the voice control session based on the detected emotion; or The wording and intonation of responses of the voice control session are adjusted based on the detected emotion.

12. A method for providing an interface customized to a detected emotional state of a user, comprising: receiving an audio signal from at least one microphone, the audio signal being indicative of a spoken word, phrase, or command; Detecting a wake-up word (WuW) in the audio signal; detecting emotions in the audio signal containing the WuW; configuring an emotion-aware processing system based on the detected emotion; and A voice control session is performed using the emotion-aware processing system configured according to the detected emotion.

13. The method according to claim 12, further comprising: Analyzing the audio signal using an automatic speech recognition (ASR) model to generate a confidence value that the audio signal includes the utterance of WuW; Comparing the confidence value with a threshold, where if the confidence value is higher than the threshold, the detection of the WuW is considered to have occurred; and In response to the confidence value exceeding the threshold, the WuW and the emotion are detected in the audio signal.

14. The method according to claim 12, further comprising: For each of a plurality of target emotions, generating a confidence value that the audio signal matches the corresponding target emotion; and The emotion is identified as a target emotion among the plurality of target emotions corresponding to a most probable confidence value among the confidence values.

15. The method according to claim 14, further comprising: extracting a feature set from the audio signal including the WuW using an emotion detector, and comparing the feature set with the plurality of target emotions to determine the confidence value, The feature set includes one or more of prosodic features, spectral features, cepstrum features, and speech quality features.

16. The method according to claim 15, further comprising: Raw data and / or processed feature data of the user detected from an image sensor is included as a further input to the emotion detector to enhance the detection of the emotion in the audio signal.

17. The method according to claim 15, further comprising: Raw data and / or processed feature data of the user detected from a physiological sensor is included as a further input to the emotion detector to enhance the detection of the emotion in the audio signal.

18. The method according to claim 14, wherein: The plurality of target emotions include categorized emotions including one or more of neutral emotion, happy emotion, angry emotion, or another target emotion category.

19. The method according to claim 14, wherein: The plurality of target emotions include dimensional emotions including components of one or more of valence, arousal, and dominance.

20. The method according to claim 12, wherein: Configuring the emotion-aware processing system based on the detected emotion includes one or more of the following: Playing ear tags based on detected emotions; Providing visual feedback based on detected emotions; or A modality of feedback to the user is adjusted based on the detected emotion.

21. The method according to claim 12, wherein: Configuring the emotion-aware processing system based on the detected emotion includes one or more of the following: Dynamically load context-specific acoustic models (AMs) based on detected emotions; Dynamically loading a context-specific language model (LM) and / or LM weight set based on the detected emotion; or Dynamically load a natural language understanding (NLU) model based on detected sentiment.

22. The method according to claim 12, wherein: Configuring the emotion-aware processing system based on the detected emotion includes one or more of the following: adjusting the verbosity of prompts of the voice control session based on the detected emotion; or The wording and intonation of responses of the voice control session are adjusted based on the detected emotion.

Images