Electronic device and non-transitory storage medium

Abstract

A method includes detecting activation of a smart assistant on an electronic device, waking up the smart assistant from a sleep mode in response to the activation, and determining an amount of vocabulary that the smart assistant functions during a listening mode based on a type of the activation.

Description

[0001] This application is a divisional application of the invention patent application with the application date of January 10, 2020, application number 202080023595.6, and invention name “Multimodal interaction with an intelligent assistant in a voice command device”. Technical Field

[0002] The present disclosure relates generally to voice command devices, and more particularly to a method and system for emergency multimodal interaction with an intelligent assistant in a voice command device. Background Art

[0003] A voice user interface provides a way to interact with an intelligent personal assistant (IPA) or virtual assistant (VA) operating on a voice command device. In layman's terms, an IPA or VA is referred to as an "intelligent assistant". A voice command device is a device that can be controlled via a voice user interface (i.e., a voice-controlled device). A voice user interface enables spoken human interaction with a device, using speech recognition to understand voice commands (i.e., spoken commands) and questions, and using text-to-speech to output responses. With advances in automatic speech recognition (ASR) and natural language understanding (NLU), voice user interfaces are becoming increasingly popular in devices such as, but not limited to, automobiles, mobile devices (e.g., smartphones, tablets, watches, etc.), appliances (e.g., washers, dryers, etc.), entertainment devices (e.g., televisions, smart speakers, etc.), and the like. Summary of the invention

[0004] Technical issues

[0005] Embodiments of the present disclosure provide a method and apparatus (system) for multi-modal interaction with an intelligent assistant.

[0006] Solution to the problem

[0007] One embodiment provides a method comprising: detecting activation of an intelligent assistant on an electronic device; waking the intelligent assistant from a sleep mode in response to the activation; and determining a vocabulary of the intelligent assistant that operates during a listening mode based on a type of the activation.

[0008] Another embodiment provides an electronic device including at least one sensor unit, at least one input / output (I / O) unit, at least one processor, and a non-transitory processor-readable memory device storing instructions that, when executed by the at least one processor, cause the at least one processor to perform operations. The operations include detecting a physical interaction with the electronic device via the at least one sensor unit, and activating an intelligent assistant on the electronic device from a sleep mode in response to the physical interaction.

[0009] One embodiment provides a non-transitory processor-readable medium comprising a program that, when executed by a processor, performs a method comprising: detecting activation of an intelligent assistant on an electronic device; waking the intelligent assistant from a sleep mode in response to the activation; and determining a vocabulary in which the intelligent assistant functions during a listening mode based on a type of the activation.

[0010] One embodiment provides an electronic device, comprising: a display; a microphone; a speaker; at least one sensor; at least one processor coupled to the display, the microphone, the speaker and the at least one sensor; and a memory storing instructions, wherein the instructions, when executed by the at least one processor, cause the electronic device to: provide first feedback via at least one of the display or the speaker based on a first occurrence of a first event, the first event comprising identifying a wake-up word from a first voice signal obtained via the microphone; generate a first timeout based on no voice signal being obtained via the microphone within a first duration after providing the first feedback; provide second feedback via at least one of the display or the speaker based on a first occurrence of a second event, the second event comprising detecting an external object within a proximity of the electronic device via the at least one sensor, and the second feedback is different from the first feedback, and generate a second timeout based on no voice signal being obtained via the microphone within a second duration after providing the second feedback, wherein the first duration is longer than the second duration.

[0011] One embodiment provides a non-transitory storage medium storing one or more programs, wherein the one or more programs include computer-executable instructions, which, when executed by at least one processor of an electronic device, cause the electronic device to: provide first feedback via at least one of a display of the electronic device or a speaker of the electronic device based on a first occurrence of a first event, wherein the first event includes recognizing a wake-up word from a first voice signal obtained via a microphone of the electronic device; generate a first timeout based on the fact that no voice signal is obtained via the microphone within a first duration after providing the first feedback; provide second feedback via at least one of the display or the speaker based on a first occurrence of a second event, wherein the second event includes detecting an external object within a proximity of the electronic device via at least one sensor of the electronic device, and the second feedback is different from the first feedback, and generate a second timeout based on the fact that no voice signal is obtained via the microphone within a second duration after providing the second feedback, wherein the first duration is longer than the second duration.

[0012] These and other aspects and advantages of one or more embodiments will become apparent from the following detailed description which, when taken in conjunction with the accompanying drawings, illustrates, by way of example, the principles of the one or more embodiments. BRIEF DESCRIPTION OF THE DRAWINGS

[0013] For a fuller understanding of the nature and advantages of the embodiments, as well as the preferred mode of use, reference should be made to the following detailed description read in conjunction with the accompanying drawings, in which:

[0014] Figure 1 An example computing architecture for implementing emergency multimodal interaction with an intelligent assistant embodied in an electronic device 50 in one or more embodiments is shown;

[0015] Figure 2 An example cluster of electronic devices in accordance with one or more embodiments is shown;

[0016] Figure 3 An example voice user interface system embodied in an electronic device in one or more embodiments is shown;

[0017] Figure 4 A first example scenario involving multimodal interaction with an intelligent assistant embodied in multiple electronic devices is shown in one or more embodiments;

[0018] Figure 5 A second example scenario involving multimodal interaction with an intelligent assistant embodied in multiple electronic devices is shown in one or more embodiments;

[0019] Figure 6A third example scenario involving multimodal interaction with an intelligent assistant embodied in multiple electronic devices is shown in one or more embodiments;

[0020] Figure 7 A fourth example scenario involving multimodal interaction with an intelligent assistant embodied in multiple electronic devices is shown in one or more embodiments;

[0021] Figure 8 A fifth example scenario involving multimodal interaction with an intelligent assistant embodied in multiple electronic devices is shown in one or more embodiments;

[0022] Fig. 9 A sixth example scenario involving multimodal interaction with an intelligent assistant embodied in multiple electronic devices in one or more embodiments is shown;

[0023] Fig.10 A seventh example scenario involving multimodal interaction with an intelligent assistant embodied in multiple electronic devices is shown in one or more embodiments;

[0024] Fig.11 is a flow chart of an example process for implementing multimodal interaction with an intelligent assistant in an electronic device in one or more embodiments;

[0025] Fig.12 is a flow chart of an example process 910 for implementing physical event activation of an intelligent assistant in an electronic device in one or more embodiments; and

[0026] Fig.13 is a high-level block diagram illustrating an information processing system including a computer system useful for implementing the disclosed embodiments. DETAILED DESCRIPTION

[0027] The following description is for the purpose of illustrating the general principles of one or more embodiments and is not meant to limit the inventive concepts claimed herein. In addition, the specific features described herein may be used in combination with other described features in each of various possible combinations and permutations. Unless otherwise specifically defined herein, all terms will be given their broadest possible interpretation, including the meanings implied in the specification and the meanings understood by those skilled in the art and / or defined in dictionaries, papers, etc.

[0028] One or more embodiments generally relate to voice command devices, and more particularly, to a method and system for emergency multimodal interaction with an intelligent assistant in a voice command device. One embodiment provides a method comprising: detecting activation of an intelligent assistant on an electronic device; waking the intelligent assistant from a sleep mode in response to the activation; and determining a vocabulary of the intelligent assistant during a listening mode based on a type of the activation.

[0029] Another embodiment provides an electronic device including at least one sensor unit, at least one input / output (I / O) unit, at least one processor, and a non-transitory processor-readable memory device storing instructions that, when executed by the at least one processor, cause the at least one processor to perform operations. The operations include detecting a physical interaction with the electronic device via the at least one sensor unit, and activating an intelligent assistant on the electronic device from a sleep mode in response to the physical interaction.

[0030] One embodiment provides a non-transitory processor-readable medium comprising a program that, when executed by a processor, performs a method comprising: detecting activation of an intelligent assistant on an electronic device; waking the intelligent assistant from a sleep mode in response to the activation; and determining a vocabulary in which the intelligent assistant functions during a listening mode based on a type of the activation.

[0031] Because speech is typically present in an environment where one or more voice command devices are present, and only a small portion of speech is intended for a specific voice command device, one of the challenges encountered by voice command devices is determining which speech is intended for that device.

[0032] Conventionally, a user can indicate that an utterance from the user (i.e., a voice command) is intended for a specific voice command device by interacting with a voice input button. For example, the voice input button can be a dedicated physical button or another type of I / O component coupled to or integrated with the device. As another example, the voice input button can be a software component of a graphical user interface (GUI) displayed on a display coupled to or integrated with the device.

[0033] Another conventional solution is for the user to produce a specific utterance (e.g., a word or phrase) that instructs a specific voice command device to pay attention to the voice command following the specific utterance. In layman's terms, such a specific utterance is referred to as a "wake-up word." For example, using the GALAXY® Note9™ smartphone from Samsung®, a user can press a dedicated physical button on the side of the phone and issue a voice command. Alternatively, the user can first say a specific utterance, such as "Hi Bixby®," and then follow the specific utterance with a voice command.

[0034] One or more embodiments provide a voice user interface system for activating a voice agent (i.e., an intelligent assistant) operating on an electronic device based on one or more physical interactions with the device that a user typically performs when using the device. In one embodiment, the voice user interface system is configured to listen for meaningful voice interactions in tandem with how the device is physically manipulated when appropriate. Unlike traditional solutions that require explicit activation from the user (e.g., via interaction with a voice input button or speaking a specific word), the voice user interface system provides automatic wakeup of the voice agent, where the wakeup is implicit (i.e., no explicit activation from the user is required).

[0035] Figure 1 An example computing architecture 10 is shown for implementing emergency multimodal interaction with an intelligent assistant embodied in an electronic device 50 in one or more embodiments. The computing architecture 10 includes an electronic device 50, which includes resources, such as one or more processor units 55 and one or more memory units 60. One or more applications can execute / operate on the electronic device 50 using the resources of the electronic device 50.

[0036] Examples of electronic devices 50 include, but are not limited to, desktop computers, mobile electronic devices (e.g., tablets, smartphones, laptops, etc.), wearable devices (e.g., smart watches, etc.), Internet of Things (IoT) devices, smart appliances (such as smart TVs, smart ovens, smart washing machines, smart lights, etc.).

[0037] In one embodiment, the electronic device 50 includes one or more sensor units 70 integrated in or coupled to the electronic device 50, such as a camera, a microphone, a GPS, a motion sensor, etc. The sensor unit 70 can be used to capture content and / or sensor-based context information. For example, an application on the electronic device 50 can utilize at least one sensor unit 70 to capture content and / or sensor-based context information, such as a microphone for audio data (e.g., voice commands, ambient noise, etc.), a camera for image data (e.g., still and / or video images of the environment surrounding the electronic device 50, etc.), a GPS for location data (e.g., location coordinates), a motion sensor for proximity / motion data (e.g., data indicating users 30 within the proximity of the electronic device 50), etc.

[0038] In one embodiment, the electronic device 50 includes one or more I / O units 65 integrated in or coupled to the electronic device 50. In one embodiment, the one or more I / O units 65 include, but are not limited to, a physical user interface (PUI) and / or a GUI, such as a keyboard, a keypad, a touch interface, a touch screen, a knob, a button, a display screen, etc. In one embodiment, the user 30 can utilize at least one I / O unit 65 to configure one or more user preferences, configure one or more parameters (e.g., user permissions), provide input (e.g., search queries), etc.

[0039] In one embodiment, the electronic device 50 is a voice command device. One or more applications on the electronic device 50 include, but are not limited to, a voice user interface system 200 configured to facilitate hands-free user interaction with the electronic device 50. In one embodiment, the system 200 provides a voice agent 205, wherein the voice agent 205 is an intelligent assistant configured to allow the user 30 to control the electronic device 50 via voice.

[0040] In one embodiment, the voice agent 205 has at least two different operating modes: a sleep mode, in which the voice agent 205 is in a sleep state (i.e., inactive or in a powered-off state); and a listening mode, in which the voice agent 205 is in an awake state (i.e., active or in a powered-on state) and listens for voice commands from the user 30 (e.g., voice commands captured via at least one sensor unit 70).

[0041] As described in detail later herein, in one embodiment, system 200 includes a listening agent 250 ( Figure 3 ), the listening agent 250 is configured to: (1) detect one or more activation conditions for waking (i.e., activating) the voice agent 205 from the sleep mode, and (2) trigger the voice agent 205 to transition from the sleep mode to the listening mode in response to detecting the one or more activation conditions.

[0042] In one embodiment, the voice agent 205 can be activated via different types of activation methods. For example, in one embodiment, the voice agent 205 is explicitly activated via wake-up word activation. Wake-up word activation is an example activation condition that requires explicit utterance ("user utterance") of a specific wake-up word from the user 30 for the electronic device 50. In one embodiment, the voice agent 205 is configured to wake up from sleep mode in response to the system 200 detecting (e.g., via the listening agent 250 and at least one sensor unit 70) a user utterance of the wake-up word. For example, in response to a microphone capturing a user utterance of the wake-up word, the voice agent 205 is awakened from sleep mode.

[0043] As another example, in one embodiment, the voice agent 205 is automatically activated via physical event activation without requiring user speech. Physical event activation is an example activation condition that requires physical interaction as is typical of the user 30 using the electronic device 50. In one embodiment, the voice agent 205 is configured to automatically wake up from sleep mode in response to the system 200 detecting (e.g., via the listening agent 250 and at least one sensor unit 70 and / or at least one I / O unit 65) a physical interaction with the electronic device 50. Examples of different physical interactions that can automatically wake the voice agent 205 from sleep mode include, but are not limited to, mechanical interactions (i.e., mechanical actuations) with movable / rotatable / adjustable components of the electronic device 50, such as door closing activation (e.g., user 30 closes the door of the electronic device 50), proximity interaction (e.g., user 30 is within physical proximity of the electronic device 50, user 30 has entered a room including the electronic device 50), user interface (UI) activation (such as actuation of a PUI (e.g., user 30 interacts with a knob, button, and / or another hardware I / O unit 65) or actuation of a GUI (e.g., user 30 interacts with a touch screen and / or another software I / O unit 65)).

[0044] For example, if the electronic device 50 includes a smart washing machine, in response to at least one sensor unit 70 detecting that the door of the washing chamber of the washing machine is closed, the voice agent 205 is automatically awakened from sleep mode, thereby eliminating the need for the user 30 to explicitly issue a wake-up word for the electronic device 50 to wake up the voice agent 205.

[0045] In one embodiment, the system 200 is configured to provide feedback to the user 30 using different interactive feedback methods based on the type of activation method used to activate the voice agent 205 (e.g., visual and / or voice feedback and / or responses indicating the current mode of the voice agent 205, etc.).

[0046] In one embodiment, the system 200 is configured to control the number and types of actions that the electronic device 50 can invoke / trigger to perform via voice commands based on contextual information, such as, but not limited to, the context of the user 30, the context of the electronic device 50, such as the current state (i.e., current configuration), the environment of the user 30 and / or the electronic device 50, such as the time, day, temperature, weather, ambient lighting, location, etc.

[0047] In one embodiment, the system 200 is configured to determine the identity of the user 30 who generated the user speech detected via the listening agent 250 based on voice identification (ID), and control the actions that the user 30 can request the electronic device 50 to perform via voice based on the identity of the user 30. In one embodiment, the voice ID involves determining the voice pattern / quality (e.g., sound, prosody, dialect, pitch, etc.) of the user 30 based on the user speech, and comparing the voice pattern / quality with the voice pattern / quality of the recognized / identified voice recording (e.g., stored on one or more storage units 60). For example, if the electronic device 50 includes a smart washing machine, the system 200 can only allow a specific authorized user 30 (such as an adult in the home) to start the washing machine via voice after the system 200 detects that the door of the washing machine is closed, without having to activate the physical UI or GUI. Limiting the start of the washing machine to only voice commands from adults after detecting that the door of the washing machine is closed ensures safe operation of the washing machine. For example, such a restriction can prevent unauthorized users 30 (such as children) from being able to start the washing machine (e.g., after closing the door of the washing machine, the child may enter the washing machine and try to start the washing machine via voice).

[0048] In one embodiment, the system 200 is configured to identify that one or more user utterances detected by the monitoring agent 250 are not directed to the electronic device 50 based on the type of activation method and voice ID used to activate the voice agent 205, thereby reducing the possibility of unintentional or undesired actions on the electronic device 50 activated by the user voice.

[0049] In one embodiment, the system 200 is configured to adjust the vocabulary required by the system 200 to interpret user utterances to be solely context-dependent based on the type of activation method used to activate the voice agent 205 and / or the current state of the electronic device 50, thereby ensuring that only voice commands appropriate to the current state of the electronic device 50 are acted upon.

[0050] In one embodiment, the one or more applications on the electronic device 50 may also include one or more software mobile applications 90 loaded onto or downloaded to the electronic device 50, such as a camera application, a social media application, etc. The software mobile application 90 on the electronic device 50 may exchange data with the system 200.

[0051] In one embodiment, the electronic device 50 includes a communication unit 80 configured to exchange data with one or more remote devices 20 and / or one or more other electronic devices 50 via a communication network / connection 40 (e.g., a wireless connection (such as a Wi-Fi connection or a cellular data connection), a wired connection, or a combination of both). The communication unit 80 may include any suitable communication circuitry operable to connect to a communication network (e.g., the communication network 40) and exchange communication operations and media from the electronic device 50 to other devices connected to the communication network 40. The communication unit 80 may be operable to interface with the communication network 40 using any suitable communication protocol (e.g., Wi-Fi (e.g., IEEE 802.11 protocol), Bluetooth, high frequency systems (e.g., 900 MHz, 2.4 GHz, and 5.6 GHz communication systems), infrared, GSM, GSM plus EDGE, CDMA, quad band and other cellular protocols, VOIP, TCP-IP, or any other suitable protocol).

[0052] For example, the remote device 20 may include a remote server (e.g., a computer, device, or program that manages network resources, etc.) that provides an online platform for hosting one or more online services (e.g., online social media services, online data sources, etc.) and / or distributing one or more software mobile applications 90. As another example, the system 200 may be loaded onto or downloaded to the electronic device 50 from the remote device 20 that maintains and distributes updates of the system 200. As yet another example, the remote device 20 may include a cloud computing environment that provides a shared pool of configurable computing system resources and higher-level services.

[0053] Figure 2 An example cluster 100 of electronic devices 50 in one or more embodiments is shown. Cluster 100 includes a combination of different electronic devices 50 (such as electronic device 1, electronic device 2, ..., and electronic device n) that a user 30 can control via voice, where n is a positive integer.

[0054] In one embodiment, the cluster 100 of different electronic devices 50 may be located within proximity of each other (e.g., within one or more rooms in a home). For example, the cluster 100 includes at least one of the following electronic devices 50 located in a kitchen: a smart microwave, a smart stove, a smart oven, a smart dishwasher, a home hub, etc. As another example, the cluster 100 includes at least one of the following electronic devices 50 located in a laundry room: a smart washing machine, a smart dryer, etc.

[0055] In one embodiment, each electronic device 50 of the cluster 100 is configured to exchange data with one or more other electronic devices 50 of the cluster 100 via a communication network / connection (e.g., a wireless connection (such as a WiFi connection or a cellular data connection), a wired connection, or a combination of both).

[0056] In one embodiment, the user 30 may simultaneously activate all voice agents 205 of all electronic devices 50 of the cluster 100 via a wake word activation.

[0057] In one embodiment, the user 30 can automatically activate only one voice agent 205 of the electronic device 50 of the cluster 100 via a physical event activation involving only one electronic device 50. Since popular voice user interfaces typically share the same wake-up word across multiple electronic devices 50 (e.g., the wake-up word "Alexa®" is used to activate different Amazon® voice command devices, the wake-up word "Siri®" is used to activate different Apple® voice command devices, the wake-up phrase "Hey Google®" is used to activate different Android® voice command devices, etc.), physical event activation eliminates the problem of knowing which specific electronic device 50 is being activated by the wake-up word, thereby reducing the possibility of multiple electronic devices 50 being activated at the same time. Physical event activation eliminates the need for the user 30 to say the wake-up word, and also eliminates the need for the user 30 to specify which specific electronic device 50 should pay attention to the voice command.

[0058] In one embodiment, the electronic device 50 of the cluster 100 may operate as a master device to control one or more other electronic devices 50 of the cluster 100 (eg, triggering / invoking one or more other electronic devices 50 to perform one or more actions in response to one or more voice commands).

[0059] In one embodiment, an action performed at one electronic device 50 of the cluster 100 may trigger the voice agent 205 of another electronic device 50 of the cluster 100 to wake up from sleep mode. For example, an action performed at the first electronic device 50 of the cluster 100 may trigger the voice agent 205 of the second electronic device 50 of the cluster 100 operating as a master device to wake up from sleep mode and listen for voice commands directed to the first electronic device 50.

[0060] In one embodiment, a portable or mobile electronic device 50 may be added to or removed from the cluster 100 , such as a smart AI speaker or a smartphone.

[0061] Figure 3An example voice user interface system 200 is shown that is embodied in an electronic device 50 in one or more embodiments. A voice agent 205 operating in the electronic device 50 is implemented by one or more components of the voice user interface system 200.

[0062] In one embodiment, the system 200 includes a UI & sensor logic unit 210, which is configured to: (1) receive UI & sensor data captured by at least one I / O unit 65 and / or at least one sensor unit 70, wherein the UI & sensor data indicates one or more physical interactions with the electronic device 50, (2) generate one or more activation signals in response to the detected one or more physical interactions, wherein the one or more activation signals trigger the voice agent 205 to automatically wake up from sleep mode, and (3) generate control data indicating one or more adjustments / updates to the current state of the electronic device 50.

[0063] In one embodiment, the system 200 includes a wake-up word detection unit 240, which is configured to: (1) receive audio data (i.e., sounds in the data) captured by at least one sensor unit 70 (e.g., a microphone), wherein the audio data includes ambient noise and / or one or more user utterances, (2) determine whether the audio data includes explicit user utterances of a wake-up word for the electronic device 50, and (3) generate one or more activation signals in response to determining that the audio data includes explicit user utterances of the wake-up word, wherein the one or more activation signals trigger the voice agent 205 to wake up from a sleep mode.

[0064] In one embodiment, the system 200 includes a device control and state logic unit 220 that is configured to provide an electromechanical application programming interface (API) for communicating with one or more mechanical components ("device mechanisms") 230 of the electronic device 50. The device control and state logic unit 220 allows one or more device mechanisms 230 to be controlled via the electromechanical API based on control data (e.g., from the UI & sensor logic unit 210 and / or from the device action initiator 274).

[0065] In one embodiment, the device control and state logic unit 220 is configured to generate state data indicating at least one of: a current state of the electronic device 50 , or one or more recent actions performed by the electronic device 50 .

[0066] In one embodiment, the system 200 includes a listening agent 250. When the voice agent 205 is in sleep mode, the listening agent 250 is configured to: (1) receive one or more activation signals from the UI & sensor logic unit 210 and / or the wake-up word detection unit 240, and (2) wake up the voice agent 205 from the sleep mode in response to the one or more activation signals and place the voice agent 205 in the listening mode. As described above, the voice agent 205 can be explicitly awakened via wake-up word activation or automatically awakened via physical event activation.

[0067] When the voice agent 205 is in the listening mode, the listening agent 250 is configured to listen for one or more voice commands from the user 30. Specifically, the listening agent 250 is configured to: (1) receive audio data captured by at least one sensor unit 70 (e.g., a microphone), wherein the audio data includes ambient noise and / or one or more user utterances, and (2) analyze the audio data.

[0068] The listening agent 250 includes one or more components for analyzing audio data. In one embodiment, the listening agent 250 includes an ambient noise analysis unit 252, which is configured to analyze the audio data to determine whether the audio data includes ambient noise. In one embodiment, the listening agent 250 includes a speech capture unit 253, which is configured to: (1) analyze the audio data to determine whether the audio data includes user speech, and (2) in response to determining that the audio data includes user speech, capture / extract the user speech in the audio data as speech data (i.e., data including spoken language). As described in detail later in this document, the speech data can be forwarded to another component of the system 200 for automatic speech recognition (ASR).

[0069] In one embodiment, the listening agent 250 includes a voice ID analysis unit 255, which is configured to determine a user ID indicating the identity of the user 30 who produced the user speech included in the audio data based on the voice ID. As described in detail later in this document, the user ID can be forwarded to another component of the system 200 for determining one or more user permissions of the user 30.

[0070] In one embodiment, the listening agent 250 includes a speech start-stop unit 254, which is configured to: (1) wait and listen for user speech within a predetermined amount of time (i.e., a timing window), and (2) generate / issue a timeout when the predetermined amount of time has passed and a complete user speech is not available. In one embodiment, the listening agent 250 is configured to generate different types of timeouts, wherein each type of timeout has its own corresponding timing window. For example, in one embodiment, the listening agent 250 generates a no-speak timeout, indicating that no user speech was detected after the corresponding time window of the no-speak timeout has passed (i.e., the listening agent 250 detected the absence of a voice command during the time window). As another example, in one embodiment, the listening agent 250 generates a command-too-long timeout, indicating that the user speech detected after the corresponding time window of the command-too-long timeout has passed is too long (i.e., the user speech is incomplete).

[0071] In one embodiment, the system 200 maintains one or more activation rules, wherein the one or more activation rules adapt one or more algorithms for controlling one or more behaviors of the electronic device 50 based on the type of activation method (e.g., wake-up word activation or physical event activation) used to activate the voice agent 205. For example, in one embodiment, based on the type of activation method used to activate the voice agent 205, one or more parameters that control one or more behaviors of the listening agent 250 are selectively adjusted / adjusted.

[0072] In one embodiment, the listening agent 250 includes an activation adjustment table 251 configured to maintain one or more activation rules, wherein the one or more activation rules are used to adapt one or more parameters that control one or more behaviors of the listening agent 250 based on the type of activation method used to activate the voice agent 205. For example, in one embodiment, for the parameter controlling the timing window of the no-speak timeout, if the voice agent 205 is activated via a physical event activation (e.g., door closing), the timing window is shorter (e.g., 1.5 seconds), and if the voice agent 205 is activated via a wake-up word activation, the timing window is longer (e.g., 10 seconds).

[0073] In one or more embodiments, Table 1 below provides an example set of activation rules maintained in activation adjustment table 251 .

[0074] [Table 1]

[0075]

[0076] In one embodiment, the system 200 includes a language interpreter 260 configured to interpret the spoken language included in the user's speech. In one embodiment, the language interpreter 260 includes an ASR unit 262 configured to receive the speech data from the speech capture unit 253, and recognize and translate the spoken language included in the speech data into text based on ASR.

[0077] In one embodiment, the language interpreter 260 includes a natural language understanding (NLU) unit 263, which is configured to receive text from the ASR unit 262 and determine the intent of the voice command from the user 30 based on the text and the NLU. For example, in one embodiment, the intent indicates an action that the user 30 wants the electronic device 50 to perform now or sometime in the future. In one embodiment, the language interpreter 260 can generate intent data indicating the intent of the voice command from the user 30 and forward it to another component of the system 200.

[0078] In one embodiment, the language interpreter 260 maintains different vocabularies that can be used to interpret the spoken language included in the user's utterance. In one embodiment, the language interpreter 260 is configured to select a specific vocabulary from the different available vocabularies for use based on the type of activation method used to activate the voice agent 205 and / or the current state of the electronic device 50. For example, in one embodiment, the system 200 is configured to detect a user voice that is not directed to the electronic device 50 when the voice agent is activated via physical event activation. In one embodiment, if the voice agent is activated via physical event activation, in order to reduce the possibility that the electronic device 50 performs an undesirable or unintentional action based on the user's voice, the language interpreter 260 is configured to select a reduced / smaller vocabulary to use compared to one or more other vocabularies available. In one embodiment, the language interpreter 260 is configured to select a vocabulary that is suitable for interpreting only the voice commands available based on the current state of the electronic device 50 to use, thereby ensuring that the electronic device 50 will perform the action that the user 30 intends or expects the electronic device 50 to perform.

[0079] As another example, if the electronic device 50 includes a smart light in a room, and the voice agent 205 is activated in response to sensor-based contextual information indicating that the user 30 has just entered the room (e.g., detected via at least one sensor unit 70 (such as a motion sensor)), the language interpreter 260 is configured to select a vocabulary to be used during a listening window (e.g., a brief period of time after the user 30 enters the room) that is reduced / smaller than one or more other vocabularies available, wherein the selected vocabulary includes only a limited / reduced set of voice commands that the user 30 is permitted to make. For example, the smart light will only turn on when the user 30 issues a voice command "turn on the light." This restriction prevents the smart light from automatically turning on when the user 30 enters the room after bedtime and reduces the likelihood of disturbing any sleeping occupants in the room. In contrast, if the voice agent 205 is activated in response to a wake word activation, the language interpreter 260 is configured to select a vocabulary to be used during the listening window that is larger than the vocabulary used in response to a physical event activation.

[0080] In one embodiment, the language interpreter 260 includes a vocabulary selector table 261 that maintains different vocabularies that can be selected and used by the language interpreter 260 based on the type of activation method used to activate the voice agent 205 and / or the current state of the electronic device 50. For example, in one embodiment, if the electronic device 50 includes a smart washing machine that is currently stopped, and the voice agent 205 is activated via closing the door of the washing chamber of the washing machine or actuating the physical UI / GUI, the language interpreter 260 selects a vocabulary that includes the word "Start" from the vocabulary selector table 261, thereby enabling the user 30 to start the washing machine by simply uttering the word "Start" (i.e., the user 30 does not need to explicitly utter the wake-up word of the electronic device 50).

[0081] In one or more embodiments, Table 2 below provides an example set of different vocabularies maintained in vocabulary selector table 261 .

[0082] [Table 2]

[0083]

[0084] In one embodiment, the system 200 includes an action planner 270 configured to receive at least one of: (1) intent data from the language interpreter 260 , and (2) a user ID from the voice ID analysis unit 255 .

[0085] In one embodiment, the action planner 270 includes a device action initiator 274, which is configured to plan / schedule one or more actions to be performed by the electronic device 50 now or in the future based on the intent data and / or the user ID. The device action initiator 274 provides an action API for communicating with the device control and state logic unit 220. The device action initiator 274 allows one or more adjustments / updates to be made to the current state of the electronic device 50 via the action API to perform one or more actions.

[0086] In one embodiment, the system 200 is configured to provide feedback to the user 30 using different interactive feedback methods based on the type of activation method used to activate the voice agent 205. In one embodiment, the system 200 includes a voice interface visual feedback component 280, which includes one or more I / O units 65 (e.g., one or more indicator lights, display screens, etc.) for displaying visual feedback indicating the current mode of the voice agent 205, such as whether the voice agent 205 is listening (i.e., waiting and listening for user utterances), whether the voice agent 205 is thinking (i.e., interpreting user utterances), whether the voice agent 205 is responding (i.e., providing feedback), whether the voice agent 205 is in sleep mode, etc. For example, the indicator light may selectively flash different colors and / or adjust intensity / brightness based on the current mode of the voice agent 205 (e.g., flash green if the voice agent 205 is listening, flash yellow if the voice agent 205 is thinking, flash red if the voice agent 205 is responding, and dim if the voice agent 205 is in sleep mode), and the display screen may selectively display different phrases based on the current mode of the voice agent 205 (e.g., display "Listening..." if the voice agent 205 is listening, display "Thinking..." if the voice agent 205 is thinking, and display "Responding..." if the voice agent 205 is responding).

[0087] In one embodiment, the system 200 includes a speech synthesizer unit 290 configured to generate a voice response (e.g., via text to speech) indicating the current mode, error (e.g., timeout), and / or acknowledgement / confirmation of the voice agent 205. In one embodiment, the system 200 includes a speech output component 285, which includes one or more I / O units 65 (e.g., speakers, etc.) for outputting the voice response received from the speech synthesizer unit 290. For example, the system 200 can selectively generate and output different voice responses based on the current mode of the voice agent 205 (e.g., if the voice agent is listening, then output "listening...", if the voice agent is thinking, then output "thinking...", if the voice agent is responding, then output "responding...").

[0088] In one embodiment, the action planner 270 includes a visual feedback unit 272, which is configured to trigger the voice interface visual feedback component 280 to display specific visual feedback based on the intent data and / or the user ID. In one embodiment, the action planner 270 includes a voice feedback unit 273, which is configured to trigger the voice synthesizer component 290 to generate a specific voice response based on the intent data and / or the user ID.

[0089] In one embodiment, the system 200 is configured to selectively generate and output a voice response based on the type of activation method used to activate the voice agent 205. For example, in one embodiment, if the voice agent 205 is activated via a wake-up word activation, and the system 200 encounters an error (e.g., the voice command includes a word not included in the vocabulary used by the language interpreter 260, a timeout, the user 30 does not have user permission to control the electronic device 50 via voice, etc.), the system 200 is configured to generate and output a voice response indicating that the received voice command is unknown. As another example, in one embodiment, if the voice agent 205 is activated via a physical event activation, and the system 200 encounters an error, the system 200 is configured not to generate and output a voice response indicating that the voice command was not received.

[0090] In one embodiment, the action planner 270 includes an error feedback table 275 that maintains one or more error feedback rules for controlling when the system 200 generates and outputs voice responses based on the type of activation method used to activate the voice agent 205.

[0091] In one or more embodiments, Table 3 below provides an example set of error feedback rules maintained in error feedback table 275 .

[0092] [Table 3]

[0093]

[0094] In one embodiment, the action planner 270 includes one or more user permission tables 271 corresponding to one or more users 30, wherein each user permission table 271 maintains one or more user permissions of the corresponding user 30. The action planner 270 utilizes the user permission table 271 corresponding to the user 30 to determine whether the user 30 is authorized to control the electronic device 30 via voice based on the user ID of the user 30 and / or the type of activation method used to activate the voice agent 205.

[0095] In one or more embodiments, Table 4 below provides an example set of user permissions maintained in user permissions table 271 corresponding to user 30 .

[0096] [Table 4]

[0097]

[0098] Figure 4 1 shows a first example scenario involving multimodal interaction with intelligent assistants embodied in multiple electronic devices 50 in one or more embodiments. In one embodiment, a user 30 may activate a voice agent 205 operating in an electronic device 50 that is closest to the user 30 or a portion of a cluster of electronic devices 50 that are closest to the user 30. Figure 4 As shown, the user 30 in the laundry room can activate the voice agent 205 of the smart washing machine W and the voice agent 205 of the smart dryer D via the wake-up word activation, wherein the smart washing machine W and the smart dryer D form a cluster of electronic devices 50 in the laundry room closest to the user 30. For example, the user 30 activates the voice agent 205 of the smart washing machine W and the voice agent of the smart dryer D via the explicit utterance of the wake-up word.

[0099] like Figure 4As shown, the voice agent 205 of the smart washing machine W and the voice agent 205 of the smart dryer D both wake up in response to the wake-up word. In one embodiment, if the voice agent 205 of the electronic device 50 detects that the user 30 is not within the proximity (e.g., via at least one sensor unit 70), the voice agent 205 determines that the electronic device 50 is neither closest to the user 30 nor part of the cluster of electronic devices 50 closest to the user 30, and ignores the wake-up word activation from the user 30. For example, since the user 30 is not within the proximity of the smart refrigerator FH and the smart dishwasher DW, the voice agent 205 of the smart refrigerator in the kitchen (operating as a home hub) and the voice agent 205 of the smart dishwasher DW in the kitchen will not be activated by the wake-up word.

[0100] like Figure 4 As shown, user 30 follows the wake-up word with a voice command, which includes asking how much time is left on smart dryer D. The voice agent 205 of smart washing machine W determines that the inquiry is not for smart washing machine W, rejects the inquiry, and displays visual feedback to user 30 indicating that the voice agent 205 is switching back to sleep mode (for example, the indicator light of smart washing machine W dims). The voice agent 205 of smart dryer D determines that the inquiry is for smart dryer D, displays visual feedback to user 30 indicating that the voice agent 205 is in thinking mode, and generates and outputs a voice response indicating the amount of time remaining on smart dryer D to user 30.

[0101] Figure 5 A second example scenario involving multimodal interaction with intelligent assistants embodied in multiple electronic devices 50 is shown in one or more embodiments. In one embodiment, a user 30 can activate a voice agent 205 operating in an electronic device 50 via different types of activation methods (including physical event activation), providing multimodal interaction. Figure 5 As shown, a user 30 located in a laundry room can automatically activate the voice agent 205 of a specific electronic device 50 (such as a smart washing machine W) in the laundry room via physical event activation. For example, the user 30 automatically activates the voice agent 205 of the smart washing machine W by closing the door of the washing chamber of the smart washing machine W. This physical interaction with the smart washing machine W eliminates the need for the user 30 to explicitly issue a wake-up word to wake up the voice agent 205 of the smart washing machine W. In addition, since this physical interaction only involves the smart washing machine W, only the voice agent 205 of the smart washing machine W is awakened; it does not activate the voice agent 205 of a different electronic device 50 (such as a smart dryer D) in the laundry room.

[0102] like Figure 5As shown, the user 30 starts the smart washing machine W via a voice command including a request to start. The voice command is simple and does not require the user 30 to explicitly issue a wake-up word to start the smart washing machine W. The voice agent 205 of the smart washing machine W determines that the request is for the smart washing machine W, and triggers / calls the smart washing machine W to start a washing cycle.

[0103] In one embodiment, the voice agent 205 operating in the electronic device 50 is configured to determine at least one other electronic device 50 with which the electronic device 50 can be used in combination. For example, based on the recent actions performed by the smart washing machine W and the smart dryer D (e.g., the user 30 first washes clothes via the smart washing machine W and then dries the washed clothes via the smart dryer D), the voice agent 205 of the smart washing machine W determines that the smart washing machine W and the smart dryer D can be used in combination. In one embodiment, the voice agent 205 of the smart washing machine W triggers the voice agent 205 of the smart dryer D to wake up from sleep mode before the current washing cycle of the smart washing machine W ends. In one embodiment, the voice agent 205 of the smart washing machine W or the voice agent 205 of the smart dryer D provides feedback (e.g., voice and / or visual feedback) to the user 30, which indicates when the current washing cycle of the smart washing machine W ends. The feedback informs the user 30 when the washed clothes can be taken out of the smart washing machine W.

[0104] In one embodiment, after the user 30 has taken out the washed clothes from the smart washing machine W and put the washed clothes into the drying room, the user 30 automatically activates the voice agent 205 of the smart dryer D by closing the door of the drying room of the smart dryer D.

[0105] Figure 6 A third example scenario involving multimodal interaction with intelligent assistants embodied in multiple electronic devices 50 is shown in one or more embodiments. In one embodiment, the voice agent 205 operating in the multiple electronic devices 50 is configured to determine which of the multiple electronic devices 50 is best suited to respond to the voice command. Figure 6 As shown, a user 30 located in a laundry room can activate each voice agent 205 of each electronic device 50 in the laundry room (such as a smart washing machine W and / or a smart dryer D) via a wake-up word activation. For example, the user 30 activates both the voice agent 205 of the smart washing machine W and the voice agent 205 of the smart dryer D via a first voice command, wherein the first voice command starts with a wake-up word, followed by a first request to order detergent.

[0106] like Figure 6As shown, the voice agent 205 of the smart washing machine W and the voice agent 205 of the smart dryer D both wake up in response to the wake-up word. In one embodiment, the voice agent 205 of the smart dryer D determines that the first request is not for the smart dryer D, rejects the first request, and displays visual feedback to the user 30 that the voice agent 205 is switching back to the sleep mode (for example, the indicator light of the smart dryer D is dimmed). The voice agent 205 of the smart washing machine W determines that the first request is for the smart washing machine W, and displays visual feedback to the user 30 that the voice agent 205 is in the thinking mode.

[0107] like Figure 6 As shown, the voice agent 205 of the smart washing machine W generates and outputs a voice response to the user 30, which identifies the purchase of detergent previously made by the user 30 and the availability of the detergent transaction. The user 30 responds to the voice agent 205 via a second voice command, which includes a second request to check the transaction. In one embodiment, the voice agent 205 of the smart washing machine W is configured to determine whether there is another electronic device 50 that is most suitable for responding to the second request within the proximity of the user 30. For example, the voice agent 205 of the smart washing machine W determines that the smart refrigerator FH in the kitchen is most suitable for responding to the second request, triggers / calls the smart refrigerator FH to display the transaction on the touch screen of the smart refrigerator FH, and generates and outputs a voice response indicating that the transaction is displayed on the smart refrigerator FH in the laundry room to the user 30. The voice agent 205 of the smart washing machine W responds to the second request with a more capable nearby electronic device 50, and implements a natural mode switch from voice interaction (via voice command) to touch interaction (via touch screen).

[0108] like Figure 6 As shown, when the user 30 moves to the kitchen to view the transaction displayed on the smart refrigerator FH, the voice agent 205 of the smart washing machine W displays visual feedback that the voice agent 205 is transitioning back to sleep mode (e.g., the indicator light of the smart washing machine W dims).

[0109] Figure 7 A fourth example scenario involving multimodal interaction with intelligent assistants embodied in multiple electronic devices 50 is shown in one or more embodiments. In one embodiment, a user 30 can activate a voice agent 205 operating in an electronic device 50 via different types of activation methods (including physical event activation), providing multimodal interaction. Figure 7As shown, a user 30 located in a kitchen can automatically activate the voice agent 205 of a specific electronic device 50 (such as a smart oven O) in the kitchen via physical event activation. For example, the user 30 automatically activates the voice agent 205 of the smart oven O by closing the door of the oven chamber of the smart oven O. This physical interaction with the smart oven O eliminates the need for the user 30 to explicitly issue a wake-up word to wake up the voice agent 205 of the smart oven O. In addition, since this physical interaction only involves the smart oven O, only the voice agent 205 of the smart oven O is awakened; it does not activate the voice agents 205 of different electronic devices 50 in the kitchen (such as a smart microwave oven M, a smart stovetop R, a smart dishwasher DW, and / or a smart refrigerator FH).

[0110] like Figure 7 As shown, user 30 starts smart oven O via a voice command, which includes a request to bake with specified settings (e.g., bake at 450 degrees for one hour and then keep warm). The voice command is simple and does not require user 30 to explicitly issue a wake-up word to start smart oven O. The voice agent 205 of smart oven O determines that the request is for smart oven O, triggers / calls smart oven O to execute the request, and displays visual feedback to user 30 via the control surface of smart oven O, wherein the visual feedback indicates the settings. If the voice agent 205 of smart oven O is still in listening mode (i.e., the listening window of voice agent 205 has not passed), user 30 can issue another voice command including a request to adjust the settings.

[0111] Other examples of physical event activation include, but are not limited to: user 30 automatically activates the voice agent 205 of the smart microwave oven M by closing the door of the smart microwave oven M (for example, after placing food in the smart microwave oven M for microwave processing), user 30 automatically activates the voice agent 205 of the smart refrigerator FH by closing the door of the smart refrigerator FH (for example, after placing food in the smart refrigerator FH for refrigeration), user 30 automatically activates the voice agent 205 of the smart dishwasher DW by closing the door of the dishwashing chamber of the smart dishwasher DW (for example, after placing dishes in the dishwashing chamber for washing), etc.

[0112] In one embodiment, an activity performed by one electronic device 50 of a cluster of electronic devices 50 may activate the voice agent 205 of one or more other electronic devices 50 of the cluster. Figure 7As shown, a smart oven O, a smart microwave M, a smart stovetop R, a smart dishwasher DW, and a smart refrigerator FH form a cluster of electronic devices 50 in a kitchen. Assume that a user 30 is viewing a recipe for chicken parmesan displayed on the touch screen of the smart refrigerator FH. If the user 30 wants to cook the recipe, the user 30 can wake up the voice agent 205 operating in the cluster via a wake-up word activation or a physical event activation (e.g., UI activation (such as selecting a recipe on the touch screen), door closing activation, etc.).

[0113] For example, assume that the user 30 takes out the frozen chicken from the smart refrigerator FH and closes the door of the smart microwave M after putting the frozen chicken into the smart microwave M for thawing. The voice agent 205 of the smart microwave M is activated in response to the closing of the door of the smart microwave M. When the user 30 issues a voice command "start", the voice agent 205 determines contextual information related to the voice command, such as the microwave settings required to execute the voice command. For example, based on a recipe and / or one or more online data sources, the voice agent 205 determines the microwave settings such as the temperature and amount of time required to thaw the frozen chicken, and triggers / calls the smart microwave M to start according to the microwave settings, and displays visual feedback via the control surface of the smart microwave M, wherein the visual feedback indicates the microwave settings. The user 30 can view the displayed microwave settings and make adjustments via voice interaction (e.g., issuing a voice command) or touch interaction (e.g., interacting with the control surface) if necessary.

[0114] If the recipe calls for grilling the chicken, the voice agent 205 of the smart stove R may be activated before the smart microwave M completes thawing the chicken. For example, assume that the voice agent 205 of the smart microwave M or the master device of the cluster triggers the voice agent 205 of the smart stove R to wake up one minute or a few seconds before the smart microwave M completes thawing. When the user 30 places the pan containing the thawed chicken on the smart stove R (after removing the chicken from the smart microwave M) and issues a voice command "start", the voice agent 205 of the smart stove R determines contextual information related to the voice command, such as the stove settings required to execute the voice command. For example, based on the recipe and / or one or more online data sources, the voice agent 205 determines the stove settings such as the temperature and amount of time required to grill the chicken, and triggers / calls the smart stove R to start according to the stove settings, and displays visual feedback via the control surface of the smart stove R, wherein the visual feedback indicates the stove settings. The user 30 can view the displayed stove settings and make adjustments via voice interaction (e.g., issuing a voice command) or touch interaction (e.g., interacting with the control surface) if necessary.

[0115] If the recipe requires that the chicken be baked after grilling, the voice agent 205 of the smart oven O may be activated before the smart stove R finishes grilling the chicken. For example, assume that the voice agent 205 of the smart stove R or the master device of the cluster triggers the voice agent 205 of the smart oven O to wake up and start preheating the smart oven O a few minutes before the smart stove R finishes grilling. The voice agent 205 determines contextual information related to preheating the smart oven O, such as the oven settings required to preheat the smart oven O. For example, the voice agent 205 determines the oven settings such as the temperature and amount of time required to bake the chicken, and triggers / calls the smart oven O to start preheating according to the temperature. When the user 30 puts the pan containing the grilled chicken into the smart oven O and issues a voice command "start", the voice agent 205 of the smart oven O triggers / calls the smart oven O to start baking according to the oven settings, and displays visual feedback via the control surface of the smart oven O, wherein the visual feedback indicates the oven settings. The user 30 can view the displayed oven settings and make adjustments via voice interaction (e.g., issuing a voice command) or touch interaction (e.g., interacting with the control surface) if necessary.

[0116] Based on context information related to the user 30 (e.g., sensor-based context information captured via at least one sensor unit 70, calendar information of the user 30, voice commands scheduled for the future, etc.), the voice agent 205 (e.g., the master device) operating in the cluster can determine whether the user 30 will leave or has left home (e.g., to pick up the child from school) while the chicken is baking in the smart oven O, and trigger / call the smart oven O to stop baking when the user 30 is not there. Once it is detected that the user 30 has returned home, the voice agent 205 can trigger / call the smart oven O to resume baking.

[0117] Figure 8 A fifth example scenario involving multimodal interaction with intelligent assistants embodied in multiple electronic devices 50 is shown in one or more embodiments. In one embodiment, a voice agent 205 operating in multiple electronic devices 50 in multiple locations (e.g., different rooms) utilizes contextual information, confirmation, and clarification to resolve multi-device disambiguation. Figure 8 As shown, a user 30 located in a kitchen can activate a voice agent 205 operating in a plurality of electronic devices 50 in the kitchen (such as a smart microwave oven M, a smart stovetop R, a smart oven O, and / or a smart home device H including a smart AI speaker) via a wake-up word activation. For example, the user 30 activates the voice agent 205 via a first voice command, wherein the first voice command begins with a wake-up word followed by a statement that it is too hot.

[0118] In one embodiment, the smart home device H operates as a master device that controls multiple electronic devices 50 in different rooms. In one embodiment, when the voice agent 205 of the smart home device H wakes up in response to the wake-up word, the voice agent 205 is configured to determine context information related to the statement based on the current state of each of the multiple electronic devices 50 (e.g., which of the multiple electronic devices 50 is a contributing factor to the statement). For example, Figure 8 As shown, if the current state of the smart oven O is that the smart oven O is baking, the voice agent 205 of the smart home device H determines that the current state of the smart oven O is relevant to the statement, and requests clarification from the user 30 by generating and outputting a voice response for the user 30 in the kitchen asking him whether the temperature of the smart oven O should be lowered.

[0119] like Figure 8 As shown, the user 30 responds to the voice agent 205 via a second voice command that clarifies that the user 30 feels hot. In response to the clarification from the user 30, the voice agent 205 of the smart home device H determines context information related to the clarification based on the current state of each of the multiple electronic devices 50 other than the smart oven O (e.g., which of the multiple electronic devices 50 other than the smart oven O is a contributing factor to the clarification). For example, Figure 8 As shown, if the current state of the smart air conditioner AC in the living room is that the temperature of the smart air conditioner AC is set too high, the voice agent 205 of the smart home device H determines that the current state of the smart air conditioner AC is relevant to the clarification, and requests clarification from the user 30 by generating and outputting a voice response for the user 30 in the kitchen asking him whether the temperature of the smart air conditioner AC should be lowered.

[0120] like Figure 8 As shown, the user 30 responds to the voice agent 205 via a third voice command, wherein the third voice command confirms that the user 30 wants the voice agent 205 to lower the temperature of the smart air conditioner AC. In response to the confirmation from the user 30, the voice agent 205 of the smart home device H triggers / calls the smart air conditioner AC to lower the temperature to which the smart air conditioner AC is set.

[0121] Fig. 9 A sixth example scenario involving multimodal interaction with intelligent assistants embodied in multiple electronic devices 50 is shown in one or more embodiments. In one embodiment, a user 30 can activate a voice agent 205 operating in multiple electronic devices 50 in multiple locations (e.g., different rooms) to provide continuous and cross-device conversational NLU. Fig. 9As shown, a user 30 located in a living room can activate a voice agent 205 operating in a plurality of electronic devices 50 (such as a smart air conditioner AC, a smart TV T, and / or a smart home mini device HM including a smart AI speaker) in the living room or in the vicinity of the living room via a wake-up word activation. For example, the user 30 activates the voice agent 205 via a first voice command, wherein the first voice command begins with a wake-up word, followed by a request to play a Y game (e.g., a basketball game involving the Y team).

[0122] In one embodiment, the smart home mini device HM operates as a master device that controls multiple electronic devices 50 in different rooms. In one embodiment, when the voice agent 205 of the smart home mini device HM wakes up in response to a wake-up word, the voice agent 205 is configured to determine which of the multiple electronic devices 50 the request is for (e.g., which of the electronic devices 50 is most suitable for executing the request), and trigger / call the electronic device 50 to which the request is directed to execute the request. For example, Fig. 9 As shown, the voice agent 205 of the smart home mini device HM determines that the request is for the smart TV T, and triggers / calls the smart TV T to start the Y game.

[0123] In one embodiment, when the user 30 moves from the living room to the kitchen, the user 30 can activate the voice agent 205 operating in multiple electronic devices 50 in the kitchen (such as a smart microwave oven M, a smart stovetop R, a smart oven O, and / or a smart home device H) via a wake-up word activation. Fig. 9 As shown, the user 30 activates the voice agent 205 via a second voice command, where the second voice command begins with a wake-up word followed by a query about the score.

[0124] In one embodiment, the smart home device H operates as a master device that controls multiple electronic devices 50 in different rooms. In one embodiment, when the voice agent 205 of the smart home device H wakes up in response to the wake-up word, the voice agent 205 is configured to determine context information related to the query based on the current state of each of the multiple electronic devices 50. For example, Fig. 9 As shown, the voice agent 205 of the smart home device H determines that the Y game on the smart TV T is contextual information related to the inquiry, calls an application (e.g., the software mobile application 90) to query the current score of the Y game, and generates and outputs a voice response providing the current score of the Y game to the user 30 in the kitchen.

[0125] Fig.10A seventh example scenario involving multimodal interaction with an intelligent assistant embodied in multiple electronic devices 50 is shown in one or more embodiments. In one embodiment, multiple users 30 can activate voice agents 205 operating in multiple electronic devices 50 in multiple locations (e.g., different rooms) to provide continuous, cross-device, and user-aware conversational NLU. Fig.10 As shown, the first user A located in the living room can activate the voice agent 205 operating in multiple electronic devices 50 (such as smart air conditioner AC, smart TV T, and / or smart home mini device HM) in the living room or in the vicinity of the living room via a wake-up word activation. For example, the first user A activates the voice agent 205 via a first voice command, wherein the first voice command starts with a wake-up word, followed by a first request to clean the living room.

[0126] In one embodiment, the smart home mini device HM operates as a master device to control multiple electronic devices 50 in different rooms. In one embodiment, when the voice agent 205 of the smart home mini device HM wakes up in response to the wake-up word, the voice agent 205 is configured to determine which of the multiple electronic devices 50 the first request is for, and trigger / call the electronic device 50 to which the first request is directed to execute the first request. For example, Fig.10 As shown, the voice agent 205 of the smart home mini device HM determines that the first request is for the smart vacuum cleaner V located in the laundry room, triggers / calls the smart vacuum cleaner V to execute the first request by starting and moving to the living room, and generates and outputs a voice response for the first user A in the living room to confirm / acknowledge the first request.

[0127] like Fig.10 As shown, the second user B located in the kitchen can activate the voice agent 205 operating in multiple electronic devices 50 (such as smart microwave oven M, smart stove R, smart oven O, smart home mini device HM, and / or smart refrigerator FH) in the kitchen or in the vicinity of the kitchen via the wake-up word activation. For example, the second user B activates the voice agent 205 via a second voice command, wherein the second voice command starts with the wake-up word, followed by a second request to stop cleaning.

[0128] In one embodiment, the smart refrigerator FH operates as a master device that controls multiple electronic devices 50 in different rooms. In one embodiment, when the voice agent 205 of the smart refrigerator FH wakes up in response to the wake-up word, the voice agent 205 is configured to determine which of the multiple electronic devices 50 the second request is for based on the current state of each of the multiple electronic devices 50. For example, Fig.10As shown, the voice agent 205 of the smart refrigerator FH determines that the second request is for the smart vacuum cleaner V, triggers / calls the smart vacuum cleaner V to execute the second request by moving back to the laundry room and stopping, and generates and outputs a voice response for the second user B in the kitchen to confirm / confirm the second request.

[0129] In one embodiment, if the voice agent 205 operating in multiple electronic devices 50 in multiple locations (e.g., different rooms) receives conflicting voice commands from multiple users 30, the voice agent 205 is configured to notify at least one of the multiple users 30 of the conflict. Fig.10 As shown, the voice agent 205 of the smart refrigerator FH determines that the first request and the second request are conflicting voice commands. When the first user A issues the first request, the first user A located in the living room is canceled, and a voice response is generated and output in the living room for the first user A to notify the second user B to issue the second request.

[0130] Fig.11 900 is a flow chart of an example process 900 for implementing multimodal interaction with an intelligent assistant in an electronic device in one or more embodiments. Process block 901 includes detecting activation of an intelligent assistant (e.g., voice agent 205) on an electronic device (e.g., electronic device 50). Process block 902 includes waking the intelligent assistant from a sleep mode in response to the activation. Process block 903 includes determining a vocabulary that the intelligent assistant is active during a listening mode based on the type of activation.

[0131] In one embodiment, process blocks 901 - 903 may be performed by one or more components of the voice user interface system 200 .

[0132] Fig.12 9 is a flow chart of an example process 910 for implementing physical event activation of an intelligent assistant in an electronic device in one or more embodiments. Process block 911 includes detecting a physical interaction with an electronic device (e.g., electronic device 50). Process block 912 includes activating an intelligent assistant (e.g., voice agent 205) on the electronic device from a sleep mode in response to the physical interaction.

[0133] In one embodiment, process blocks 911 - 912 may be performed by one or more components of the voice user interface system 200 .

[0134] Fig.136 is a high-level block diagram showing an information processing system including a computer system 600 useful for implementing the disclosed embodiments. System 200 may be incorporated in computer system 600. Computer system 600 includes one or more processors 601, and may also include an electronic display device 602 (for displaying video, graphics, text, and other data), a main memory 603 (e.g., random access memory (RAM)), a storage device 604 (e.g., a hard drive), a removable storage device 605 (e.g., a removable storage drive, a removable memory module, a tape drive, an optical drive, a computer-readable medium having computer software and / or data stored therein), a viewer interface device 606 (e.g., a keyboard, a touch screen, a keypad, a pointing device), and a communication interface 607 (e.g., a modem, a network interface (such as an Ethernet card), a communication port, or a PCMCIA slot and card). Communication interface 607 allows software and data to be transferred between the computer system and external devices. System 600 also includes a communication infrastructure 608 (e.g., a communication bus, a crossbar, or a network) to which the aforementioned devices / modules 601 to 607 are connected.

[0135] The information transmitted via the communication interface 607 may be in the form of signals, such as electronic, electromagnetic, optical or other signals that can be received by the communication interface 607 via a communication link that carries the signals and may be implemented using wire or cable, optical fiber, telephone line, cellular telephone link, radio frequency (RF) link and / or other communication channels. The computer program instructions representing the block diagrams and / or flow charts herein may be loaded onto a computer, programmable data processing apparatus or processing device to cause a series of operations to be performed thereon to generate a computer-implemented process. In one embodiment, the process 900 ( Fig.11 ) and process 910 ( Fig.12 ) can be stored as program instructions in the memory 603, the storage device 604 and / or the removable storage device 605 for execution by the processor 601.

[0136] Embodiments have been described with reference to flowchart illustrations and / or block diagrams of methods, devices (systems) and computer program products. Each block of such illustrations / diagrams or a combination thereof may be implemented by computer program instructions. When provided to a processor, the computer program instructions produce a machine so that instructions executed by the processor create a device for implementing the functions / operations specified in the flowchart and / or block diagram. Each block in the flowchart / block diagram may represent a hardware and / or software module or logic. In alternative embodiments, the functions mentioned in the blocks may not occur in the order mentioned in the figure, may occur simultaneously, and so on. The method (or some operations, actions, actions, blocks, steps) according to an embodiment of the present disclosure may be performed by at least one processor (e.g., processor 55) or one of electronic devices (e.g., electronic device 50).

[0137] The terms "computer program medium", "computer usable medium", "computer readable medium" and "computer program product" are generally used to refer to media such as main memory, secondary memory, removable storage drives, hard disks installed in hard drives, and signals. These computer program products are devices for providing software to computer systems. Computer readable media allow a computer system to read data, instructions, messages or message packets, and other computer readable information from the computer readable medium. For example, computer readable media may include non-volatile memory such as floppy disks, ROMs, flash memory, disk drive memory, CD-ROMs, and other permanent storage devices. For example, it can be used to transfer information, such as data and computer instructions, between computer systems. Computer program instructions may be stored in a computer readable medium that can direct a computer, other programmable data processing device, or other device to operate in a specific manner so that the instructions stored in the computer readable medium produce an article of manufacture including instructions for implementing the functions / actions specified in one or more blocks of the flowchart and / or block diagram.

[0138] As will be appreciated by those skilled in the art, aspects of the embodiments may be embodied as systems, methods, or computer program products. Thus, aspects of the embodiments may take the form of a full hardware embodiment, a full software embodiment (including firmware, resident software, microcode, etc.), or an embodiment combining software and hardware aspects, which may all be generally referred to herein as a "circuit," "module," or "system." In addition, aspects of the embodiments may take the form of a computer program product embodied in one or more computer-readable media having computer-readable program code embodied thereon.

[0139] Any combination of one or more computer readable media may be utilized. A computer readable medium may be a computer readable storage medium. A computer readable storage medium may be, for example, but not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any suitable combination of the foregoing. More specific examples (a non-exhaustive list) of computer readable storage media would include the following: an electrical connection with one or more conductors, a portable computer floppy disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing. In the context of this document, a computer readable storage medium may be any tangible medium that can contain or store a program used by or in conjunction with an instruction execution system, apparatus, or device.

[0140] Computer program code for performing operations of various aspects of one or more embodiments may be written in any combination of one or more programming languages, including object-oriented programming languages ​​such as Java, Smalltalk, C++, etc., and traditional procedural programming languages ​​such as the "C" programming language or similar programming languages. The program code may be executed entirely on the user's computer, partially on the user's computer, as a separate software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In the latter case, the remote computer may be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0141] Aspects of one or more embodiments are described above with reference to flowchart illustrations and / or block diagrams of methods, devices (systems) and computer program products. It should be understood that each block of the flowchart illustrations and / or block diagrams and the combination of blocks in the flowchart illustrations and / or block diagrams can be implemented by computer program instructions. These computer program instructions can be provided to a special-purpose computer or other programmable data processing device to produce a machine, so that instructions executed via a processor of the computer or other programmable data processing device create a device for implementing the functions / actions specified in one or more blocks of the flowchart and / or block diagram.

[0142] These computer program instructions may also be stored in a computer-readable medium, which can instruct a computer, other programmable data processing apparatus, or other device to operate in a specific manner so that the instructions stored in the computer-readable medium produce an article of manufacture including instructions for implementing the functions / actions specified in one or more blocks of the flowchart and / or block diagram.

[0143] The computer program instructions may also be loaded onto a computer, other programmable data processing apparatus, or other device to cause a series of operating steps to be performed on the computer, other programmable apparatus, or other device to produce a computer-implemented process, so that the instructions executed on the computer or other programmable apparatus provide a process for implementing the functions / actions specified in one or more blocks of the flowchart and / or block diagram.

[0144] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functions and operations of possible implementations of the systems, methods and computer program products according to various embodiments. In this regard, each block in the flowchart or block diagram may represent a module, segment or instruction portion, which includes one or more executable instructions for implementing (multiple) specified logical functions. In some alternative embodiments, the functions mentioned in the blocks may not occur in the order mentioned in the figure. For example, two blocks shown in succession can actually be executed substantially simultaneously, or these blocks can sometimes be executed in reverse order, depending on the functions involved. It will also be noted that each block illustrated in the block diagram and / or flowchart and the combination of blocks in the block diagram and / or flowchart can be implemented by a system based on special hardware, wherein the system based on special hardware performs a specified function or action or performs a combination of special hardware and computer instructions.

[0145] Unless expressly stated otherwise, reference to a singular element in a claim does not mean "one and only" but rather "one or more." All structural and functional equivalents to the elements of the above-described exemplary embodiments that are currently known or later come to be known to one of ordinary skill in the art are intended to be encompassed by the present claims. Unless a claim element is expressly recited herein using the phrase "means for" or "step for," that element is not to be construed under the provisions of 35 USC § 112, paragraph 6.

[0146] The terms used herein are only for describing specific embodiments and are not intended to limit the present invention. As used herein, the singular forms "a", "an" and "the" are intended to include the plural forms as well, unless the context clearly indicates otherwise. It will be further understood that when used in this specification, the terms "include" and / or "comprise" specify the presence of stated features, integers, steps, operations, elements and / or components, but do not exclude the presence or addition of one or more other features, integers, steps, operations, elements, components and / or groups thereof.

[0147] The corresponding structures, materials, acts, and equivalents of all means or step plus function elements in the following claims are intended to include any structure, material, or act for performing the function in combination with other claimed elements specifically claimed. The description of the embodiments has been presented for the purposes of illustration and description, but is not intended to be exhaustive or limited to the embodiments in the disclosed form. Many modifications and variations will be apparent to those of ordinary skill in the art without departing from the scope and spirit of the invention.

[0148] Although the embodiments have been described with reference to specific versions thereof; however, other versions are possible. Therefore, the spirit and scope of the appended claims should not be limited to the description of the preferred versions contained herein.

Claims

1. An electronic device, comprising: monitor; microphone; speaker; at least one sensor; at least one processor coupled to the display, the microphone, the speaker, and the at least one sensor; as well as Memory, storage instructions, Wherein, when the instructions are executed by the at least one processor, the electronic device: providing first feedback via at least one of the display or the speaker based on a first occurrence of a first event, the first event comprising recognizing a wake-up word from a first voice signal obtained via the microphone; generating a first timeout based on no speech signal being obtained via the microphone for a first duration after providing the first feedback; providing second feedback via at least one of the display or the speaker based on a first occurrence of a second event, the second event comprising detecting, via the at least one sensor, an external object within a proximity of the electronic device, and the second feedback being different from the first feedback, and A second timeout is generated based on no speech signal being obtained via the microphone for a second duration after providing the second feedback, wherein the first duration is longer than the second duration.

2. The electronic device according to claim 1, wherein: The first feedback includes sound-based feedback provided via the speaker or visual indication-based feedback provided via the display.

3. The electronic device according to claim 1, wherein: The first duration is different from the third duration, and the second duration is different from the fourth duration.

4. The electronic device according to claim 1, wherein: The instructions, when executed by the at least one processor, cause the electronic device to provide a visual indication indicating a transition to a sleep mode based on generating the first timeout.

5. The electronic device according to claim 1, wherein: The instructions, when executed by the at least one processor, cause the electronic device to provide a visual indication indicating a transition to a sleep mode based on generating the second timeout.

6. The electronic device according to claim 1, wherein: The instructions, when executed by the at least one processor, cause the electronic device to control the display to display a text phrase indicating a transition to a listening mode based on a second voice signal being obtained via the microphone and before a third timeout is generated.

7. The electronic device according to claim 1, wherein: The instructions, when executed by the at least one processor, cause the electronic device to control the display to display a text phrase indicating a transition to a listening mode based on a third voice signal being obtained via the microphone and before a third timeout is generated.

8. The electronic device according to claim 1, wherein: When executed by the at least one processor, the instructions cause the electronic device to provide a voice response via the speaker indicating that no voice command was recognized from the second voice signal during the generation of the third timeout.

9. The electronic device according to claim 1, wherein: The instructions, when executed by the at least one processor, cause the electronic device to provide visual feedback via the display indicating that no voice command was recognized from the third voice signal during the generation of the fourth timeout.

10. The electronic device according to claim 3, wherein: The third duration is longer than the fourth duration.

11. A non-transitory storage medium storing one or more programs, the one or more programs comprising computer executable instructions, the computer executable instructions, when executed by at least one processor of an electronic device, causing the electronic device to: providing first feedback via at least one of a display of the electronic device or a speaker of the electronic device based on a first occurrence of a first event, the first event comprising recognizing a wake-up word from a first voice signal obtained via a microphone of the electronic device; generating a first timeout based on no speech signal being obtained via the microphone for a first duration after providing the first feedback; providing second feedback via at least one of the display or the speaker based on a first occurrence of a second event, the second event comprising detecting, via at least one sensor of the electronic device, that an external object is within a proximity of the electronic device, and the second feedback is different from the first feedback, and A second timeout is generated based on no speech signal being obtained via the microphone for a second duration after providing the second feedback, wherein the first duration is longer than the second duration.

12. The non-transitory storage medium according to claim 11, wherein: The first feedback includes at least one of sound-based feedback provided via the speaker and visual indication-based feedback provided via the display, and Wherein, the second feedback comprises feedback based on visual indication provided via the display.

13. The non-transitory storage medium according to claim 11, wherein: The first duration is different from the third duration, and the second duration is different from the fourth duration.

14. The non-transitory storage medium according to claim 11, wherein: The computer executable instructions, when executed by the at least one processor, cause the electronic device to provide a visual indication indicating a transition to a sleep mode based on generating the first timeout.

15. The non-transitory storage medium according to claim 11, wherein: The computer executable instructions, when executed by the at least one processor, cause the electronic device to provide a visual indication indicating a transition to a sleep mode based on generating the second timeout.

Images