Natural Machine Conversation Methods and Devices

Abstract

This document discloses apparatuses, methods, and storage media associated with spoken dialogue systems. In embodiments, an apparatus for engaging in natural machine conversation with a user may include a listening component for detecting keywords indicating the start of a conversation; a dialogue engine for engaging in conversation with the user during the conversation; and a controller for selectively activating or causing one of the listening component or the dialogue component to be activated, at least in part based on the state of the conversation, and transferring control to the activated listening component or the activated dialogue engine. Other embodiments may be disclosed or claimed.

Description

Technical Field

[0001] This disclosure relates to the fields of computing, machine conversation, and artificial intelligence; and more particularly to devices and methods associated with natural machine conversations with users. Background Technology

[0002] The background description provided herein is for the purpose of generally presenting the context of this disclosure. Unless otherwise indicated herein, the materials described in this section are not prior art to the claims of this application and are not admitted as prior art by virtue of their inclusion in this section.

[0003] With the development of integrated circuits, computing, artificial intelligence, speech recognition, and other related technologies, spoken dialogue systems have become increasingly popular. Examples of spoken dialogue systems include, but are not limited to, Siri from Apple Computer, Google Home from Google, Echo from Amazon, Cortana from Microsoft, and many more. For portability, many of today's spoken dialogue systems are battery-powered. To extend battery life, a low-power always-listening (LPAL) component with a magic word strategy is typically used. The LPAL component runs a very low-power automatic speech recognition (ASR) engine that recognizes and responds to only one "magic word" or phrase. When the "magic word" or phrase is recognized, LPAL activates a much more powerful ASR component embedded in the spoken dialogue engine to recognize and respond to the user's utterance. During operation, the more powerful ASR component consumes more power than the LPAL component, so it is turned off after each response to the user's utterance. As a result, in this type of system, the user must use the magic word / phrase before each utterance they make to the system.

[0004] However, in collaborative, natural conversations, participants give each other opportunities to interject or take over the conversation, and they don't need to call each other's names every time they speak to ensure the other is listening. Therefore, current spoken dialogue systems with LPAL and magic word strategies are unnatural and annoying to many users. Solutions are needed that support more natural machine conversational interaction with users while conserving battery power. Brief description of the attached diagram

[0005] The embodiments will be readily understood from the following detailed description taken in conjunction with the accompanying drawings. For ease of description, the same reference numerals denote the same structural elements. In the drawings, embodiments are illustrated by way of example rather than limitation.

[0006] Figure 1 This is a simplified block diagram of the spoken dialogue system of the present disclosure according to various embodiments.

[0007] Figure 2 This is a simplified block diagram of a spoken dialogue system according to various embodiments, which includes the interaction of components of the spoken dialogue system before the start of a conversation.

[0008] Figure 3 The diagram shown is a simplified block diagram of a spoken dialogue system according to various embodiments, which includes the interaction of components of the spoken dialogue system during a conversation.

[0009] Figure 4 The diagram shown is a simplified block diagram of a spoken dialogue system according to various embodiments, which includes the interaction of components of the spoken dialogue system at the end of a conversation.

[0010] Figure 5 A computing system according to various embodiments is shown that is suitable for practicing various aspects of this disclosure.

[0011] Figure 6 The various embodiments are shown to be configured for practice. Figure 1-4 An exemplary computer-readable storage medium for the instructions of the methods (aspects) shown. Detailed Implementation

[0012] This document discloses devices, methods, and storage media associated with spoken dialogue systems. In embodiments, the device for engaging in natural machine conversation with a user may include a listening component for detecting keywords indicating the start of a conversation; a dialogue engine for conversing with the user during the conversation; and a controller coupled to the listening component and the dialogue engine for selectively activating or causing one of the listening component or the dialogue engine to be activated, at least partially based on the state of the conversation, and transferring control to the activated listening component or the activated dialogue engine. When one of the listening component and the dialogue engine is activated or causes to be activated, the other of the listening component and the dialogue engine is deactivated or causes to be deactivated.

[0013] In an embodiment, a machine-implemented method for natural conversation with a user may include: after activating the dialogue engine, setting a timer for the dialogue engine by the computer device's dialogue engine; after setting the timer, having the dialogue engine engage in a conversation with the user of the computer device; and after the timer expires, the dialogue engine notifying the controller of the computer device of the timer's expiration. In response to the timer's expiration, the controller may activate or cause the listening component to be activated, and deactivate or cause the dialogue engine to be disabled.

[0014] These and other aspects will be further described below with reference to the accompanying drawings. As those skilled in the art will appreciate from the following description, the spoken dialogue system of this disclosure for machine conversations with users can better mimic human-to-human behavior by using magic words / phrases only when humans would expect to attract the attention of their conversational partner—at the beginning of the conversation and when the conversation is resumed after a period of inactivity (but not before each interaction). As a result, a more natural and better user experience can be achieved.

[0015] In the following description, reference is made to the accompanying drawings, which form a part of this document, wherein like reference numerals designate like portions throughout, and wherein illustrative embodiments that can be practiced are shown. It should be understood that other embodiments may be utilized and structural or logical changes may be made without departing from the scope of this disclosure. Therefore, the following detailed description is not intended to be limiting, and the scope of the embodiments is defined by the appended claims and their equivalents.

[0016] The operations of the methods can be described in turn as multiple discrete actions or operations in a manner most helpful for understanding the claimed subject matter. However, the order of description should not be construed as implying that these operations necessarily depend on the order. Specifically, these operations may not be performed in the order presented. The described operations may be performed in an order different from the described embodiments. In additional embodiments, various additional operations may be performed and / or the described operations may be omitted, split, or combined.

[0017] For the purposes of this disclosure, the phrase "A and / or B" means (A), (B), or (A and B). For the purposes of this disclosure, the phrase "A, B, and / or C" means (A), (B), (C), (A and B), (A and C), (B and C), or (A, B, and C).

[0018] The specification may use the phrases "in embodiments" or "in multiple embodiments," each of which may refer to one or more of the same or different embodiments. Furthermore, the terms "comprising," "including," and "having" are used synonymously with respect to embodiments of this disclosure.

[0019] As used below (including the claims), the term "module" can refer to an application-specific integrated circuit (ASIC), electronic circuit, processor (shared, dedicated, or grouped), and / or memory (shared, dedicated, or grouped) that executes one or more software or firmware programs generated from a plurality of programming instructions, combinational logic circuits, and / or other suitable components that provide the described functionality, or can be a part of, or include, all of the foregoing.

[0020] Now for reference Figure 1The diagram illustrates a simplified block diagram of the components of a spoken dialogue system according to various embodiments of the present disclosure. As shown, the spoken dialogue system 100 may include a dialogue engine 102, a listening component 104, a controller (also referred to as a listening inference engine) 106, and an audio feature extractor 108, all coupled to each other as shown.

[0021] The dialogue engine 102 can be configured to engage in conversation with a user, possessing the ability to understand the user's utterances and respond audibly to them. Conversely, the listening component 104 can be configured to listen for and understand one or more (a limited number) keywords or phrases (also known as magic words or phrases) that indicate the start or beginning of a conversation. Therefore, when compared to the listening component 104, the circuitry or logic implementing the dialogue engine 102 is more complex or sophisticated, and thus tends to consume more power during operation than the listening component 104.

[0022] The controller (also referred to as the listening inference engine) 102 can be configured to selectively activate / cause the dialogue engine 102 and the listening component 104, or deactivate / cause the dialogue engine 102 and the listening component 104, at least in part based on the state of the session with the user. More specifically, the controller 102 can be configured to activate / cause the dialogue engine 102 to be activated at the beginning of a session, and keep the dialogue engine 102 activated until the end of the session, and thereafter keep the dialogue engine 102 deactivated. Simultaneously, the controller 102 can be configured to activate / cause the listening component 104 to be activated at the end of a session, and keep the listening component 104 activated until another session begins, and after the start of a new session, keep the listening component 104 deactivated.

[0023] Therefore, the spoken dialogue system 100 used for machine-to-machine conversations with users can better mimic human behavior by using magic words / phrases only when humans expect to attract the attention of their conversational partners—at the beginning of the conversation and when resuming the conversation after a period of silence and inactivity. As a result, a more natural and better user experience can be achieved.

[0024] In an embodiment, controller 102 may be configured to infer and evaluate the state, start, continuation, or end of a session based at least in part on a session state indicator 124. The session state indicator 124 may include audio features extracted from audio input 122 of a user who wants to participate in or is participating in a session with the spoken dialogue engine 100. Examples of audio features may include, but are not limited to, pitch, amplitude, tempo, etc., of the audio input 122. In an embodiment, the session state indicator 124 may include data or information provided by the listening component 104 and the dialogue engine 102. Examples of this data may include, but are not limited to, indications from the listening component 104 that keywords / phrases have been identified, or indications from the dialogue engine 102 that user utterances have been identified and responses to user utterances are being provided, etc.

[0025] Controller 102 can infer or infer the start of a session from session status indicators 124 indicating that a keyword / phrase has been identified. Controller 102 can further infer or infer the continuation of the session from the pitch, amplitude, tempo, etc., of audio input 122, or from indicators indicating that a user utterance has been identified or that a response to a user utterance often elicits further utterances from the user. Similarly, controller 102 can infer or infer the end of a session from the pitch, amplitude, tempo, etc., of audio input 122, or from indicators indicating that no user utterance has been identified or that a response to a user utterance has been sent for some time (an extended quiet period), or from user utterances or responses that often indicate the end of a session (e.g., a user utterance or response to “goodbye” or “goodnight” from system 100, or that the user has uttered similar utterances).

[0026] Audio feature extractor 108 can be configured to receive and process audio input 122 from a user who wants to participate in or is participating in a conversation with spoken dialogue engine 100. As mentioned earlier, audio feature extractor 108 can be configured to extract audio features from audio input 122, such as pitch, amplitude, tempo, etc. During extraction, audio feature extractor 108 can provide these audio features of audio input 122 as a conversation status indicator 124 to controller 102.

[0027] In an embodiment, dialogue engine 102 may include asynchronous speech recognition (ASR) engine 116, timer 118, and dialogue management 120, which will be described more fully below. Listening component 104 may include ASR engine 114, which, while typically less capable than ASR 116, consumes significantly less power.

[0028] In embodiments, each of the dialogue engine 102, the monitoring component 104, the controller 106, and the audio feature extractor 108 may be implemented using an application-specific integrated circuit (ASIC), programmable circuitry (such as a field-programmable gate array (FPGA)), or in software having instructions to be executed by a computer processor having one or more processor cores. In embodiments, the dialogue engine 102 may be any of a plurality of dialogue engines known in the art, except for the timer 118, the dialogue manager 120, and the cooperation between the dialogue engine 102 and the controller 106. Similarly, except for cooperation with the controller 106, the monitoring component 104 may be any of a plurality of low-power monitoring components known in the art, such as an LPAL component. Likewise, except for cooperation with the controller 106, the audio feature extractor 108 may be any of a plurality of audio feature extractors known in the art.

[0029] Before proceeding to the remainder of the specification, it should be noted that, for ease of understanding, only data and control flows substantially relating to the selective activation and deactivation of the dialogue engine 102 and the listening component 104 are shown. The spoken dialogue system 100 may include other components and / or other data / control flows. For example, the spoken dialogue system 100 may include a battery (not shown) to provide power, and / or each of ASRs 114 and 116 may also receive audio input 122 to perform their speech recognition functions.

[0030] Now for reference Figure 2 The diagram illustrates a simplified block diagram of a spoken dialogue engine, according to various embodiments, showing the interaction of components of the spoken dialogue engine prior to the start of a session. As shown, at stage A, prior to the start of a session (e.g., after the end of a previous session), the ASR 114 of the listening component 104 may receive and process audio input 122 and listen for keywords(multiple) / phrases(multiple). Next, at stage B, the keywords / phrases may be identified by the ASR 114.

[0031] Subsequently, at stage C, the listening component 104 may notify the controller 106 that a keyword / phrase has been detected. Next, at stage D, upon receiving the notification, the controller 106 may infer or deduce the start or initiation 126 of a new session between the system 100 and the user. The controller 106 may activate the conversation engine 102, or cause the conversation engine 102 to be activated or reactivated from a deactivated state. In one embodiment, the controller 106 may transmit an activation signal to the conversation engine 102 to cause the conversation engine 102 to return from a deep sleep or paused state that consumes substantially very little or almost no power to an active state. In other embodiments, the controller 106 may send a signal to a power unit or switch to cause power to be supplied to the conversation engine 102 to exit a power-off state and enter a power-on state.

[0032] In this embodiment, the conversation engine 102 can be controlled after activation / reactivation. In addition to initiating a conversation with the user, the conversation engine 102 can set a timer 118 to expire after a certain period. The length of this period can be application-dependent and, in this embodiment, configurable for different applications.

[0033] At stage F, controller 106 can retract control from listening component 104, disable listening component 104, or cause listening component 104 to be deactivated. In one embodiment, controller 106 may transmit a deactivation signal to listening component 104, causing listening component 104 to return from a deep sleep or paused state that consumes virtually no power to an inactive state. In other embodiments, controller 106 may send a signal to a power unit or switch to cause the power being supplied to listening component 104 to stop, placing listening component 104 in a power-off state. In an alternative embodiment, listening component 104 may automatically relinquish control and place itself in a sleep state or turn off its own power after notifying controller 106 of the detection of a keyword / phrase.

[0034] In embodiments, in addition to the procedures described above for initiating a session, the conversation engine (DE) 102 can also initiate a session with the user. In these embodiments, at stage A, while the listening component 104 is listening for keywords, the conversation engine 102 can automatically wake up or activate itself in response to any of a plurality of host system events and initiate a session with the user. To initiate a session, the conversation engine 102 may send a notification to the controller 106, informing it that it expects the user to participate in the session (similar to when the listening component 104 detects a keyword in the audio input 122, the listening component 104 sends a notification to the controller 106). After receiving the notification from the conversation engine 102, the controller 106 can continue and begin the session, essentially as before. The controller 106 may pass control to the conversation engine 102, including instructions to set the timer 118, and deactivate the listening component 104, as previously described. Since the conversation engine 102 is already activated, for this scenario, the controller 106 may skip waking up or activating the conversation engine 102.

[0035] Now for reference Figure 3 A simplified block diagram of a spoken dialogue system, including the interaction of components of a spoken dialogue system during a conversation, is shown according to various embodiments. Figure 3 The diagram shows the components and their interactions in three separate scenarios: scenario 302a, where the conversation continues due to user utterance 316; scenario 302b, where the conversation continues due to system speech 326 in response to user utterance; and scenario 302c, where a feature 336 that often indicates the continuation of the conversation is detected (e.g., rising or flattening pitch, sustained pitch range, filled pauses, or system responses that often elicit further user utterances).

[0036] In scenario 302a, after recognizing user utterance 316, dialogue engine 102 (e.g., ASR 116) may notify controller 106 of this recognition. In response, controller 106 may allow dialogue engine 102 to retain control by resetting timer 118 314 to the beginning of a time period or causing timer 118 to be reset 314 to the beginning of a time period. In an embodiment, controller 106 may further maintain 312 an activation signal indicating that dialogue engine 102 wants to remain active.

[0037] In scenario 302b, after initiating a response to a user utterance (system speech) 326, the dialogue engine 102 (e.g., ASR 116) may notify the controller 106 of this initiation. In response, the controller 106 may similarly allow the dialogue engine 102 to retain control, resetting timer 118 324 to the start of a time period or causing timer 118 to be reset 324 to the start of a time period. In an embodiment, the controller 106 may similarly further maintain 322, indicating that the dialogue engine 102 wants to remain active.

[0038] In scenario 302c, after recognizing the continuing feature 336, the dialogue engine 102 (e.g., ASR 116) may notify the controller 106 of this recognition. In response, the controller 106 may allow the dialogue engine to retain control, resetting timer 118 334 to the start of the time period or causing timer 118 to be reset 334 to the start of the time period. In an embodiment, the controller 106 may further maintain 332, indicating that the dialogue engine 102 wants to remain active.

[0039] Now for reference Figure 4 A simplified block diagram of a spoken dialogue system, including components of a spoken dialogue system at the end of a conversation, is shown according to various embodiments. Figure 4 The diagram shows the components and their interaction in two separate scenarios: scenario 400a, where the session ends due to the timeout of timer 118 (long silence), and scenario 400b, where the characteristics of the end are detected (e.g., a lowered pitch, a pitch range not being maintained, a prolonged pause, or a system response that often does not elicit further user speech).

[0040] In scenario 400a, after timer 118 expires / times out 402, the dialogue engine 102 (e.g., timer 118) may notify controller 106 of this timeout event. In response, controller 106 may activate or cause listening component 104 to be activated 404a to return from an inactive state to an active state, and pass control to listening component 104 to listen for keywords / phrases again. In an embodiment, controller 106 may further revoke control from dialogue engine 102, deactivate or cause dialogue engine 102 to be deactivated 404b, to place dialogue engine 102 in a low-power or power-off state.

[0041] In scenario 400b, upon detecting a session end characteristic 406, the dialogue engine 102 (e.g., dialogue manager 118) may notify the controller 106 of this detection. In response, the controller 106 may activate 408a the listening component 104 or cause the listening component 104 to be activated 408a to return from an inactive state to an active state, and pass control to the listening component 104 to listen for keywords / phrases again. In an embodiment, the controller 106 may further revoke control from the dialogue engine 102, deactivating 408b the dialogue engine 102 or causing the dialogue engine 102 to be deactivated 408b, to place the dialogue engine 102 in a low-power or power-off state.

[0042] Now for reference Figure 5 Block diagrams of computer devices suitable for practicing this disclosure, according to various embodiments, are shown. As shown, computer device 500 may include one or more processors 502, system memory 504, and a spoken dialogue system 100. Each processor 502 may include one or more processor cores. System memory 504 may include any known volatile or non-volatile memory. Spoken dialogue system 100 may include a dialogue engine 102, a listening component 104, and a controller 106, as previously described.

[0043] Additionally, the computer device 500 may include a mass storage device 506 (such as a solid-state drive), an input / output device interface 508 (for connecting to various input / output devices such as a mouse, cursor controller, display devices (including touch-sensitive screens), etc.), and a communication interface 510 (such as a network interface card, modem, etc.). In embodiments, the communication interface 510 may support wired or wireless communication, including near-field communication. Components may be coupled to each other via a system bus 512, which may represent one or more buses. In the case of multiple buses, they may be bridged via one or more bus bridges (not shown).

[0044] Each of these components can perform its conventional functions known in the art. Specifically, system memory 504 and mass storage device 506 may be employed to store working and permanent copies of executable code of the operating system and programming instructions for one or more applications, collectively referred to as computing logic 522. In embodiments, one or more components of the operating system or applications may be configured to implement aspects of the spoken dialogue system 100, such as timer functions or some ASR functions, etc. The programming instructions may include assembly instructions supported by processor 502 or a high-level language (such as, for example, C) that can be compiled into such instructions.

[0045] A permanent copy of the executable code of the programming instructions may be placed in a permanent mass storage device 506 at the factory or in the field via, for example, a distribution medium (not shown) (such as an optical disc (CD)) or via a communication interface 510 (from a distribution server (not shown)).

[0046] The quantity, performance, and / or capacity of these components 510-512 may vary depending on the intended use of the exemplary computer device 500, for example, whether the exemplary computer device 500 is a wearable device, personal digital assistant, smartphone, tablet, ultrabook, laptop, server, in-vehicle infotainment system, set-top box, game console, camera, etc. The configuration of these components 510-512 is known in other ways and will therefore not be described further.

[0047] Figure 6 The various embodiments are shown having been configured as described prior to practice. Figure 1-4 An exemplary computer-readable storage medium of instructions for methods (aspects thereof). As shown, computer-readable storage medium 602 may include executable code of a plurality of programming instructions 604. The executable code of programming instructions 604 may be configured to cause a device, such as a spoken dialogue system 100 or a computer device 500, to perform, in response to the execution of the executable code / programming instructions. Figure 1-4 The method (aspect). In an alternative embodiment, executable code / programming instructions 604 may instead be disposed on a plurality of non-transient computer-readable storage media 602. In an embodiment, the computer-readable storage media 602 may be non-transient. In yet another embodiment, executable code / programming instructions 604 may be encoded in a transient computer-readable medium, such as a signal.

[0048] Refer back Figure 1 In one embodiment, the controller 106 may be packaged with a computer-readable storage medium having been configured to be implemented as previously referenced. Figure 1-4The described controller 106 comprises some or all of the executable code in instructions 604 for all or selected operations in its operation. In one embodiment, the controller 106 may be packaged with a computer-readable storage medium to form a system-in-package (SiP) having some or all of the executable code in instructions 604. In one embodiment, the controller 106 may be integrated with a computer-readable storage medium having some or all of the executable code in instructions 604 on the same die. In one embodiment, the controller 106 may be packaged with a computer-readable storage medium having some or all of the executable code in instructions 604 to form a system-on-a-chip (SoC). In at least one embodiment, the SoC may be used, for example, but not limited to, a voice-based portable personal assistant.

[0049] Example 1 could be a device for engaging in natural machine conversations with a user, comprising: a listening component for detecting keywords indicating the start of a conversation; a dialogue engine for engaging in conversation with the user during the conversation; and a controller coupled to the listening component and the dialogue engine to selectively activate or cause one of the listening component or the dialogue component to be activated, at least in part based on the state of the conversation, and to pass control to the activated listening component or the activated dialogue engine.

[0050] Example 2 can be Example 1, where the listening component consumes less power during operation than the dialogue engine consumes during its operation.

[0051] Example 3 can be Example 1, where the listening component can send a session state indicator to the controller indicating that a keyword has been detected or that a session has begun, or the dialogue engine can send a desired session state indicator to the controller indicating that a session has begun.

[0052] Example 4 can be Example 3, where the controller can activate or cause the dialogue engine to be activated upon receiving a session state indicator indicating the detection of a keyword or the start of a session, or an expected session state indicator indicating that the dialogue engine is starting a session, and pass control to the dialogue engine.

[0053] Example 5 can be Example 4, where, upon receiving a session state indicator indicating the detection of a keyword or the start of a session, or upon receiving a desired session state indicator indicating the start of a session, the controller may further disable the listening component or cause the listening component to be disabled.

[0054] Example 6 could be Example 4, where the dialogue engine can set a timer after activation to recognize and respond to user utterances.

[0055] Example 7 can be Example 6, where the controller can reset the timer or cause the timer to be reset after receiving a session state indicator that indicates the continuation of the session.

[0056] Example 8 can be Example 7, where the dialogue engine can send, and the controller can receive from, a session state indicator indicating that the dialogue engine recognizes a user utterance, the dialogue engine responds to a user utterance, or the dialogue engine detects a characteristic of continuing the session.

[0057] Example 9 can be Example 6, where the controller can activate or cause the listening component to be activated after receiving a session state indicator indicating that the session is in an ended state or a timer has expired, and pass control to the listening component.

[0058] Example 9 can be Example 5, wherein the dialogue engine can further detect session end features, and wherein the controller can activate the listening component and pass control to the listening component after receiving a session state indicator from the dialogue engine indicating that session end features have been detected.

[0059] Example 11 can be any of Examples 1-10, where the controller can activate the listening component and pass control to the listening component after receiving a session state indicator indicating that the end of the session has been detected.

[0060] Example 12 could be a machine implementation method for natural conversation with a user, comprising: after the dialogue engine is activated, the dialogue engine of the computer device sets a timer for the dialogue engine; after the timer is set, the dialogue engine engages in a conversation with the user of the computer device; and after the timer expires, the dialogue engine notifies the controller of the computer device of the timer expiration.

[0061] Example 13 can be Example 12, further including sending a session state indicator indicating that a keyword has been detected or that a session with the user has begun to be initiated by a listening component of a computer device, or sending a desired session state indicator indicating that a session with the user has begun to be initiated by a dialogue engine to the controller.

[0062] Example 14 can be Example 13, further including, upon receiving a session state indicator indicating that a keyword has been detected or a session has begun with the user, or an expected session state indicator indicating that the dialogue engine has begun a session with the user, the controller activates or causes the dialogue engine to be activated, and passes control to the dialogue engine.

[0063] Example 15 can be Example 14, further including, upon receiving a session state indicator indicating that a keyword has been detected or a session with the user has begun, or upon receiving a session state indicator indicating a desired start of a session with the user, the controller disables the listening component or causes the listening component to be disabled.

[0064] Example 16 can be Example 14, further including setting a timer by the dialogue engine after activation; recognizing user utterances by the dialogue engine; and responding to user utterances by the dialogue engine.

[0065] Example 17 can be Example 16, further including resetting the timer by the controller or causing the timer to be reset after receiving a session state indicator indicating the continuation of the session.

[0066] Example 18 can be Example 17, further including the dialogue engine sending a session state indicator to the controller, indicating that the dialogue engine recognizes user utterances, the dialogue engine responds to user utterances, or the dialogue engine detects characteristics of continuing the session.

[0067] Example 19 can be Example 16, further including, upon receiving a session state indicator indicating that the session is in an ended state or a timer has expired, the controller activates or causes the listening component to be activated, and passes control to the listening component.

[0068] Example 20 can be Example 15, further including the detection of session end features by the dialogue engine, and the activation of the listening component by the controller after receiving a session state indicator from the dialogue engine indicating that the session end features have been detected, and the transfer of control to the listening component.

[0069] Example 21 can be any of Examples 12-18, further including, upon receiving a session state indicator indicating the detection of the end of the session, the controller activates the listening component and passes control to the listening component.

[0070] Example 22 may be one or more computer-readable media (CRM) having instructions to cause a computer device, in response to execution of the instructions, to: analyze an indicator representing the state of a session between the computer device and a user using a controller of the computer device; and, based at least in part on the state of the session with the user, selectively activate one of the computer device's listening components or dialogue engines using the controller, causing the selected listening component or dialogue engine to be activated, and transferring control to the activated listening component to detect a keyword indicating the start of a session, or transferring control to the dialogue engine to engage in a session with the user.

[0071] Example 23 could be Example 22, in which the computer device could be further caused to send a session state indicator to the controller using the listening component, indicating that a keyword or the start of a session has been detected, or the dialogue engine could send a desired session state indicator to the controller, indicating the start of a session.

[0072] Example 24 can be Example 23, wherein the computer device may be further caused to activate the dialogue engine using the controller and pass control to the dialogue engine upon receiving a session state indicator indicating the detection of a keyword or the start of a session or an expected session state indicator indicating that the dialogue engine has started a session.

[0073] Example 25 could be Example 24, wherein the computer device may be further caused to disable the listening component using the controller after receiving a session state indicator indicating the detection of a keyword or the start of a session, or after receiving a desired session state indicator indicating the start of a session.

[0074] Example 26 could be Example 24, in which the computer device could be further configured to utilize the dialogue engine to set a timer upon activation, recognize user utterances, and respond to user utterances.

[0075] Example 27 can be Example 26, wherein the computer device may be further caused to reset the timer using the controller after receiving a session state indicator indicating the continuation of the session.

[0076] Example 28 can be Example 27, wherein the computer device may be further caused to use the dialogue engine to send a session state indicator to the controller, indicating that the dialogue engine recognizes user utterances, the dialogue engine responds to user utterances, or the dialogue engine detects characteristics of continuing the session.

[0077] Example 29 can be Example 26, wherein the computer device may be further caused to activate the listening component using the controller and pass control to the listening component upon receiving a session state indicator indicating that the session state is in an ended state or that a timer has expired.

[0078] Example 30 can be Example 25, wherein the computer device may be further configured to detect session end features using a dialogue engine, and wherein the controller may activate a listening component and pass control to the listening component after receiving a session state indicator from the dialogue engine indicating that a session end feature has been detected.

[0079] Example 31 can be any of Examples 22-28, wherein the computer device may be further caused to activate the listening component using the controller and pass control to the listening component after receiving a session state indicator indicating that the end of the session has been detected.

[0080] Example 32 may be a device for engaging in natural machine conversation with a user, including: a conversation device for engaging in conversation with a user of the device; wherein the conversation device includes a timer setting device for setting a timer duration after activation of the device for engaging in conversation with the user; and a controller device, external to the device for engaging in conversation, for resetting the timer to the beginning of the timer duration before the timer duration expires whenever the controller device receives an indication indicating continuation of conversation with the user.

[0081] Example 33 can be Example 32, further including a listening device for sending a session state indicator indicating that a keyword has been detected or that a session with the user has begun to start, to a controller device, or the dialogue engine sending a desired session state indicator indicating the start of a session with the user to the controller.

[0082] Example 34 may be Example 33, wherein the controller means may further include means for activating or causing the session means to be activated and transferring control to the session means after receiving a session state indicator indicating that a keyword has been detected or a session has begun with the user or an expected session state indicator indicating that the dialogue engine has begun a session with the user.

[0083] Example 35 may be Example 34, wherein the controller device may further include means for disabling the listening device or causing the listening device to be disabled upon receiving a session state indicator indicating that a keyword has been detected or that a session with the user has begun, or upon receiving a session state indicator indicating a desire to begin a session with the user.

[0084] Example 36 can be Example 34, wherein the conversation device may further include means for recognizing a user utterance after the activation of a timer, and means for responding to a user utterance.

[0085] Example 37 can be Example 36, wherein the controller means may further include means for causing a timer to be reset upon receiving a session state indicator indicating the continuation of the session.

[0086] Example 38 may be Example 37, wherein the session device may further include means for sending a session state indicator indicating that the session device recognizes a user utterance, the session device responds to a user utterance, or the session device detects a characteristic of continuing the session to the controller device.

[0087] Example 39 can be Example 36, wherein the controller means may further include means for activating or causing the listening device to be activated upon receiving a session state indicator indicating that the session is in an ended state or a timer has expired, and for transferring control to the listening device.

[0088] Example 40 may be Example 35, wherein the session device may further include means for detecting a session end feature, and the controller device includes means for activating the listening device and passing control to the listening device after receiving a session state indicator from the session device indicating that a session end feature has been detected.

[0089] Example 41 may be any of Examples 33-40, wherein the control device may further include means for activating or causing the listening device to be activated upon receiving a session state indicator indicating that the end of the session has been detected, and for transferring control to the listening device.

[0090] While certain embodiments have been illustrated and described for purposes of description, various alternative and / or equivalent embodiments or implementations calculated to achieve the same purpose may replace the illustrated and described embodiments without departing from the scope of this disclosure. This application is intended to cover any adaptations or variations of the embodiments discussed herein. Therefore, it is expressly intended that the embodiments described herein be defined solely by the claims.

[0091] Wherein this disclosure refers to an element or a "first" element or its equivalent, such disclosure includes one or more such elements, neither requiring nor excluding two or more such elements. Furthermore, the sequential indicators for the identified elements (e.g., first, second, or third) are used to distinguish between the elements and do not indicate or imply a requirement or limitation on the number of such elements, nor do they indicate a particular position or order of such elements unless otherwise specified.

Claims

1. A device for engaging in natural machine conversation with a user, comprising: The listening component includes a first automatic speech recognition engine for detecting keywords that indicate the start of a conversation; A dialogue engine, including a second automatic speech recognition engine with higher speech recognition capabilities than the first automatic speech recognition engine, is used to engage in a dialogue with the user during the session, the dialogue engine being separate from and distinct from the listening component; as well as A controller, coupled to the listening component and the dialogue engine, selectively activates or causes one of the listening component or the dialogue engine to be activated, at least in part based on the state of the session, and transfers control to the activated listening component or the activated dialogue engine, the state of the session including at least a start state and an end state. The controller is configured to: initially activate the listening component or cause the listening component to be activated and deactivate the dialogue engine or cause the dialogue engine to be deactivated; deactivate the listening component or cause the listening component to be deactivated and activate the dialogue engine or cause the dialogue engine to be activated, at least in part based on the start state of the session; and activate the listening component or cause the listening component to be activated and deactivate the dialogue engine or cause the dialogue engine to be deactivated, at least in part based on the end state of the session. Furthermore, the state of the session is indicated by a session state indicator, which includes at least the pitch, amplitude, and tempo of the user's audio input.

2. The device as described in claim 1, characterized in that, The listening component consumes less power during its operation than the dialogue engine consumes during its operation.

3. The device as described in claim 1, characterized in that, The listening component is used to send a session state indicator to the controller indicating that the keyword has been detected or the session has started, or the dialogue engine is used to send a desired session state indicator to the controller indicating that the session has started.

4. The device as described in claim 3, characterized in that, The controller is configured to activate or cause the dialogue engine to be activated upon receiving the session state indicator indicating that the keyword or the start of the session has been detected, or upon receiving the expected session state indicator indicating that the dialogue engine has started the session, and to pass control to the dialogue engine.

5. The device as described in claim 4, characterized in that, The controller is configured to further disable the listening component or cause the listening component to be disabled after receiving the session state indicator indicating that the keyword or the start of the session has been detected, or after receiving the session state indicator indicating that the start of the session is expected.

6. The device as described in claim 4, characterized in that, The dialogue engine is used to set a timer after activation, recognize user speech, and respond to the user speech.

7. The device as described in claim 6, characterized in that, The controller is configured to reset the timer or cause the timer to be reset upon receiving a session state indicator that indicates the continuation of the session.

8. The device as described in claim 7, characterized in that, The dialogue engine is used to send, and the controller is used to receive from the dialogue engine, a session state indicator indicating that the dialogue engine recognizes a user utterance, the dialogue engine responds to a user utterance, or the dialogue engine detects a feature indicating that the session continues.

9. The device as described in claim 6, characterized in that, The controller is configured to activate or cause the listening component to be activated upon receiving a session state indicator indicating that the session is in an ended state or the timer has expired, and to pass control to the listening component.

10. The device as described in claim 5, characterized in that, The dialogue engine is used to further detect session end characteristics, and the controller is used to activate the listening component and pass control to the listening component after receiving a session state indicator from the dialogue engine indicating that session end characteristics have been detected.

11. The device according to any one of claims 1-8, characterized in that, The controller is configured to activate the listening component and pass control to the listening component upon receiving a session state indicator indicating that the end of the session has been detected.

12. A machine implementation method for engaging in natural conversation with a user, comprising: After the dialogue engine is activated, the dialogue engine of the computer device sets the timer of the dialogue engine, and the dialogue engine also includes a first automatic speech recognition engine. After the timer is set, the dialogue engine initiates a conversation with the user of the computer device; as well as Upon expiration of the timer, the dialogue engine notifies the controller of the computer device of the timer's expiration. Specifically, upon notification of the timer's expiration, the dialogue engine is deactivated and the listening component of the computing device is activated instead to detect keywords or initiate a conversation with the user. The dialogue engine and the listening component are separate and distinct from each other. The listening component has a second automatic speech recognition engine with a lower speech recognition capability than the first automatic speech recognition engine. Furthermore, the method further includes: upon receiving a session state indicator, the controller selectively activates one of the listening components or the dialogue engine, or causes one of the listening components or the dialogue engine to be activated, and transfers control to the activated listening component or the activated dialogue engine, wherein the session state indicator includes at least the pitch, amplitude, and tempo of the user's audio input.

13. The method of claim 12, further comprising the listening component of the computer device sending a session state indicator indicating that a keyword has been detected or that a session with the user has begun to be initiated to the controller, or the dialogue engine sending a desired session state indicator indicating that a session with the user has begun to be initiated to the controller.

14. The method of claim 13, further comprising, upon receiving the session state indicator indicating the detection of the keyword or the start of the session with the user, or the session state indicator indicating the expectation that the dialogue engine will begin a session with the user, the controller activates or causes the dialogue engine to be activated, and passes control to the dialogue engine.

15. The method of claim 14, further comprising, upon receiving the session state indicator indicating the detection of the keyword or the start of the session with the user, or upon receiving the session state indicator indicating a desire to begin a session with the user, the controller deactivates the listening component or causes the listening component to be deactivated.

16. The method of claim 14, further comprising: setting a timer by the dialogue engine after activation; recognizing user utterances by the dialogue engine; and responding to user utterances by the dialogue engine.

17. The method of claim 16, further comprising, upon receiving a session state indicator indicating the continuation of the session, having the controller reset the timer or causing the timer to be reset.

18. The method of claim 17, further comprising sending a session state indicator, indicating that the dialogue engine recognizes a user utterance, the dialogue engine responds to a user utterance, or the dialogue engine detects a feature indicating that the session continues, to the controller.

19. The method of claim 16, further comprising, upon receiving a session state indicator indicating that the state of the session is in an ended state or the timer has expired, the controller activates the listening component or causes the listening component to be activated, and passes control to the listening component.

20. The method of claim 15, further comprising detecting a session end feature by the dialogue engine, and, upon receiving a session state indicator from the dialogue engine indicating that a session end feature has been detected, activating the listening component by the controller and passing control to the listening component.

21. The method of claim 12, further comprising, upon receiving a session state indicator indicating the detection of the end of the session, the controller activates the listening component and passes control to the listening component.

22. One or more computer-readable media (CRMs) having instructions to cause a computer device to perform any one of the methods of claims 12-21 in response to the execution of the instructions.

23. An apparatus for engaging in natural machine conversation with a user, comprising: A conversational device for engaging in conversation with a user of the device, the conversational device having a first automatic speech recognition engine; The session device includes a timer setting device for setting a timer duration after the device for having a session with the user is activated; as well as A controller device, external to the device for the session, is configured to, upon receiving a session status indicator, selectively activate one of the session device or the device's listening device, or cause one of the session device or the listening device to be activated, and transfer control to the activated session device or the activated listening device. The session status indicator includes at least the pitch, amplitude, and tempo of the user's audio input. Whenever the controller device receives a session status indicator indicating continuation of the session with the user, before the expiration of a timer duration, the controller device causes the timer device to reset the timer to the beginning of the timer duration and keeps the listening device deactivated until the session with the user ends. The listening device is used to detect keywords or initiate a session with the user and is separate from and distinct from the session device. The listening device has a second automatic speech recognition engine with a lower speech recognition capability than the first automatic speech recognition engine.

24. The device as claimed in claim 23, characterized in that, The listening device is used to send a session state indicator indicating that the keyword has been detected or that a session with the user has begun to start, to the controller device, or the dialogue engine sends a desired session state indicator indicating that a session with the user has begun to start to the controller.

25. The device as claimed in claim 24, characterized in that, The control device further includes means for activating or causing the listening device to be activated upon receiving a session state indicator indicating that the end of the session has been detected, and for transferring control to the listening device.

Images