Determining whether to automatically resume first automatic assistant session after aborting interruption of second session

The described techniques optimize session management in standalone multimodal assistant devices by determining whether to resume or transition based on session characteristics, addressing resource wastage and user convenience issues.

CN119068879APending Publication Date: 2024-12-03GOOGLE LLC
View PDF 0 Cites 2 Cited by

Patent Information

Application Number
CN202411270188.1
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Priority Date
2018-05-07
Filing Date
2019-05-01
Publication Date
2024-12-03

AI Technical Summary

Technical Problem

Existing standalone multimodal assistant devices face challenges in managing multiple overlapping conversations efficiently, leading to resource wastage and user inconvenience when an active session interrupts a prior session without clear guidance on whether to automatically resume or not.

Method used

Implementing techniques to determine whether to automatically resume or transition to an alternative state for the previous session based on characteristics such as content classification and user interaction, optimizing resource allocation and user interaction efficiency.

Benefits of technology

Enhances resource management and user interaction by selectively resuming or transitioning sessions, reducing unnecessary resource consumption and user input, and improving task execution efficiency.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN119068879A_ABST
    Figure CN119068879A_ABST
Patent Text Reader

Abstract

The invention discloses determining whether to automatically resume a first automatic assistant session after aborting an interruption of a second session. It is determined whether to perform following an abortion of a second automatic assistant session that has been interrupted and replaced by a previous first automatic assistant session: (1) automatically resume the previous first automatic assistant session, or (2) transition to an alternative automatic assistant state in which the previous first session is not automatically resumed. Implementations also involve selectively, based on the determination and after abortion of the second automatic assistant session, automatically recovering or transitioning the interrupted previous first automatic assistant session to the state in which the first session is not automatically recovered.
Need to check novelty before this filing date? Find Prior Art

Description

Description of the case

[0001] This application is a divisional application of Chinese invention patent application No. 201980037325.8, with a filing date of May 1, 2019. Technical Field

[0002] The present disclosure relates to determining whether to automatically resume a first automated attendant session after suspending an interrupted second session. Background Art

[0003] Humans may engage in human-computer conversations using interactive software applications referred to herein as "automated assistants" (also referred to as "chatbots," "interactive personal assistants," "intelligent personal assistants," "personal voice assistants," "conversational agents," etc.). For example, humans - who may be referred to as "users" when they interact with automated assistants - may provide commands, queries, and / or requests using free-form natural language input. The free-form natural language input may include spoken utterances converted to text using automatic speech recognition and / or typed free-form natural language input.

[0004] Standalone voice response speakers that provide users with the ability to audibly engage with automated assistants are becoming increasingly common. These devices typically include only minimal hardware input mechanisms to supplement the microphone(s), other than perhaps a mute button, a touch-sensitive interface for adjusting volume, etc. The goal of these speakers is to allow users to easily audibly engage with automated assistants without requiring the user to physically interact with certain user interface elements (such as a keyboard or mouse) to perform various tasks.

[0005] Traditional stand-alone voice response speakers typically lack a full display. These traditional stand-alone voice response speakers tend to include relatively simple visual output mechanisms, such as light emitting diodes, etc., at most, which are able to convey simple messages using basic colors and / or animations. The next generation of stand-alone voice response speakers may include more robust visual output mechanisms, such as displays or even touch screen displays. In contrast to stand-alone voice response speakers, these devices will be referred to as "stand-alone multimodal assistant devices" in this article. As with traditional stand-alone interactive speakers, stand-alone multimodal assistant devices can be designed to interact with a keyboard, mouse, or other complex physical input components vocally, and will typically not include a keyboard, mouse, or other complex physical input components. However, stand-alone multimodal assistant devices will typically include (multiple) microphones for vocal interaction and a touch screen for interacting via various touch inputs received via the touch screen. Summary of the invention

[0006] With standalone multimodal assistant devices, users engaging with these devices can typically interact with the devices using touch input or verbal speech. In addition, users can engage with the devices when they are relatively close to the device (e.g., near or within a few feet), relatively far from the device (e.g., more than ten feet away), or even when they move around the environment. For example, a user can engage with a multimodal assistant device to seek assistance in performing a multi-step task (such as installing and configuring a new smart thermostat). During the performance of the task, the user can interact with the multimodal assistant device via touch input and / or verbal input to cause the multimodal assistant device to render various steps and / or other guidance related to installing and configuring the new smart thermostat auditorily and / or graphically. When providing such input and / or when ascertaining content rendered by the multimodal assistant device, the user can move throughout the environment. As a result, the user's viewpoint relative to the device's display may change, thereby affecting the user's ability to view content rendered via the display. In addition, the user's changing position may affect his / her ability to hear any content rendered auditorily via the (multiple) speakers of the multimodal device.

[0007] In view of these and other considerations, embodiments disclosed herein render only content relevant to an active session with a user during the active session. For example, in response to the spoken utterance “walk me through installing smartthermostat X”, the display of an independent multimodal assistant device and the speaker(s) of the device may render exclusively content related to installing and configuring “smart thermostat X”. In other words, any previously rendered content may be completely replaced by the content of the active session. For example, assume that in response to a previous spoken utterance of “today'sweather forecast”, a daily weather forecast was rendered on a display in a previous session. The display of the weather forecast may be replaced in a new active session by visual content related to the installation of “smart thermostat X” in response to the spoken utterance of “walk me through installing smart thermostat X”. In these and other ways, rendering of content for the active session can utilize the full range of the display and / or speaker(s) of the multimodal assistant device, without “splitting” the display to present content from both sessions, and without simultaneously rendering audio from two different sessions (e.g., with one of the sessions at a “lower” volume). In addition to saving resources by preventing simultaneous rendering of content from two sessions, this can also enable the user to more easily and quickly ascertain content in the active session, thereby potentially shortening the duration of the active session.

[0008] While rendering only content relevant to an active session provides various advantages, it may also present problems when an active session interrupts a prior session before the prior session is completed. For example, upon abort of an active session that interrupted a prior session, it may be unclear as to whether the interrupted prior session should: automatically resume (e.g., resume after abort of the active session and no further user interface input is required); not automatically resume, but suggest resume (e.g., a displayed interface element may be touchable to resume the prior session); neither automatically resume nor suggest resume, but resumeable in response to an explicit user request (e.g., a spoken utterance of "resume prior session"); or not automatically resume and terminate completely (e.g., data is cleared from memory to resume the prior session, which, in its interrupted state, is not possible without extended user interaction to recreate the state of the prior session).

[0009] Techniques that exclusively and completely terminate a previous session may directly result in a waste of computer and network resources. For example, assume that a previous session is in a state resulting from multiple human-automated assistant conversation turns, and the user wishes to return to the previous session after the interruption of the session. Through techniques that exclusively and completely terminate a previous session, the previous session cannot be restored without re-performing resource-intensive multiple human-automated assistant conversation turns to recreate the state of the previous session. Techniques that exclusively and always turn to a previous session may also directly result in a waste of computer and network resources. For example, in the event that the user does not wish to return to the previous session after the interruption of the session, re-rendering the content from the previous session and / or storing its state in memory may unnecessarily consume various resources.

[0010] Challenges exist generally in assistant devices that ensure efficient management of multiple overlapping conversations and the resources provided therefor. In particular, challenges exist in presenting parallel conversation information in an environment that includes linear presentation aspects where one or more aspects of the interface presented by the assistant are implemented as audio (e.g., via spoken utterance). While multimodal assistant devices, such as those that include a display, may provide opportunities for additional interface information to be conveyed, these multimodal assistant devices also provide greater opportunities for interactions that include multiple conversations. There are challenges in managing the use of resources applied to overlapping tasks in an assistant environment that are not amenable to solutions previously used in alternative interfaces.

[0011] In view of these and other considerations, embodiments disclosed herein relate to determining whether to perform the following operations upon termination of a second automated assistant session that interrupts and replaces a previous first automated assistant session: (1) automatically resume the previous first session, or (2) transition to an alternative state in which the previous first session is not automatically resumed. In some embodiments, when determining to transition to an alternative state in which the previous first session is not automatically resumed, the alternative state may be an alternative state in which the previous session is not automatically resumed and is completely terminated; an alternative state in which the previous session is not automatically resumed but is suggested to be resumed via a user interface output (e.g., via rendering of a graphical element); or an alternative state in which the previous session is neither automatically resumed nor suggested to be resumed but may be resumed in response to an explicit user request. When determining to transition to an alternative state in which the first session is not automatically resumed, some of these embodiments may always transition to the same alternative state. Other embodiments may select from one or more of those alternative states when determining to transition to an alternative state in which the first session is not automatically resumed.

[0012] Embodiments described herein also relate to selectively causing an interrupted previous first session to be automatically restored or to be transitioned to a state where the first session is not automatically restored after the termination of the second session. These embodiments selectively cause one of the two actions to occur based on the following determination: whether to (1) automatically restore the previous first session or (2) transition to an alternative state where the previous first session is not automatically restored. As described in detail herein, various techniques can be utilized to make such a determination, such as techniques that consider one or more characteristics of the previous first session and / or the interrupted second session. Making such a determination and selectively causing only one of the two actions to be performed can directly result in various advantages. In particular, by providing an alternative method to manage transitions between sessions, an improved allocation of resources can be achieved. For example, the selective automatic restoration of the previous first session can reduce (e.g., to zero) the amount of input that a user must perform to restore the previous first session, while being performed only selectively and based on (a plurality of) various considerations to reduce the risk of automatic restoration when it is not expected. In addition, for example, selectively transitioning to a state in which a previous first session is not automatically resumed can prevent automatic rendering of content from the first session and / or can cause the assistant device to transition to a low-power state more quickly, while being performed only selectively and based on (multiple) various considerations to reduce the risk of non-automatic resumption when undesirable. In addition, the selective execution of one of the two actions provides an improved interaction protocol that facilitates human-automatic assistant interaction in the execution of various tasks (such as controlling a smart device, configuring a smart device, installing a smart device, etc.).

[0013] In some embodiments, determining whether to perform the following operations after termination of a second session that interrupts and replaces a previous first session is based at least in part on one or more characteristics of the previous first session and / or one or more characteristics of the second session: (1) automatically resuming the previous first session, or (2) transitioning to an alternative state in which the previous first session is not automatically resumed. In some of those embodiments, the utilized characteristics of the first session may include a classification assigned to the first content and / or the utilized characteristics of the second session may include a classification assigned to the second content. The classification may indicate, for example, whether the corresponding content is transient or persistent. In this way, it can be recognized that sessions with different classifications can be treated differently when included in a multi-session environment, thereby appropriately allocating resources to the content they need or expect.

[0014] Persistent content may include, for example, content rendered through multiple conversation turns and / or content dynamically rendered depending on user interface input during a conversation turn. For example, content rendered when guiding a user to install and configure a smart device may be persistent because the content is rendered through multiple conversation turns and / or because the content is rendered in a dynamic manner (e.g., if the user selects wiring option A in a conversation turn, another content A will be provided, and if the user selects wiring option B in a conversation turn instead, another content B will be provided). Persistent content may additionally or alternatively include, for example, content whose entire rendering takes at least a threshold duration and / or content belonging to a certain type. For example, many (or all) videos may be classified as persistent, such as music videos, which include at least a threshold duration of content rendered visually and audibly. In contrast, transient content may include static and / or content rendered only in a single conversation turn, such as audible and / or visual weather forecasts, definitions of terms, answers to questions. In some embodiments, content may be classified as persistent or transient based at least in part on the source of the content. For example, content from an agent providing video content may all be classified as persistent, while content from an agent providing weather information may be classified as transient. In some embodiments, content may be classified as persistent or transient based at least in part on an intent determined based on a spoken utterance or a notification to which the content is responsive. For example, as described herein, a spoken utterance may be processed to derive an intent, and / or a notification may be associated with a corresponding intent, and some intents may be associated with persistent content, while other intents may be associated with transient content. Additional and / or alternative techniques for classifying content as transient or persistent may be provided.

[0015] In some embodiments, determining to automatically resume the previous first session may be based on the content of the previous first session being classified as persistent and the content of the second session being classified as transient. In some additional or alternative embodiments, determining to transition to an alternative state in which the previous first session is not automatically resumed may be based on the content of the previous first session being classified as transient. In some of those embodiments, the alternative state is a state in which the previous first session is not automatically resumed and is completely terminated. In some additional or alternative embodiments, determining to transition to an alternative state in which the previous first session is not automatically resumed may be based on the content of the previous first session being classified as persistent and the content of the second session being also classified as persistent. In some versions of those embodiments, the alternative state is a state in which the previous first session is not automatically resumed but is suggested to be output via a user interface (e.g., via rendering a graphical element on a "home" screen, described in more detail herein). In some other versions of those embodiments, the alternative state is a state in which the previous session is neither automatically resumed nor suggested to be resumed but is recoverable in response to an explicit user request. In some additional or alternative embodiments, determining to automatically resume a previous first session may be based on another determination that content of the previous first session is classified as persistent, content of the second session is classified as persistent, and content of the previous first session implements an entity and a defined relationship exists between the entity content of the second session.

[0016] In various embodiments described herein, interruption data is received during rendering of first content of a first session, and in response to receiving the interruption data, alternative content is rendered during a second session. In some of those embodiments, the interruption data is a user's verbal input, and is determined to be interruption data based on the verbal input including a request for alternative content and the alternative content being different from the first content of the first session. For example, the first content of the first session may be related to configuring a smart device, and the verbal input may be "how do you make a mint julep", and may be determined to be interruption data based on including a request for content (a guide related to making mint juleps) that is unrelated to the smart device configuration content of the first session. In contrast, if the first content displayed during the first session is one of a plurality of steps related to configuring a smart device, and the verbal input is a "next" step, the verbal input will not be determined to be not interruption data, but instead will be determined to be a request to render the next portion of the first content.

[0017] In some additional or alternative embodiments, the interruption data may be a user interface input indicating a desire to interact with a notification received during the first session and rendered visually and / or audibly during the rendering of the first content during the first session. For example, the notification may be an incoming audio and / or visual call, and the interactive graphical interface element may be overlaid on the visually rendered content of the first session. A positive selection of the notification graphical interface element may be the interruption data, and the alternative content rendered during the second session may be an audio and / or visual call. The positive selection may be, for example, a specific positive touch selection (e.g., swiping right or double-clicking) or a positive voiced speech (e.g., "OK", "accept the notification"). Conversely, a negative selection of the graphical interface element (e.g., a specific negative touch selection, such as swiping down) may cause the notification to be canceled without interrupting the first session. As used herein, the interruption data refers to data indicating a desire to render the alternative content as part of the alternative session, and may be used in conjunction with a user interface input that attempts to terminate the current termination (e.g., a verbal command "stop" or "pause", an interactive "X" touch, or other termination graphical element).

[0018] In various embodiments, when the interruption data is received, the first session data of the interrupted first session can be stored and used to automatically restore the first session in the state of the first session. In some of those embodiments, the first session data is optionally persisted only in response to determining to restore the first session. The first session data can be stored in various ways. In some embodiments, the entire transcript of the human-computer dialogue of the first session can be retained so that, for example, the state of the first session can be reconstructed or restored by detecting intent, slot values, entities, etc. in the retained transcript, for example, in a hurry and / or as needed. In addition or alternatively, in some embodiments, only the core elements of the state (such as (multiple) detected intents, slot values, mentioned entities, etc.) can be retained in various formats, such as JavaScript Object Notation ("JSON") or other similar formats. The first session data can be retained in various locations. In some embodiments, the first session data can be saved in a memory local to a computing device operated by a user to engage with an automated assistant. This situation can produce various technical benefits, such as reducing the waiting time for the restoration of the first session. Additionally or alternatively, in some implementations, the first session data may be saved remotely from the user's computing device, such as in the memory of one or more computing systems operating in conjunction with what are often referred to as “cloud-based services.”

[0019] The above description is provided only as an overview of the various embodiments described herein. Additional descriptions of those various embodiments as well as additional embodiments are provided herein.

[0020] In some implementations, a method performed by one or more processors is provided and includes receiving user input data indicating user input detected via a user interface input component of a client device. The method also includes identifying first content responsive to the user input based on the user input data. The method also includes, in response to receiving the user input data, causing the client device to render at least a portion of the first content during a first session. The method also includes receiving interruption data during the rendering of the first content by the client device during the first session. In response to another user interface input of a user detected during the rendering of the first content during the first session, the interruption data is received. The method also includes, in response to receiving the interruption data, causing the client device to render substitute content during a second session that at least temporarily replaces the first session. The substitute content is different from the first content, and causing the client device to render the substitute content during the second session includes causing the client device to render the substitute content in place of the first content. The method also includes determining whether to perform the following operations after the termination of the second session: causing the client device to automatically resume the first session, or causing the client device to transition to an alternate state in which the client device does not automatically resume the first session. The determination is based at least on one or more characteristics of the first session. The method also includes, after the termination of the second session and depending on the determination, selectively causing the client device to automatically resume the first session or transition to an alternate state in which the first session is not automatically resumed.

[0021] These and other implementations of the techniques described herein may include one or more of the following features.

[0022] In some implementations, the user input data includes spoken utterance data indicative of spoken utterances of a user detected via one or more microphones of the client device.

[0023] In some embodiments, the at least one characteristic of the first session on which the determination is based comprises a classification assigned to the first content. In some versions of those embodiments, the classification assigned to the first content indicates whether the first content is transient or persistent. In some of those versions, the determination comprises determining to cause the client device to transition to an alternate state in which the client device does not automatically resume the first session based on the classification assigned to the content indicating that the first content is transient.

[0024] In some embodiments, the determination is based on at least one characteristic of the first session indicating whether the first content is transient or persistent, and the determination is also based on one or more characteristics of the second session. In some of those embodiments, at least one characteristic of the second session indicates whether the substitute content is transient or persistent. In some versions of those embodiments, the determination includes determining to cause the client device to automatically resume the first session based on a classification assigned to the first content indicating that the first content is persistent and at least one characteristic of the second session indicating that the substitute content is transient. In some other versions of those embodiments, the determination includes determining to cause the client device to transition to an alternate state in which the client device does not automatically resume the first session based on a classification assigned to the first content indicating that the content is persistent and at least one characteristic of the second session indicating that the substitute content is persistent.

[0025] In some embodiments, the other user interface input is another spoken utterance of the user. In some of those embodiments, the method further comprises determining that the other spoken input is interruption data based on the other spoken input comprising a request for alternate content and the alternate content is different from the first content of the first session.

[0026] In some implementations, the at least one characteristic of the first session based on which the determination is based includes an entity implemented by the first content, and the determination includes determining to cause the client device to automatically resume the first session based on determining a relationship between the alternative content and the entity implemented by the first content.

[0027] In some embodiments, the method further comprises: during rendering of the first content by the client device during the first session, receiving a notification to be rendered at the client device; and in response to receiving the notification, causing the client device to render the notification. In some of those embodiments, the other user interface input is an affirmative touch or verbal input by a user provided in response to the client device rendering the notification, the affirmative input by the user indicating a desire of the user to interact with the notification, and the alternative content is based on the notification.

[0028] In some embodiments, the method further comprises storing first session state data for the first session, the first session state data indicating a state of the first session when the interruption data was received. In those embodiments, causing the client device to automatically resume the first session comprises using the first session state data to resume the first session at the state of the first session when the interruption data was received.

[0029] In some embodiments, determining includes determining to cause the client device to transition to a state in which the client device does not automatically resume the first session. In some of those embodiments, the method also includes, in response, clearing session data for the first session from a memory of the client device or a remote server in network communication with the client device.

[0030] In some implementations, the client device includes a display and at least one speaker, and presenting the content includes rendering the content via the display and the at least one speaker.

[0031] In some embodiments, the alternate state in which the client device does not automatically resume the first session includes display of a home screen or an ambient screen. In some versions of those embodiments, the display of the home screen or the ambient screen lacks any reference to the first session. In some other versions of those embodiments, the display of the home screen or the ambient screen includes a selectable graphical interface element that can be selected to resume the first session.

[0032] In addition, some embodiments include one or more processors of one or more computing devices, wherein the one or more processors are operable to execute instructions stored in an associated memory, and wherein the instructions are configured to cause the execution of any of the foregoing methods. Some embodiments also include one or more non-transitory computer-readable storage media storing computer instructions executable by the one or more processors to perform any of the foregoing methods. BRIEF DESCRIPTION OF THE DRAWINGS

[0033] Figure 1 is a block diagram of an example environment in which implementations disclosed herein may be implemented.

[0034] Figure 2 Depicted are example state machines that may be implemented in accordance with various implementations.

[0035] Figure 3 Depicted is a flow chart illustrating an example method according to implementations disclosed herein.

[0036] Figure 4 An example is illustrated of rendering first content during a first session, rendering alternate content during a second session in response to receiving interruption data during rendering of the first content during the first session, and automatically resuming the first session upon suspension of the second session.

[0037] Figure 5 An example is illustrated of rendering first content during a first session, rendering alternate content during a second session in response to receiving interruption data during rendering of the first content during the first session, and transitioning upon suspension of the second session to an alternate state in which the first session is suggested to be resumed but is not automatically resumed.

[0038] Figure 6 Another example is illustrated of rendering first content during a first session, rendering alternate content during a second session in response to receiving interruption data during rendering of the first content during the first session, and automatically resuming the first session upon suspension of the second session.

[0039] Figure 7 An example is shown of rendering first content during a first session, rendering alternate content during a second session in response to receiving interruption data during rendering of the first content during the first session, and transitioning upon suspension of the second session to an alternate state in which the first session is not automatically resumed and resumption is not suggested.

[0040] Figure 8 Another example is illustrated of rendering first content during a first session, rendering alternate content during a second session in response to receiving interruption data during rendering of the first content during the first session, and resuming the first session upon suspension of the second session.

[0041] Fig. 9 is a block diagram of an example computing device. DETAILED DESCRIPTION

[0042] Now go to Figure 1 , illustrating an example environment in which the technology disclosed herein can be implemented. The example environment includes multiple client computing devices 106 1-N Each client device 106 may execute a respective instance of an automated assistant client 118. One or more cloud-based automated assistant components 119, such as a natural language understanding engine 135, may be implemented on one or more computing systems (collectively referred to as "cloud" computing systems) that are communicatively coupled to the client devices 106 via one or more local area networks and / or wide area networks (e.g., the Internet), generally indicated at 110. 1-N .

[0043] In some embodiments, an instance of automated assistant client 118, through its interaction with one or more cloud-based automated assistant components 119, can form what appears to be a logical instance of automated assistant 120 from the user's perspective, and the user can participate in a human-computer dialogue with the logical instance of automated assistant 120. Figure 1 1 and 2 depict two examples of such automated assistants 120. The first automated assistant 120A, surrounded by a dashed line, serves a first user (not depicted) operating a first client device 1061 and includes an automated assistant client 1181 and one or more cloud-based automated assistant components 119. The second automated assistant 120B, surrounded by a double-dashed line, serves a first user (not depicted) operating a first client device 1061. N A second user (not depicted) is provided with services and includes an automated assistant client 118 N and one or more cloud-based automated assistant components 119 .

[0044] Thus, it should be understood that each user engaging with an automated assistant client 118 executing on a client device 106 may actually engage with his or her own logical instance of automated assistant 120. For the sake of brevity and simplicity, the term "automated assistant" as used herein to "serve" a particular user will refer to the combination of an automated assistant client 118 executing on a client device 106 operated by the user and one or more cloud-based automated assistant components 119 (which may be shared among multiple automated assistant clients 118). It should also be understood that in some implementations, automated assistant 120 may respond to requests from any user, regardless of whether a particular instance of automated assistant 120 is actually "serving" that user.

[0045] Client device 106 1-N It may include, for example, one or more of a desktop computing device, a laptop computing device, a tablet computing device, a mobile phone computing device, a computing device of a user's vehicle (e.g., an in-vehicle communication system, an in-vehicle entertainment system, an in-vehicle navigation system), a standalone interactive speaker, a smart appliance such as a smart TV, and / or a wearable device of a user that includes a computing device (e.g., a user's watch with a computing device, a user's glasses with a computing device, a virtual or augmented reality computing device). Additional and / or alternative client computing devices may be provided.

[0046] For the purpose of this disclosure, Figure 1 In the embodiment of the present invention, the first client device 1061 takes the form of a stand-alone multimodal auxiliary device having a speaker 1091 and a display 1111, and may lack complex hardware input components (in some embodiments, other than the display 111 being a touch screen) such as a keyboard or mouse. Although the techniques described herein will be described in the context of being performed using a stand-alone multimodal auxiliary device such as 1061, this is not meant to be limiting. The techniques described herein may be implemented on client devices having other form factors (but still lacking a standard keyboard and mouse), such as a vehicle computing device that is intended to be interacted with primarily via voice interaction and / or touch interaction. The second client device 106 N is an independent voice response speaker, which includes a speaker 109 N , the automated assistant 120A can communicate with the speaker 109 N Provide natural language output. N With the second client device 106 N The techniques described herein may additionally or alternatively be implemented in association with the audio content of the rendered session.

[0047] As described in more detail herein, automated assistant 120 communicates with one or more client devices 106 via 1-NThe user interface input and output devices of the client device 1061 may participate in a human-computer conversation session with one or more users. In the case of a stand-alone multimodal auxiliary device (such as the client device 1061), these input devices may be limited to a microphone (not depicted) and a display 111 (in embodiments where the display 111 is a touch screen) and optionally other sensors (e.g., PIR, camera), which may be used, for example, to detect the presence of nearby people. In some embodiments, in response to a user via the client device 106 1-N Automated assistant 120 may engage in a human-computer conversation session with a user based on user interface input provided by one or more user interface input devices of the automated assistant 120. In some of those embodiments, the user interface input is explicitly directed to automated assistant 120. For example, the specific user interface input may be a user interaction with a hardware button and / or a virtual button (e.g., a tap, a long press), a verbal command (e.g., "Hey Automated Assistant"), and / or other specific user interface input.

[0048] In some embodiments, even when the user interface input is not explicitly directed to the automated assistant 120, the automated assistant 120 may participate in a human-computer dialogue session in response to the user interface input. For example, the automated assistant 120 may examine the content of the user interface input and participate in a dialogue session in response to certain terms present in the user interface input and / or based on other prompts. In many embodiments, the user may issue commands, conduct searches, etc., and the automated assistant 120 may utilize speech recognition to convert the speech into text, and respond to the text accordingly, for example, by providing search results, general information, and / or taking one or more response actions (e.g., playing media, starting a game, ordering food, etc.). In some embodiments, the automated assistant 120 may additionally or alternatively respond to the speech without converting the speech into text. For example, the automated assistant 120 may convert the speech input into an embedding, into an entity representation (indicating one / more entities present in the speech input) and / or other "non-text" representations, and operate on such non-text representations. Therefore, the embodiments described herein as operating based on text converted from speech input may additionally and / or alternatively directly operate on the speech input and / or other non-text representations of the speech input.

[0049] Client computing device 106 1-N Each of the one or more computing devices running the cloud-based automated assistant component 119 may include one or more memories for storing data and software applications, one or more processors for accessing data and executing applications, and other components that facilitate communication over the network. 1-NAnd / or the operations performed by automated assistant 120 may be distributed across multiple computer systems. Automated assistant 120 may be implemented, for example, as a computer program running on one or more computers in one or more locations coupled to each other via a network.

[0050] As described above, in various embodiments, the client computing device 106 1-N Each of the automated assistant clients 118 may operate. In various embodiments, each automated assistant client 118 may include a corresponding voice capture / text to speech (“TTS”) / speech to text (“STT”) module 114. In other embodiments, one or more aspects of the voice capture / TTS / STT module 114 may be implemented separately from the automated assistant client 118. Each voice capture / TTS / STT module 114 may be configured to perform one or more functions: for example, capturing the user's voice via a microphone (voice capture); converting the captured audio to text and / or other representations or embeddings (STT); and / or converting text to speech (TTS). For example, in some embodiments, because the client device 106 may be relatively constrained in terms of computing resources (e.g., processor cycles, memory, battery, etc.), the local STT module 114 for each client device 106 may be configured to convert a limited number of different spoken phrases into text (or other forms, such as lower dimensional embeddings). Other voice inputs may be sent to a cloud-based automated assistant component 119, which may include a cloud-based STT module 117.

[0051] The cloud-based STT module 117 can be configured to utilize the virtually unlimited resources of the cloud to convert the audio data captured by the voice capture / TTS / STT module 114 into text (which can then be provided to the natural language processor 122). The cloud-based TTS module 116 can be configured to utilize the virtually unlimited resources of the cloud to convert text data (e.g., natural language responses formulated by the automated assistant 120) into computer-generated speech output. In some embodiments, the TTS module 116 can provide the computer-generated speech output to the client device 106 for direct output, such as using one or more speakers. In other embodiments, the text data (e.g., natural language responses) generated by the automated assistant 120 can be provided to the voice capture / TTS / STT module 114, which can then convert the text data into computer-generated speech that is output locally.

[0052] Automated assistant 120 (and in particular, cloud-based automated assistant component 119) may include natural language understanding engine 135, the aforementioned TTS module 116, the aforementioned STT module 117, and other components described in more detail below. In some embodiments, one or more engines and / or modules of automated assistant 120 may be omitted, combined, and / or implemented in components separate from automated assistant 120. In some embodiments, one or more components of automated assistant 120, such as natural language understanding engine 135, speech capture / TTS / STT module 114, etc., may be implemented at least in part on client device 106 (e.g., outside of the cloud).

[0053] In some embodiments, the automated assistant 120 responds to the client device 106 1-N The automated assistant 120 may generate responsive content based on various inputs generated by one of the users during the human-machine dialogue session with the automated assistant 120. The automated assistant 120 may provide the responsive content (e.g., over one or more networks while separate from the user's client device) for presentation to the user as part of the dialogue session. For example, the automated assistant 120 may respond to a request received via the client device 106 1-N The free-form natural language input provided by one of the embodiments is used to generate the response content. As used herein, the free-form natural language input is an input formulated by the user and is not constrained by a set of options presented for the user to select.

[0054] The natural language processor 122 of the natural language understanding engine 135 processes the user's 1-N The generated natural language input may be processed by natural language processor 122 and may generate annotation output (e.g., in text form) for use by one or more other components of automated assistant 120. For example, natural language processor 122 may process natural language free-form input generated by a user via one or more user interface input devices of client device 1061. The generated annotation output includes one or more annotations of the natural language input and, optionally, one or more (e.g., all) terms of the natural language input.

[0055] In some embodiments, the natural language processor 122 is configured to identify and annotate various types of grammatical information in the natural language input. For example, the natural language processor 122 may include a morphological engine that can separate individual words into morphemes and / or annotate these morphemes, for example, with the class of the morpheme. The natural language processor 122 may also include a part-of-speech tagger that is configured to annotate the term with the grammatical role of the term. In addition, for example, in some embodiments, the natural language processor 122 may additionally and / or alternatively include a dependency parser that can be configured to determine the syntactic relationship between terms in the natural language input.

[0056] In some embodiments, the natural language processor 122 may additionally and / or alternatively include an entity tagger configured to annotate entity references in one or more segments, such as references to people (including, for example, literary characters, celebrities, public figures, etc.), organizations, locations (real and fictional), etc. In some embodiments, data about entities may be stored in one or more databases, such as a knowledge graph (not depicted). In some embodiments, the knowledge graph may include nodes representing known entities (and in some cases, entity attributes) and edges connecting nodes and representing relationships between entities. For example, a "banana" node may be connected (e.g., as a child node) to a "fruit" node, which in turn may be connected (e.g., as a child node) to a "product" and / or "food" node. As another example, a restaurant called "Hypothetical Cafe" may be represented by a node that also includes attributes such as its address, the type of food served, business hours, contact information, etc. It is connected to one or more other nodes, such as a "restaurant" node, a "business" node, a node representing the city where the restaurant is located and / or the state in which the restaurant is located, etc., through an edge (e.g., representing the relationship between the child node and the parent node).

[0057] The entity tagger of the natural language processor 122 may annotate references to entities at a higher level of granularity (e.g., to enable identification of all references to a class of entities such as people) and / or at a lower level of granularity (e.g., to enable identification of all references to a specific entity such as a specific person). The entity tagger may rely on the content of the natural language input to resolve the specific entity and / or may optionally communicate with a knowledge graph or other entity database to resolve the specific entity.

[0058] In some implementations, the natural language processor 122 may additionally and / or alternatively include a coreference resolver (not depicted) configured to group or "cluster" references to the same entity based on one or more contextual cues. For example, a coreference resolver may be utilized to resolve the term "there" to "Hypothetical Cafe" in the natural language input "I liked Hypothetical Cafe last time we ate there".

[0059] In some embodiments, one or more components of the natural language processor 122 can rely on annotations of one or more other components of the natural language processor 122. For example, in some embodiments, a named entity tagger can rely on annotations from a coreference resolver and / or a dependency resolver when annotating all mentions of a particular entity. For example, in some embodiments, a coreference resolver can rely on annotations from a dependency resolver when clustering references to the same entity. In some embodiments, when processing a particular natural language input, one or more components of the natural language processor 122 can use relevant previous inputs and / or other relevant data outside of the particular natural language input to determine one or more annotations.

[0060] The natural language understanding engine 135 may also include an intent matcher 136 configured to determine the intent of a user participating in a human-machine conversation session with the automated assistant 120 based on the annotation output of the natural language processor 122. Figure 1 122, but in other embodiments, the intent matcher 136 can be an integral part of the natural language processor 122 (or more generally, a pipeline including the natural language processor 122). In some embodiments, the natural language processor 122 and the intent matcher 136 can together form the aforementioned "natural language understanding" engine 135.

[0061] Intent matcher 136 can use various techniques to determine the user's intent. In some embodiments, intent matcher 136 can have access to one or more databases 137, which include, for example, multiple mappings between grammars and response actions (or more generally, intents). Additionally or alternatively, in some embodiments, one or more databases 137 can store one or more machine learning models that are trained to generate outputs indicating the user's intent based on the user's input.

[0062] Grammars may be selected, customized (e.g., by hand), and / or learned over time, for example, to represent a user's most common intents. For example, one grammar, "Play <artist>," may be mapped to an intent that invokes a response action that causes music by <artist> to be played on a client device 106 operated by the user. Another grammar, "[weather|forcast]today," may match user queries such as "what's the weather today" and "what's the forecast for today?" As seen in the "Play <artist>" example grammar, some grammars have slots (e.g., <artist>) that can be filled with slot values ​​(or "parameters"). Slot values ​​may be determined in a variety of ways. Users typically actively provide slot values. For example, for the grammar "Order me a <topping>pizza (order for me <topping>pizza), the user might say "orderme a sausage pizza", in which case the slot is automatically filled <topping>Additionally or alternatively, if the user invokes a syntax that includes slots to be filled with slot values, without the user actively providing the slot values, automated assistant 120 can solicit those slot values ​​from the user (e.g., "what type of crust do you want on your pizza?").

[0063] In contrast to many grammars (which can be created manually), machine learning models can be automatically trained using, for example, logs of interactions between a user and an automated assistant. Machine learning models can take various forms, such as neural networks. They can be trained in various ways to predict user intent based on user input. For example, in some embodiments, training data including individual training examples can be provided. Each training example can include, for example, free-form input from a user (e.g., in text or non-text form) and can be labeled by intent (e.g., by hand). The training examples can be applied as input to a machine learning model (e.g., a neural network) to generate an output. The output can be compared to the label to determine the error. For example, the error can be used to train the model using techniques such as gradient descent (e.g., random, batch, etc.) and / or back propagation to adjust the weights associated with the hidden layers of the model. Once such a model is trained by a (usually very large) large number of training examples, it can be used to generate outputs that predict intent based on unlabeled free-form natural language input.

[0064] In some embodiments, the automated assistant 120 can facilitate (or "negotiate") transactions between a user and an application (also referred to as an agent) such as a third-party application. These applications may or may not be run on a computer system separate from those computing systems that operate, for example, the cloud-based automated assistant component 119. Thus, an intent matcher 136 can identify that a user intends to engage in an application such as a third-party application. For example, the automated assistant 120 can provide access to an application programming interface ("API") to a pizza delivery service. A user can invoke the automated assistant 120 and provide a command such as "I'd like to order a pizza". The intent matcher 136 can map the command to a grammar (which can be added to the database 137 by a third party in some cases) that triggers the automated assistant 120 to engage with a third-party pizza delivery service. The third-party pizza delivery service can provide the automated assistant 120 with a list of the minimum number of slots that need to be filled in order to fulfill the pizza delivery order. The automated assistant 120 can generate a natural language output that solicits parameters from the slots and provide it to the user (via the client device 106).

[0065] Fulfillment engine 124 may be configured to receive the intent output by intent matcher 136 and any associated slot values ​​(whether actively provided by the user or solicited from the user) and fulfill the intent. In various embodiments, fulfillment of the user's intent may result in various fulfillment information being generated / obtained, for example, by fulfillment engine 124. As will be described below, in some implementations, the fulfillment information may be provided to natural language generator 126, which may generate natural language output based on the fulfillment information.

[0066] Because intent can be fulfilled in a variety of ways, fulfillment information can take a variety of forms. Assume that a user requests pure information, such as "Where were the outdoor shots of 'The Shining' filmed?" The user's intent can be determined as a search query, for example, by intent matcher 1S6. The intent and content of the search query can be provided to fulfillment engine 124, which can be used as a search engine. Figure 1 As depicted in FIG. 1 , the search engine 124 may communicate with one or more search engines 150 configured to search for response information in a corpus of documents and / or other data sources (e.g., a knowledge graph, etc.). The fulfillment engine 124 may provide data indicative of a search query (e.g., the text of the query, a reduced dimensionality embedding, etc.) to the search engine 150. The search engine 150 may provide response information, such as "Timberline Lodge, Mt. Hood, Oregon". The response information may form part of the fulfillment information generated by the fulfillment engine 124.

[0067] Additionally or alternatively, fulfillment engine 124 may be configured to receive the user's intent and any slot values ​​provided by the user or determined using other means (e.g., the user's GPS coordinates, user preference settings, etc.), for example, from natural language understanding engine 135, and trigger a response action. Response actions may include, for example, ordering goods / services, engaging in an interactive dialogue to complete a task, starting a timer, setting a reminder, initiating a phone call, playing media, sending a message, etc. In some such embodiments, fulfillment information may include slot values ​​associated with the fulfillment, confirmation response (which may be selected from a predetermined response in some cases), etc.

[0068] As described above, the natural language generator 126 can be configured to generate and / or select natural language output (e.g., spoken words / phrases designed to mimic human speech) based on data obtained from various sources. In some embodiments, the natural language generator 126 can be configured to receive as input fulfillment information associated with the fulfillment engine 124 fulfillment intent and generate natural language output based on the fulfillment information. Additionally or alternatively, the natural language generator 126 can receive information (e.g., required slots) from other sources such as third-party applications, which can be used to customize the natural language output for the user. In addition, as described herein, the content provided to the user as output for rendering can include graphical content, optionally together with corresponding audible content.

[0069] The session engine 138 may be configured to identify a session that is initiated via the first client device 1061 or via the second client device 106 N The active automated assistant session that occurred. Conversation engine 138 may cause only content related to the active session to be rendered only during the active session. In addition, conversation engine 138 may be configured to, in response to receiving interruption data during the active first session, cause the corresponding client device to render substitute content during a second session that at least temporarily replaces the second session. The substitute content rendered during the second session is different from the content of the first session, and causing the client device to render the substitute content includes causing the client device to render the substitute content in place of the content of the first session.

[0070] The conversation engine 138 may determine that some instances of user interface input provided by the user constitute interruption data based on those instances of requesting to provide alternative content in place of content provided during the first conversation. In making such a determination, the conversation engine 138 may rely on output from the natural language understanding engine 135, such as output indicating a user's verbal input and / or other natural language input indicating a change from a current intent to a new intent. When such verbal input is received as interruption data, the conversation engine 138 may cause the alternative content responsive to the new intent and determined by the implementation engine 124 to be replaced with the content rendered by the first conversation. The conversation engine 138 may additionally or alternatively determine that the user interaction is interruption data through a notification provided during the first conversation. When such an affirmative user interaction is received, the conversation engine 138 may cause the alternative content corresponding to the notification to be replaced with the content rendered by the first conversation.

[0071] The session engine 138 may also be configured to determine whether to perform the following operations after the termination of the second session: (1) automatically resume the previous first session, or (2) transition to an alternative state in which the previous first session is not automatically resumed. The session engine 138 is also configured to selectively cause the interrupted previous first session to be automatically resumed or transition to a state in which the first session is not automatically resumed after the termination of the second session. The session engine 138 may selectively cause one of the two actions to occur based on the following determination: (1) whether to automatically resume the previous first session, or (2) transition to an alternative state in which the previous first session is not automatically resumed. As described herein, when the session engine 138 determines to transition to an alternative state in which the previous first session is not automatically resumed, the alternative state may be: an alternative state in which the previous session is not automatically resumed and terminated completely; an alternative state in which the previous session is not automatically resumed but a suggestion for resumption is output via a user interface; or an alternative state in which the previous session is neither automatically resumed nor suggested for resumption but can be resumed in response to an explicit user request.

[0072] The session engine 138 may also be configured to cause selective transitions between various states, such as the display off, ambient, home, and session states described herein, in accordance with embodiments described herein. For example, the session engine 138 may selectively transition from an interrupted session state to a state in which a previous session state is automatically restored, or alternatively a state in which a previous session state is not automatically restored. Furthermore, in embodiments in which the session engine 138 transitions to a home state in which a previous session state is not automatically restored, the session engine 138 may optionally determine whether to adapt to the home state (e.g., determine whether to include an interactive element for restoring a previous session state) based on the various considerations described herein. Figures 2 to 8 ) describes additional descriptions of these and other aspects of the conversation engine 138. It should be noted that although the conversation engine 138 Figure 1 1 as part of cloud-based automated assistant component(s) 119 , in various implementations, all or part of conversation engine 138 may be implemented by a corresponding automated assistant client 118 .

[0073] Figure 2 An example state diagram is depicted that may be implemented, for example, by the session engine 138, according to various embodiments. The depicted state diagram includes five states, display off 181, environment 183, home page 185, first session 187, and second session 189. Although five states are depicted, fewer or more states may be provided. For example, as described herein, the second session 189 is not always active, but may be created on demand in response to interruption data being received while content is being rendered in the first session 187. Furthermore, for example, additional third session states may be created on demand by the session engine 138 in response to other interruption data being received while content is being rendered in the second session 189. When such a third session state is created, transitions from the third session state may be handled in a similar manner as described herein with respect to transitions from the second session (e.g., the second session 189 may be automatically resumed, or an alternate state in which the second session 189 is not automatically resumed).

[0074] Display off 181 can be a default state where display 111 remains dormant, e.g., using little or no power. Display off 181 can maintain the current state while standalone multimodal assistant device 1061 remains alone, with no one nearby. In some implementations, while the current state is display off 181, a user (not yet detected as present) can still request activity from automated assistant 120, e.g., by speaking an invocation phrase followed by a specific request, which can transition the current state directly to the first conversation 187 state.

[0075] In some embodiments, the current state may be transitioned to the ambient 183 state when one or more people are detected in the vicinity (i.e., "occupancy"). In the ambient 183 state, the automated assistant 120 may cause the display 1111 of the client device 1061 to render ambient content that may be selected, for example, based on its aesthetic appearance. For example, one or more digital images and / or videos of a landscape or other similar content may be optionally displayed visually with an audible rendering of relaxing natural sounds. It should be noted that, although content is rendered in the ambient 183 state, the ambient 183 state is not considered a session as that term is used herein. For example, the ambient 183 state may not be considered a session because the content rendered in the ambient 183 state is not identified and rendered in response to spoken utterances or in response to the user's acceptance of a notification. In some embodiments, the current state may transition from ambient 183 back to display off 181 if it is determined, for example, that the occupant is no longer co-present with the standalone multimodal assistant device for at least a predetermined period of time.

[0076] like Figure 2 As indicated in , in some embodiments, while the current state is ambient 183, the user can still request an activity from the automated assistant 120, for example, by speaking an invocation phrase followed by a specific request, which can transition the current state to the first conversation 187 state. In other embodiments, there may be no ambient 183 state, and the current state can transition directly from display off 181 to home page 185 in response to detecting human co-presence (occupancy).

[0077] In the home page 185 state, various graphical elements may be rendered, such as suggested actions for the user to perform by interacting with the automated assistant, such as the current time, current weather conditions, a brief summary of the user's calendar, and the like. In some embodiments, data items may be displayed as cards or tiles that may or may not be interactive (e.g., depending on whether the display 111 is a touch screen). In some cases, data items may be ranked based on various criteria, such as the priority assigned to the data item (automatically or manually), the identity of the co-existing person (if determined), the time of day, the time of year, and the like. When data items are rendered as cards, for example, in a stacked form, the ranking may be reflected, for example, by the top card having the highest priority and the bottom card having a relatively lower priority. When data items are presented as tiles, for example, occupying a portion of the display 111, the ranking may be reflected, for example, in the placement of the tile (e.g., the upper left corner or upper right corner may be the highest priority) and / or in the size of the tile (e.g., the larger the tile, the higher the priority). As described in detail herein, in various embodiments, before the home 185 state is transitioned to a session including persistent content, the data item optionally includes a selectable graphical element that can be selected to resume the session. It should be noted that although content is rendered in the home 185 state, the home 185 state is not considered a session as that term is used herein. For example, the home 185 state may not be considered a session because the content rendered in the home 185 state is not identified and rendered in response to spoken utterance or in response to a user's acceptance of a notification.

[0078] While in the home 185 state, if the user engages with one or more of the graphical elements representing data items (e.g., by clicking on a tile or card), the current state can transition to the first conversation 187 state and render content responsive to the graphical element(s) with which the user interacted. For example, if one of the graphical elements is a suggestion of a music video that the user may be interested in, the first conversation 187 state can cause the audible content and graphical content of the music video to be rendered. Similarly, if the user speaks a vocal request to the automated assistant 120 (e.g., "OK, Assistant, what is...?"), the current state can transition to the first conversation 187 state and render content responsive to the vocal request. In some embodiments, if the co-existing user does not vocally engage with the automated assistant 120 or interact with the data items rendered on the display 111 for at least a predetermined time interval (i.e., a timeout), the current state can transition from the home 185 back to the ambient 183 state, or even back to the display off 181 if there is no ambient 183 state. Other events that may trigger a transition from the Home 185 state to the Ambient 183 (or Display Off 181) state include, but are not limited to, a specific request from the user (e.g., clicking an Exit button on the display), a back gesture (e.g., waving a hand in front of a camera or other sensor) that may signal the co-present user's intent to transition back to Ambient 183, etc.

[0079] In the first conversation 187 state, content related to the requested activity or task can be rendered exclusively on the display 111, and optionally rendered exclusively via (multiple) speakers 191. For example, assume that the co-existing user speaks a vocal request to guide the installation of a smart device. In response, the entire display 111 and (multiple) speakers 191 can be dedicated to multiple dialogue turn interactions to guide the user through the smart device installation process. As another example, assume that the co-existing user speaks a vocal request for information about a celebrity. In some embodiments, the response content can be provided audibly through the speaker 109 as a natural language output by the automated assistant 120, and / or rendered on the display 111. In some embodiments, other content in response to the user's request (but not necessarily specifically requested by the user) can be displayed, and the automated assistant 120 provides the response content auditorily. For example, if the user asks for a celebrity's birthday, the celebrity's birthday can be output auditorily, and other information about the celebrity (e.g., a deep link to show the time of a movie starring the celebrity, (multiple) photos of the celebrity, etc.) can be rendered on the display 111.

[0080] In response to the termination of the first session 187, the current state may transition from the first session 187 state back to the home 185 state (or even the environment 183 or display off 181). The termination of the first session 187 may occur in response to completion of rendering of persistent content of the first session state 187 (e.g., completion of playing of a music video), completion of rendering of transient content, and optionally after a timeout. Other events may be considered termination events, but do not constitute completion of the first session 187, such as an explicit input by the user to cancel the first session 187 (e.g., a "stop" spoken word, a "back" touch input, an "X" or other undo touch input on an interactive graphical element), or a specific request to return to the home 185 state, etc.

[0081] In some embodiments, when there is a pause that causes a transition to the home 185 state, the activities and / or tasks of the first session 187 may not be completed and / or may remain open (e.g., not explicitly canceled). For example, a user may pause a song or video in the middle of its rendering. As another example, a user may start a request task that requires multiple slots to be filled with activity parameters, but may not be able to fill all the required slots. For example, a user may start ordering a pizza, but may stop and leave the room to ask others what toppings they want or request payment information from others. In some of those embodiments, a new tile or card may be rendered on the display 111 in the home 185 state representing the unfinished first session 187. In some cases, the user may click on the new tile or card to continue the first session 187. In many embodiments, the new tile or card may be generated only for incomplete tasks that are classified as persistent tasks (and not for incomplete tasks that are instead classified as transient tasks).

[0082] Likewise Figure 2 , in response to receiving the interruption data, the current state may transition from the first session 187 state to the second session 189 state. This may occur while the first session 187 is still active and content of the first session 187 is rendered exclusively via the display 111 and / or the speaker(s) 109. As described herein, the interruption data may be a spoken utterance provided by a user when it requests that alternative content be provided in place of content provided during the first session 187. The interruption data may additionally or alternatively be an affirmative user interaction in which a notification was provided during the first session 187. Regardless of the interruption data, receipt of the interruption data causes a transition to the second session 189 state in which alternative content corresponding to the interruption data replaces content rendered during the first session 187 state.

[0083] After the suspension of the second session 189 occurs, there is a transition to: the first session 187, where the rendering of the content of the first session 187 is automatically resumed; or a home 185 state or an environment 183 state, where the rendering of the content of the first session 187 state is not automatically resumed. As described herein, whether a transition to the first session 187 state or a transition to the home 185 or environment 183 state should occur can be dynamically determined based on one or more characteristics of the first session 187 and / or the second session 189. Such characteristics can include, for example, whether the first session 187 renders persistent content or transient content, and / or whether the second session renders persistent content or transient content. In the event that a transition to the home 185 state occurs and the first session 187 state renders persistent content, a selectable tile, card, or other graphical element can be optionally rendered in the home 185 state, where selection causes a transition back to the first session 187 state, where the first session 187 is then resumed.

[0084] Suspension of the second session 189 state may occur in response to completion of rendering of the second session 189's persistent content (e.g., completion of playback of a video), completion of rendering of transient content, and optionally after a timeout. Other events may be considered suspension events for the second session 189, but do not constitute completion of the second session 189, such as explicit input by a user to cancel the second session 189 (e.g., "stop" voiced words, "back" touch input, "X" or other undo touch input on an interactive graphical element). In many cases, suspension of the second session 189 occurs without direct user input indicating whether the user wishes to return to the first session 187. Therefore, the embodiments described herein address this situation and can selectively, dynamically, and in a context-sensitive manner determine whether to return to and automatically resume the first session 187 after the second session 189 is suspended.

[0085] Figure 3 300 is a flowchart illustrating an example method 300 according to embodiments disclosed herein. For convenience, the operations of the flowchart are described with reference to a system performing the operations. The system may include various components of various computer systems, such as one or more components of a computing system implementing automated assistant 120. Furthermore, while the operations of method 300 are shown in a particular order, this is not meant to be limiting. One or more operations may be reordered, omitted, or added.

[0086] At box 302, the system identifies content in response to a user interface input at the assistant device. For example, the user interface input can be a spoken utterance, and the system can identify content in response to the spoken utterance. For example, in response to "play video X," a video corresponding to "video X" can be identified. As another example, the user interface input can be a touch interaction with a graphical element rendered on a touch-sensitive display of the assistant device, and the responsive content can be in response to the graphical element. For example, the graphical element can be a recommendation for "video X," and "video X" is identified in response to the interaction with the graphical element.

[0087] At box 304, the system renders content during the first session. The system can render the content via a user interface output device of the assistant device (such as a display and / or speaker(s) of the assistant device). In various embodiments, the system renders the content exclusively. That is, the system renders the content and does not render any content not related to the first session (lacking content related to incoming notifications, which can be rendered briefly (optionally, while pausing the content of the first session) to enable the user to affirmatively accept the notification). Rendering the content 304 during the first session can occur over one or more conversation turns and / or over a long duration.

[0088] During rendering of content during the first session, at box 306, the system monitors for a suspension of the first session. If there is a suspension of the first session at box 306, the system can proceed to box 316 and transition the display of the assistant device to a home screen or an ambient screen, as described herein. In the event that the display is the home screen and the suspension of the first session detected at box 306 is not complete and not overturned, box 316 can include an iteration of optional box 317, where a resume suggestion for the first session is rendered on the home screen. For example, in the event that a suspension is detected at box 306 in response to a user speaking "pause" and a timeout occurs, an optional graphical element can be rendered on the home screen that, if selected, causes resumption of the first session and continued execution of boxes 304, 306, and 308.

[0089] If there is no suspension of the first session at block 306, the system may proceed to block 308 and determine whether interruption data has been received. Figure 3 304, but it should be noted that in various embodiments, blocks 306 and 308 can be performed in parallel and / or "on demand" in response to receiving corresponding inputs. In addition, blocks 304, 306, and 308 can all be performed in parallel. For example, block 304 can be performed continuously while blocks 306 and 308 are running in the background. If no interruption data is received at block 308, the system returns to block 304 and rendering content 304 continues during the first session (e.g., without any interruption).

[0090] If interruption data is received at an iteration of box 308, the system proceeds to box 310 and renders substitute content in a second session that replaces the first session. The substitute content can be responsive to the interruption data. For example, the substitute content can be responsive to a spoken utterance associated with the interruption data, or can be responsive to a notification associated with the interruption data. In various embodiments, the system renders the substitute content exclusively at box 310. That is, the system renders the substitute content without rendering any content not related to the second session (missing content related to the incoming notification, which can be rendered briefly (optionally, while pausing the content of the first session) to enable the user to affirmatively accept the notification). Rendering the substitute content 310 during the second session can occur over one or more conversation turns and / or over a long duration.

[0091] In some implementations of block 310, rendering the alternate content in the second session may replace the first session in response to determining that at least one desired output modality for the alternate content (e.g., audible and / or visual) conflicts with at least one output modality used to render the first content in the first session. In some of those implementations, if the modalities do not conflict (e.g., one of the alternate and first content is only audible, and the other of the alternate and first content is only visual), block 314 may be skipped because the first session is not stalled because both the first session and the second session may occur simultaneously through different output modalities.

[0092] While rendering the alternate content during the second session, the system monitors for suspension of the second session at block 312. When suspension of the second session is detected at an iteration of block 312, the system proceeds to block 314 and determines whether to resume the first session.

[0093] If the system determines to resume the first session at block 314 , the system proceeds back to block 304 and optionally automatically resumes rendering content during the first session from a state of the first conversation corresponding to the state of the first session having detected the interruption data at block 308 .

[0094] If the system determines at block 314 not to resume the first session, the system proceeds to block 316 and transitions the display of the assistant device to a home screen or an ambient screen, as described herein. Where the display is a home screen, block 316 may include an iteration of optional block 317, where a resume suggestion for the first session and / or a resume suggestion for the second session is rendered on the home screen. For example, where the first session includes persistent content, an optional graphical element may be rendered on the home screen that, if selected, causes the first session to resume and blocks 304, 306, and 308 to continue. Additionally, for example, where the second session includes persistent content and the second session abort of block 312 is not completed or undone, an optional graphical element may be rendered on the home screen that, if selected, causes the second session to resume.

[0095] In various implementations, determining whether to resume the first session at block 314 may be based at least in part on one or more characteristics of the first session and / or one or more characteristics of the second session, such as whether the content of the first session is persistent content or transient content and / or whether the replacement content of the second session is persistent content or transient content. Figure 3 310, 312, and 314 may be performed in parallel in various embodiments. For example, block 310 may be performed continuously while block 312 runs continuously in the background, and the determination of block 314 may be predetermined in an iteration before block 312 occurs.

[0096] Figure 4 An example is illustrated of rendering first content 491 during a first session, rendering alternate content 493 during a second session in response to receiving interruption data during rendering of the first content during the first session, and automatically resuming the first session after suspension of the second session.

[0097] exist Figure 4 , and as represented by the top-most first client device 1061, first content 491 may be rendered on a display of the first client device 1061 in response to a user request for the automated assistant 120 to assist with the wiring of a smart thermostat. The first content 491 represents only a portion of the first content that was rendered to such user request during the first session and was rendered during one of a plurality of conversation turns of the first session. It should be noted that, although not depicted, corresponding audible content of the first session may also be rendered by a speaker of the first client device 1061. Prior to the suspension of the first session, and while the first content 491 is being rendered, the user provides a spoken utterance 492 of "what's today's weather forecast". Such spoken utterance 492 constitutes interruption data because it is a request for alternative content that is unrelated to the first session.

[0098] In response to the spoken utterance, and as represented by the first client device 1061 in the middle, the automated assistant causes the rendering of the first content 491 to be replaced by alternative content 493 in the second session. The alternative content 493 replaces the first content 491 in response to the spoken utterance 492, and is rendered exclusively by the display. It should be noted that in various embodiments, the audible alternative content of the second session can also be rendered by the speaker of the first client device 1061.

[0099] After the suspension of the second session, and as represented by the bottom-most first client device 1061, the first session is automatically resumed. For example, the suspension of the second session can occur in response to a timeout or in response to the user indicating completion via a user interface input (e.g., speaking "Done", sliding "back", etc.). In some implementations, the first session can be automatically resumed based at least in part on the first content 491 of the first session being classified as persistent content and the second content 493 of the second session being classified as transient content.

[0100] Figure 5 An example is illustrated of rendering first content 591 during a first session, rendering alternate content 593 during a second session in response to receiving interruption data during rendering of the first content during the first session, and transitioning to an alternate state 595 upon suspension of the second session in which the first session is suggested to be resumed but is not automatically resumed.

[0101] exist Figure 5 , and as represented by the top-most first client device 1061, the first content 591 may be rendered on the display of the first client device 1061 in response to a user request for the automated assistant 120 to assist with the wiring of the smart thermostat. The first content 591 may be similar to Figure 4 Prior to the termination of the first session, and while the first content 591 is being rendered, the user provides a spoken utterance 592 of "show me a video for making a mint julep". This spoken utterance 592 constitutes interruption data because it is a request for alternative content that is not related to the first session.

[0102] In response to the spoken utterance, and as represented by the first client device 1061 in the middle, the automated assistant causes the rendering of the first content 591 to be replaced by alternative content 593 in the second session. The alternative content 593 replaces the first content 591 in response to the spoken utterance 592 and is rendered exclusively by the display. Figure 5 In the example of , alternative content 593 is a persistent video related to making mint juleps, and the video is automatically played in response to spoken utterance 592 constituting interruption data.

[0103] After the suspension of the second session, and as represented by the bottom-most first client device 1061, the first session is not automatically resumed. Instead, the display of the client device 1061 is transformed into an alternative "home" state 595 in which the first session is suggested to be resumed via a graphical element 5951, but is not automatically resumed. If the graphical element 5951 is selected by touch input, the graphical element can resume the first session. In some embodiments, the first session can be not automatically resumed based at least in part on the alternative content 593 of the second session being classified as persistent content. In some embodiments, the graphical element 5951 can be provided based at least in part on the first session content 591 of the first session being classified as persistent content. It should be noted that under the home state 595, other graphical elements 5952 and 5953 are also provided, which display upcoming events and local weather forecasts customized for the user, respectively.

[0104] Figure 6 Another example is illustrated of rendering first content 691 during a first session, rendering alternate content 693 during a second session in response to receiving interruption data during rendering of the first content during the first session, and automatically resuming the first session after suspension of the second session.

[0105] exist Figure 6 , and as represented by the top-most first client device 1061, the first content 691 may be rendered on the display of the first client device 1061 in response to a user request for the automated assistant 120 to assist with the wiring of the smart thermostat. The first content 691 may be similar to Figure 4 Prior to the suspension of the first session, and while the first content 691 is being rendered, the user provides a spoken utterance 692 of "show me a video for distinguishing ground and neutral". This spoken utterance 692 constitutes interruption data because it is a request for alternative content that is not related to the first session. For example, this spoken utterance does not result in the continuation of the presentation of the first content from the first session, but rather constitutes a request for alternative content that is not included in the first content.

[0106] In response to the spoken utterance, and as represented by the first client device 1061 in the middle, the automated assistant causes the rendering of the first content 691 to be replaced by alternative content 693 in the second session. The alternative content 693 replaces the first content 691 in response to the spoken utterance 692 and is rendered exclusively by the display. Figure 6 In the example of , the alternative content 693 is a persistent video related to distinguishing between ground and neutral wires, and the video is automatically played in response to the spoken words 692 constituting the interruption data.

[0107] After the suspension of the second session, and as represented by the bottom-most first client device 1061, the first session is automatically resumed. Figure 6 In the example of , the first session is automatically resumed even though the alternative content of the second session is classified as persistent content. In some implementations, when the alternative content of the second session is classified as persistent content, the first session is not automatically resumed unless one or more other conditions exist. Such additional conditions exist in Figure 6 In the example of . That is, the additional condition may be to determine that the content of the previous first session embodies one or more entities, such as an "electrical wire" entity and / or a finer "ground wire" entity and / or a "neutral wire" entity, and that a defined relationship exists between the entity of the first session and the alternative content of the second session. For example, the relationship may be that the video of the second session also embodies the "electrical wire" entity and / or a finer "ground wire" entity and / or a "neutral wire" entity. Determining that such an entity is embodied by the first content of the first session and / or the alternative content of the second session may be based on terms, titles and / or other characteristics of the content and the alternative content, and may optionally be based on a reference to a knowledge graph, such as the knowledge graph described above. For example, the natural language processor 122 ( Figure 1 ) can tag such entities based on text associated with the first content and the alternative content.

[0108] Figure 7 An example is illustrated of rendering first content 791 during a first session, rendering alternate content 793 during a second session in response to receiving interruption data during rendering of the first content during the first session, and transitioning to an alternate state upon suspension of the second session in which the first session is not automatically resumed and resumption is not suggested.

[0109] exist Figure 7 , and as represented by the top-most first client device 1061, first content 791 may be rendered on a display of the first client device 1061 in response to a user spoken utterance of "how tall is the empire state building". The first content 791 is transient content, and represents the entirety of the first content rendered during the first session (lacking corresponding audio content that may optionally be rendered simultaneously). Prior to suspension of the first session, and while the first content 791 is being rendered, the user provides spoken utterance 792 of "what's today's weather forecast". This spoken utterance 792 constitutes interruption data because it is a request for alternative content that is unrelated to the first session.

[0110] In response to the spoken utterance, and as represented by the intermediate first client device 1061, the automated assistant causes rendering of the first content 791 to be replaced by alternative content 793 in the second session. The alternative content 793 replaces the first content 791 in response to the spoken utterance 792 and is rendered exclusively by the display.

[0111] After the suspension of the second session, and as represented by the bottom-most first client device 1061, the first session is not automatically resumed. Instead, the display of the client device 1061 is transformed into an alternative "home" state 795, in which the first session is not automatically resumed and is not recommended for resumption via any graphical element. In some embodiments, the first session may not be automatically resumed and is not recommended for resumption based at least in part on the first session content 791 of the first session being classified as transient content. It should be noted that under the home state 795, graphical elements 7951 and 7952 are also provided, which display upcoming events and local weather forecasts customized for the user, respectively.

[0112] Figure 8 Another example is illustrated of rendering first content 891 during a first session, rendering alternate content 893 during a second session in response to receiving interruption data during rendering of the first content during the first session, and automatically resuming the first session after suspension of the second session.

[0113] exist Figure 8 , and as represented by the top-most first client device 1061, first content 891 may be rendered on the display of the first client device 1061 in response to a user request for automated assistant 120 to assist with wiring the smart thermostat. Prior to termination of the first session, and while the first content 891 is being rendered, a notification 897 is temporarily rendered over the first content 891 during the first session. Notification 897 notifies the user of an available exercise score update. Figure 8 As indicated in FIG. 8A , the user “clicks” notification 897 which constitutes interrupt data.

[0114] In response to the interruption data, and as represented by the intermediate first client device 1061, the automated assistant causes the rendering of the first content 891 to be replaced by alternative content 893 in the second session. The alternative content 893 is responsive to the notification 897 and includes additional visual and selectable content related to the notified athletic score update. The alternative content 893 replaces the first content 891 and is rendered exclusively by the display.

[0115] After the suspension of the second session, and as represented by the bottom-most first client device 1061, the first session is automatically resumed. For example, the suspension of the second session can occur in response to a timeout, in response to the user indicating completion via a user interface input (e.g., speaking "Done", sliding "back", etc.). In some implementations, the first session can be automatically resumed based at least in part on the first content 891 of the first session being classified as persistent content and the second content 893 of the second session being classified as transient content.

[0116] Fig. 9 is a block diagram of an example computing device 910 that can optionally be used to perform one or more aspects of the techniques described herein. In some implementations, one or more of the client computing device, the user-controlled resource engine 130, and / or other components can include one or more components of the example computing device 910.

[0117] The computing device 910 typically includes at least one processor 914 that communicates with a number of peripheral devices via a bus subsystem 912. These peripheral devices may include a storage subsystem 924, including, for example, a memory subsystem 925 and a file storage subsystem 926; a user interface output device 920; a user interface input device 922, and a network interface subsystem 916. The input and output devices allow a user to interact with the computing device 910. The network interface subsystem 916 provides an interface to an external network and is coupled to corresponding interface devices in other computing devices.

[0118] User interface input devices 922 may include a keyboard; a pointing device such as a mouse, trackball, touchpad, or graphics tablet; a scanner; a touch screen incorporated into a display; an audio input device such as a voice recognition system; a microphone; and / or other types of input devices. In general, the use of the term "input device" is intended to include all possible types of devices and ways of inputting information into computing device 910 or a communication network.

[0119] User interface output device 920 may include a display subsystem, a printer, a fax machine, or a non-visual display such as an audio output device. The display subsystem may include a cathode ray tube (CRT), a flat panel device such as a liquid crystal display (LCD), a projection device, or other mechanisms for creating visual images. The display subsystem may also provide a non-visual display such as via an audio output device. Generally, the use of the term "output device" is intended to include all possible types of devices and ways to output information from computing device 910 to a user or another machine or computing device.

[0120] The storage subsystem 924 stores programming and data structures that provide the functionality of some or all of the modules described herein. For example, the storage subsystem 924 may include an executable Figure 3 Selected aspects of the methods of the invention and logic for implementing the various components described in the figures and / or herein.

[0121] These software modules are typically executed by the processor 914 alone or in conjunction with other processors. The memory subsystem 925 used in the storage subsystem 924 may include multiple memories, including a main random access memory (RAM) 930 for storing instructions and data during program execution and a read-only memory (ROM) 932 for storing fixed instructions. The file storage subsystem 926 can provide persistent storage for program and data files, and may include a hard drive, a floppy disk drive and associated removable media, a CD-ROM drive, an optical drive, or a removable media cartridge. Modules that implement the functionality of certain embodiments may be stored by the file storage subsystem 926 in the storage subsystem 924, or stored in other machines accessible to the processor 914.

[0122] The bus subsystem 912 provides a mechanism for the various components and subsystems of the computing device 910 to communicate with each other as intended. Although the bus subsystem 912 is shown schematically as a single bus, alternative implementations of the bus subsystem may use multiple busses.

[0123] The computing device 910 can be of various types, including a workstation, a server, a computing cluster, a blade server, a server farm, or any other data processing system or computing device. Due to the ever-changing nature of computers and networks, Fig. 9 The description of computing device 910 depicted in FIG. 1 is intended only as a specific example for purposes of illustrating some embodiments. Many other configurations of computing device 910 may have more Fig. 9 The computing devices depicted may have greater or fewer components.

[0124] Where certain embodiments discussed herein may collect or use personal information about a user (e.g., user data extracted from other electronic communications, information about a user's social network, a user's location, a user's time, a user's biometric information, and a user's activities and demographic information, relationships between users, etc.), the user is provided with one or more opportunities to control whether information is collected, whether personal information is stored, whether personal information is used, and how information about the user is collected, stored, and used. That is, the systems and methods discussed herein collect, store, and / or use a user's personal information only after receiving explicit authorization to do so from the relevant user.

[0125] For example, a user is provided with control over whether a program or feature collects user information about that particular user or other users associated with the program or feature. One or more options are presented to each user for whom personal information is to be collected to allow control over the collection of information related to that user to provide permission or authorization as to whether information is to be collected and which portions of the information are to be collected. For example, one or more such control options may be provided to a user over a communications network. In addition, certain data may be processed in one or more ways before being stored or used so that personally identifiable information is removed. As an example, the identity of a user may be processed so that personally identifiable information cannot be determined. As another example, a user's geographic location may be summarized as a larger area so that the user's specific location cannot be determined.< / topping> < / topping> < / topping>

Claims

1. A method implemented using one or more processors, comprising: receiving spoken utterance data indicative of spoken utterances of a user detected via one or more microphones of a client device; identifying, based on the spoken utterance data, first content responsive to the spoken utterance of the user; in response to receiving the spoken utterance data, causing the client device to render at least a portion of the first content during a first session; receiving interruption data during rendering of the first content by the client device during the first session, the interruption data being received in response to another user interface input of the user detected during rendering of the first content during the first session; In response to receiving the interrupt data: storing the session data of the first session in a local memory of the client device or in a remote memory of a remote server in network communication with the client device, wherein the session data indicates a state of the first session when the interruption data is received; causing the client device to render alternate content during a second session that at least temporarily replaces the first session, wherein the alternate content is different from the first content, and wherein causing the client device to render the alternate content during the second session includes causing the client device to render the alternate content in place of the first content; determining whether to cause the client device to resume the first session; In response to determining to cause the client device to resume the first session: retrieving the stored session data for the first session, and resuming the first session in the state indicated by the session data; and In response to determining not to cause the client device to resume the first session: The stored session data of the first session is caused to be cleared from the local memory of the client device or the remote memory of the remote server in network communication with the client device.

2. The method according to claim 1, wherein: Resuming the first session in the state indicated by the session data includes automatically resuming the first session upon suspension of the second session.

3. The method according to claim 2, wherein: Determining whether to cause the client device to resume the first session is based on one or more characteristics of the first session.

4. The method according to claim 3, wherein: The one or more characteristics of the first session include a classification assigned to the first content.

5. The method according to claim 4, wherein: The classification assigned to the first content indicates whether the first content is transient or persistent.

6. The method according to claim 1, wherein: Determining whether to cause the client device to resume the first session is based on one or more characteristics of the first session.

7. The method according to claim 6, wherein: Determining whether to cause the client device to resume the first session is further based on one or more characteristics of the second session.

8. The method according to claim 1, wherein: Storing the session data for the first session includes storing the session data in the local memory without storing the session data in the remote memory.

9. A method implemented using one or more processors, comprising: receiving spoken utterance data indicative of spoken utterances of a user detected via one or more microphones of a client device; identifying, based on the spoken utterance data, first content responsive to the spoken utterance of the user; in response to receiving the spoken utterance data, causing the client device to render at least a portion of the first content during a first session; receiving interruption data during rendering of the first content by the client device during the first session, the interruption data being received in response to another user interface input of the user detected during rendering of the first content during the first session; in response to receiving the interruption data, causing the client device to render substitute content during a second session that at least temporarily replaces the first session, wherein the substitute content is different from the first content, and wherein causing the client device to render the substitute content during the second session includes causing the client device to render the substitute content in place of the first content; Determining whether, upon termination of the second session: causing the client device to automatically resume the first session, or causing the client device to transition to an alternate state in which the client device does not automatically resume the first session, wherein the determining is based on one or more characteristics of the first session; upon suspension of the second session, selectively causing the client device to automatically resume the first session or transition to the alternate state in which the first session is not automatically resumed, wherein the selectively causing is dependent upon the determination; Wherein, in the alternate state, a resume suggestion output is rendered suggesting that the first session be resumed, and wherein the first session is resumed in response to receiving an affirmative user interface input to the resume suggestion output in the alternate state.

10. The method according to claim 9, wherein: In the alternate state, one or more upcoming events are rendered along with the resumption suggestion output.

11. The method according to claim 9, wherein: In the alternate state, a local weather forecast is rendered along with the recovery advice output.

12. The method according to claim 9, wherein: The rendered restoration suggestion output is a selectable graphical element of a visual rendering.

13. The method according to claim 12, wherein: The affirmative user interface input is a touch input selection of the selectable graphical element.

14. The method according to claim 9, wherein: Determining whether to cause the client device to automatically resume the first session or transition the client device to the alternate state upon suspension of the second session is based on one or more characteristics of the first session.

15. The method according to claim 14, wherein: The one or more characteristics of the first session include a classification assigned to the first content.

16. The method according to claim 15, wherein: The classification assigned to the first content indicates whether the first content is transient or persistent.

17. A client device comprising: one or more microphones; one or more processors; a memory operatively coupled to the one or more processors, wherein the memory stores instructions that, in response to execution of the instructions by the one or more processors, cause the one or more processors to: identifying, based on the spoken utterance data, first content responsive to the spoken utterance of the user; rendering at least a portion of the first content during a first session in response to the spoken utterance data; receiving interruption data during rendering of the first content by the client device during the first session, the interruption data being received in response to another user interface input of the user detected during rendering of the first content during the first session; In response to receiving the interrupt data: storing the session data of the first session in the memory, wherein the session data indicates a state of the first session when the interruption data is received; rendering substitute content during a second session that at least temporarily replaces the first session, wherein the substitute content is different from the first content, and wherein the substitute content is rendered in place of the first content; determining whether to resume the first session; In response to determining to resume the first session: retrieving the stored session data for the first session from the memory, and resuming the first session in the state indicated by the session data; and In response to determining not to resume the first session: The stored session data for the first session is caused to be cleared from the memory.

18. A method implemented using one or more processors, the method comprising: receiving spoken utterance data indicative of a plurality of spoken utterances of the user detected via one or more microphones of a client device during a plurality of conversation turns of a first conversation session between the user and the automated assistant; identifying a plurality of instances of first content based on the spoken utterance data, each instance being responsive to a corresponding one of the plurality of spoken utterances of the user; causing the client device to render the first content during the first conversation session; receiving interruption data during rendering of at least a portion of the first content by the client device during the first conversation session, the interruption data being received in response to another user interface input of the user detected during rendering of the at least a portion of the first content during the first conversation session; In response to receiving the interrupt data: storing the session data of the first conversation session in a local memory of the client device or in a remote memory of a remote server in network communication with the client device, wherein the session data indicates a state of the first dialog session when the interruption data is received; causing the client device to render substitute content during a second conversation session that at least temporarily replaces the first conversation session, wherein the substitute content is different from the first content, and wherein causing the client device to render the substitute content during the second conversation session includes causing the client device to render the substitute content in place of the first content; determining whether to cause the client device to automatically resume the first conversation session; In response to determining to cause the client device to automatically resume the first conversation session: retrieving the stored session data for the first conversation session, and automatically resuming the first conversation session in the state indicated by the session data; and In response to determining not to cause the client device to resume the first conversation session: The client device is caused to transition to an alternate state in which the client device does not automatically resume the first conversation session.

19. The method according to claim 18, wherein: Determining whether to cause the client device to resume the first conversation session is based on one or more characteristics of the first conversation session.

20. The method according to claim 19, wherein: The one or more characteristics of the first conversation session include a classification assigned to the first content.

21. The method according to claim 20, wherein: The classification assigned to the first content indicates whether the first content is transient or persistent.

22. The method according to claim 18, wherein: Determining whether to cause the client device to resume the first conversation session is based on one or more characteristics of the first conversation session and further based on one or more characteristics of the second conversation session.

23. The method according to claim 18, wherein: Storing the session data for the first conversation session includes storing the session data in the local memory without storing the session data in the remote memory.

24. The method according to claim 18, wherein: The alternate state in which the client device does not automatically resume the first dialog session includes display of a home screen or an ambient screen.

25. The method according to claim 24, wherein: The display of the home screen or the ambient screen lacks any reference to the first conversation session.

26. The method according to claim 24, wherein: The display of the home screen or the environments screen includes a selectable graphical interface element selectable to resume the first conversation session.

27. A system comprising: one or more processors; a memory operatively coupled to the one or more processors, wherein the memory stores instructions that, in response to execution of the instructions by the one or more processors, cause the one or more processors to: receiving spoken utterance data indicative of a plurality of spoken utterances of the user detected via one or more microphones of a client device during a plurality of conversation turns of a first conversation session between the user and the automated assistant; identifying a plurality of instances of first content based on the spoken utterance data, each instance being responsive to a corresponding one of the plurality of spoken utterances of the user; causing the client device to render the first content during the first conversation session; receiving interruption data during rendering of at least a portion of the first content by the client device during the first conversation session, the interruption data being received in response to another user interface input of the user detected during rendering of the at least a portion of the first content during the first conversation session; In response to receiving the interrupt data: storing the session data of the first conversation session in a local memory of the client device or in a remote memory of a remote server in network communication with the client device, wherein the session data indicates a state of the first dialog session when the interruption data is received; causing the client device to render substitute content during a second conversation session that at least temporarily replaces the first conversation session, wherein the substitute content is different from the first content, and wherein, when causing the client device to render the substitute content during the second conversation session, one or more of the processors cause the client device to render the substitute content in place of the first content; determining whether to cause the client device to automatically resume the first conversation session; In response to determining to cause the client device to automatically resume the first conversation session: retrieving the stored session data for the first conversation session, and automatically resuming the first conversation session in the state indicated by the session data; and In response to determining not to cause the client device to resume the first conversation session: The client device is caused to transition to an alternate state in which the client device does not automatically resume the first conversation session.

28. The system of claim 27, wherein: When determining whether to cause the client device to resume the first conversation session, one or more of the processors determine whether to cause the client device to resume the first conversation session based on one or more characteristics of the first conversation session.

29. The system of claim 28, wherein: The one or more characteristics of the first conversation session include a classification assigned to the first content.

30. The system of claim 29, wherein: The classification assigned to the first content indicates whether the first content is transient or persistent.

31. The system of claim 27, wherein: When determining whether to cause the client device to resume the first conversation session, one or more of the processors determine whether to cause the client device to resume the first conversation session based on one or more characteristics of the first conversation session and further based on one or more characteristics of the second conversation session.

32. The system of claim 27, wherein: When storing the session data of the first conversation session, one or more of the processors stores the session data in the local memory without storing the session data in the remote memory.

33. The system of claim 27, wherein: The alternate state in which the client device does not automatically resume the first dialog session includes display of a home screen or an ambient screen.

34. The system of claim 33, wherein: The display of the home screen or the ambient screen lacks any reference to the first conversation session.

35. The system of claim 33, wherein: The display of the home screen or the environments screen includes a selectable graphical interface element selectable to resume the first conversation session.

Citation Information

Cited By

  • Accessing corporate resources through an enrolled user device

    US12284169B2

  • Accessing corporate resources through an enrolled user device

    US20230018433A1