Transition between previous conversation contexts by an automated assistant

Abstract

To provide a method of acquiring prior contexts with automated assistants.SOLUTION: The method comprises: receiving one or more instances of free-form natural language input from a user during a human-to-computer dialog session between the user and an automated assistant; generating a first dialog context based thereon; receiving one or more additional instances of free-form natural language input from the user during the same human-to-computer dialog session or a different human-to-computer dialog session between the user and the automated assistant; generating a second dialog context based on the additional instances received; receiving a transition command from the user to return the automated assistant from the second dialog context to the first dialog context; and causing natural language output to be presented at the same computing device or a different computing device operated by the user.SELECTED DRAWING: Figure 4

Description

[Technical Field]

[0001] The present disclosure relates to transitioning between previous interaction contexts by an automated assistant. [Background technology]

[0002] Humans may engage in human-computer interactions using interactive software applications referred to herein as "automated assistants" (also referred to as "chatbots," "interactive personal assistants," "intelligent personal assistants," "personal voice assistants," "conversational agents," etc.). For example, when interacting with an automated assistant, a human, sometimes referred to as a "user," may provide commands, queries, and / or requests using free-form natural language input. The free-form natural language may include spoken utterances and / or typed free-form natural language input converted to text using speech recognition.

[0003] Automated assistants allow users to obtain information, access services, and / or perform various tasks. For example, users can perform searches, get directions, and in some cases, interact with third-party applications. Interaction with these third-party applications, which may be facilitated using application programming interfaces (“APIs”) provided by the automated assistant and / or the third-party applications, allows users to perform various actions, such as hailing a car from a ride-sharing application, ordering goods or services (e.g., pizza), making a reservation, etc.

[0004] Automated assistants, for example, converse with users using speech recognition and natural language processing, also utilizing machine learning and other artificial intelligence techniques, in part to predict user intent. Automated assistants may be adept at conversing with users in a natural and intuitive manner, in part because they understand the dialogue context. To utilize dialogue context, an automated assistant may store recent input from the user, questions asked by the user, and / or responses / questions provided by the automated assistant. For example, a user may ask, "Where is the nearest coffee shop?" to which the automated assistant may respond, "It's two blocks east." The user may then ask, "How late is it open there?" By storing at least some form of dialogue context, the automated assistant can determine that the pronoun "there" refers to the "coffee shop" (i.e., coreference analysis).

[0005] However, with conventional automated assistants, when a user abandons one topic of human-computer interaction to engage the automated assistant in another, different topic, the interaction context associated with the first topic may be lost. If the user wants to resume a previous conversation related to the abandoned first topic, the user may be forced to restart the conversation from the beginning so that the abandoned interaction context can be reconstructed. This loss of interaction context poses technical problems in many common use cases for users, as the user and assistant may need to reconstruct the context from the beginning if the related conversation is to be resumed. This may involve the assistant receiving a particular user input a second time and re-storing such re-received input in computer memory. Furthermore, receiving such re-input may require computer processing power to reprocess the input. In addition, power may be required, for example, from the battery on the device, to power a speaker and / or microphone on the device to enable the assistant to prompt or otherwise converse with the user as the user re-enters relevant information to reconstruct the interaction context. One example of such a common use case involves an interaction between an automated assistant and a third-party application. Although automated assistants can interact with third-party applications, they often cannot replicate the full functionality of some third-party applications and therefore must often transfer the user to the third-party application, for example, using a deep link displayed in the facsimile of the human-computer interaction. When the user returns to the automated assistant from the third-party application, the previous interaction context may be lost, especially if a significant amount of time has passed since the previous human-computer interaction.

[0006] This may not be a problem if the user does little else but goes to the third-party application, accomplishes the task, and then quickly returns to the automated assistant, for example, using a back button provided by the operating system running on the user's mobile device. In such a scenario, the automated assistant may simply retain the recent dialogue context so that the conversation can be seamlessly resumed. However, with users typically multitasking and dealing with myriad distractions, returning to the assistant may take many different paths. In a somewhat similar situation, web browser users often use multiple tabs to keep multiple research threads, trains of thought, and the like. Summary of the Invention [Means for solving the problem]

[0007] Described herein are techniques for preserving multiple semantically distinct dialogue contexts generated during a human-to-computer interaction between a user and an automated assistant. Additionally, described herein are techniques for enabling a user to transition to and / or between multiple dialogue contexts to resume a previous conversation with the automated assistant, for example, by providing a transition command during an interaction with a third-party application associated with a particular dialogue context. The subject matter underlying two different dialogue contexts need not be completely unrelated for the dialogue contexts to be considered "semantically distinct." Rather, two dialogue contexts may be "semantically distinct" if, for example, they have non-matching and / or non-overlapping intents, relate to different predefined topics from a library of predefined topics, etc.

[0008] In various implementations, the dialogue context generated during a human-to-computer interaction may include various information. In some implementations, the dialogue context may include one or more user intents detected based on various signals, such as free-form natural language input provided by a user and / or natural language output provided by an automated assistant. In some implementations, the dialogue context may also include one or more references to known entities (e.g., people, places, things) mentioned (by the user and / or the automated assistant) during the human-to-computer interaction session.

[0009] In some implementations, the dialogue context may further include one or more slot values ​​that can be used to fulfill the user's intent, i.e., by filling one or more "slots" associated with the user's intent. A slot value may be any value that can be used as a parameter for a task to be performed by the automated assistant. Slot values ​​may be obtained from various sources. A user may provide slot values ​​actively and / or in response to a request from the automated assistant. For example, a particular third-party application may require one or more slots to fulfill a request. In some cases, a user may actively provide values ​​for required slots. A third-party application may also cause the automated assistant to request values ​​for any required slots for which the user has not actively provided values. In addition, other slot values ​​may be automatically determined using, for example, location coordinates provided by a computing device operated by the user, user preferences, slot values ​​used under similar circumstances in the past, and slot values ​​extracted from recent statements by the user and / or the automated assistant.

[0010] The dialogue context may be saved in various ways. In some implementations, for example, an entire transcript of a human-computer interaction may be saved so that the dialogue context can be reconstructed or resumed, for example, on-the-fly and / or as needed, by detecting intents, slot values, entities, etc., within the saved transcript. In some implementations, particularly those in which a user interacts with an automated assistant using a graphical user interface (“GUI”), this may provide the added benefit of allowing the user to review the transcript. In some cases, the transcript may be annotated with data such as interruption points, intents, mentioned entities, etc. Additionally or alternatively, in some implementations, only core elements of the dialogue context, such as detected intents, slot values, mentioned entities, etc., may be saved in various formats, such as JavaScript® Object Notation (“JSON”) or other similar formats.

[0011] The dialogue context may be saved in various locations. In some implementations, the dialogue context may be persisted in memory local to the computing device operated by the user to engage with the automated assistant. This may result in various technical benefits, such as protecting the user's privacy regarding previous conversations with the automated assistant and / or allowing the user to revisit previous conversations using the techniques described herein even while offline. Additionally or alternatively, in some implementations, the dialogue context may be persisted remotely from the user's computing device, for example, in memory of one or more computing systems operating collectively, often referred to as a "cloud-based service." In various implementations, the dialogue context may be saved for various amounts of time and / or across various events. For example, in some implementations, the dialogue context may be persisted across multiple logically distinct dialogue sessions. Additionally or alternatively, in some implementations, the dialogue context may be persisted for various amounts of time, such as days, weeks, or other times, which may be automatically selected and / or manually set by the user. In some implementations, the conversation context may be maintained in memory as a stack, with the most recent conversation context being "poppable" from the top of the stack to access older conversation contexts.

[0012] The saved dialogue context can be used by the user to resume a previous conversation with the automated assistant. In this way, the user can transition or “toggle” through previous conversations with the automated assistant in an intuitive manner without losing the previous thread and / or thought process and without having to re-enter information previously provided to the assistant. In some implementations where a touchscreen, capacitive touchpad, or other visual input / output (“I / O”) mechanism is available, the user may be presented with a selectable element, such as a “back button,” that the user can manipulate to transition to the previous dialogue context. This selectable element may be provided, for example, simultaneously with a particular third-party application that is associated in memory with the previous dialogue context. Additionally, or alternatively, in scenarios where the user is verbally engaged in a human-to-computer dialogue with the automated assistant, the user may say, “Hey assistant, let’s go back to<the prior conversation> One or more commands may be uttered, such as "Hey Assistant, let's go back to <previous conversation>."

[0013] In some implementations, the automated assistant may be configured to present the user with an enumerated list of available past dialogue contexts so that the user can select a dialogue context that the user wants to revisit. For example, in implementations where the device operated by the user has display capabilities, the automated assistant may present a visual list of past conversations associated with a past dialogue context that the user can select by, for example, touching a list item or speaking a phrase in association with the list item. In some implementations, the enumerated list of available previous dialogue contexts may be presented in response to a user command provided as free-form natural language input, such as "Hey @Assistant, what were we talking about before?"

[0014] In some implementations, the automated assistant may be configured to generate a summary of the previous dialogue context, e.g., so that the user can be quickly provided with necessary information about the previous conversation. For example, in some implementations, the automated assistant may generate a natural language output based on the dialogue context. The natural language output may convey various aspects of the conversation that led to the generation of the dialogue context, such as identifying one or more intents, slot values, mentioned entities, etc. For example, the automated assistant may generate a natural language output that reminds the user what the user was discussing, as well as any slot values ​​that have already been filled (e.g., pizza toppings).

[0015] As an illustrative example, suppose a user previously engaged with an automated assistant to buy size 7 women's dress shoes. Further, suppose the user is interrupted from shopping by a text message from a friend to make a dinner reservation, and the user engages with the automated assistant to open a third-party application to make the dinner reservation (i.e., change the subject). When the user returns to the automated assistant to continue shoe shopping, the last dialogue context is related to the dinner reservation. Rather than reconstructing the shoe shopping conversation from scratch, the user may use the techniques described herein to resume the shoe shopping conversation where the user left off when interrupted by the friend. Specifically, the user may be able to issue one or more commands to transition to a previous dialogue context generated while the user was shoe shopping, for example, by pressing a back button or by issuing one or more voice commands. This may be achieved by forming an association in computer memory between the previous dialogue context and a particular third-party application associated with that context, such as a third-party shoe shopping application. Upon returning to the shoe shopping application, the user may be able to issue a transition command to return to the assistant in the previous dialogue context, i.e., buying size 7 women's dress shoes. This may include pressing a back button displayed or otherwise provided concurrently with the shoe shopping application. The transition command may cause a stored association between the shoe shopping application and the shoe shopping context to be used to resume the shoe shopping conversation in the Assistant. Similarly, if the user subsequently wants to return to the dinner reservation conversation, the user may return to the third-party dinner reservation application and issue a transition command to return to the Assistant in the dinner reservation context. This may include pressing a back button displayed or otherwise provided concurrently with the dinner reservation application.

[0016] In some implementations, a voice command to resume a previous dialogue context can be matched to the previous dialogue context based on the language spoken by the user. Continuing the example above, the user might say something like, "Hey Assistant, let's go back to shoe shopping." The automated assistant can identify the previous dialogue context that is most similar to "shoe shopping" and resume that conversation with the appropriate intent / slot values ​​entered.

[0017] In some implementations, one or more breakpoints may be inserted into the conversation transcript, e.g., in connection with a change in dialogue context. For example, if a user begins discussing shoe shopping but then changes the subject to weather, two breakpoints may be inserted into the transcript, one for each dialogue context. In some such implementations, when a user transitions between previous dialogue contexts (e.g., by pressing a back button), the portion of the transcript associated with the selected dialogue context may be loaded onto the screen, e.g., by using the breakpoint associated with that dialogue context. In this way, the user can view the entirety of a previous conversation related to a topic the user wants to revisit.

[0018] In various implementations, the method includes the steps of: receiving, at one or more input components of a computing device operated by the user, one or more instances of free-form natural language input from the user during a human-to-computer interaction session between the user and an automated assistant implemented at least in part by one or more of the processors; generating a first interaction context based on the one or more instances of free-form natural language, the first interaction context including one or more intents of the user and one or more slot values ​​associated with the one or more intents; and, after generating the first interaction context, receiving, at one or more of the input components, one or more instances of free-form natural language input from the user during the same human-to-computer interaction session or a different human-to-computer interaction session between the user and the automated assistant. The method may include receiving one or more additional instances of free-form natural language input; generating a second dialogue context based on the one or more additional instances of the free-form natural language input, wherein the second dialogue context includes one or more additional intentions of the user, and wherein the second dialogue context is semantically different from the first dialogue context; receiving a transition command from the user at one or more of the input components to return the automated assistant to the first dialogue context; generating a natural language output based on the first dialogue context that conveys at least one or more of the intentions of the first dialogue context and one or more of the slot values ​​of the first dialogue context; and causing the natural language output to be presented on the same computing device or a different computing device operated by the user.

[0019] In various implementations, the transition command may include free-form natural language input spoken or typed by the user. In various implementations, the transition command may include a user selection of a selectable user interface element. In various implementations, the selectable user interface element may include a back button. In various implementations, the back button may be displayed on a touchscreen of a computing device operated by the user. In various implementations, the computing device operated by the user may be a stand-alone interactive speaker, and the back button is displayed on a capacitive touch surface of the stand-alone interactive speaker.

[0020] In various implementations, the automated assistant may cause a graphical user interface (“GUI”) to be rendered on the user's computing device, the GUI presenting a transcript of messages exchanged between the user and the automated assistant during the human-computer interaction. In various implementations, the method may further include launching a separate software application on the user's computing device in response to one or more of the intents of the first dialogue context, where launching the separate software application includes causing the user to present another GUI associated with the separate software application in place of the GUI presented by the automated assistant, and the transition command is received via a selectable element of the another GUI. In various implementations, the transcript of messages may include a first breakpoint associated with the first dialogue context and a second breakpoint associated with the second dialogue context, and the transition command causes a portion of the transcript associated with the first breakpoint to be rendered within the GUI presented by the automated assistant.

[0021] In various implementations, the method may further include receiving a list dialogue context command from the user at one or more of the input components for the automated assistant to provide a list of saved dialogue contexts, identifying at least first and second dialogue contexts in response to the list dialogue context command, and causing information indicative of the at least first and second dialogue contexts to be presented on the same computing device or a different computing device operated by the user. In various implementations, the transition command may include a selection by the user of an element of information indicative of at least the first and second dialogue contexts corresponding to the first dialogue context.

[0022] In another aspect, the method may include receiving, at one or more input components of a computing device operated by the user, one or more instances of free-form natural language input from the user during a first human-to-computer interaction between the user and an automated assistant implemented at least in part by one or more of the processors; associating one or more stored first parameters of the first human-to-computer interaction between the user and the automated assistant with a first third-party application accessible by the user from the first human-to-computer interaction between the user and the automated assistant; receiving, at one or more input components of the computing device, one or more further instances of free-form natural language input from the user during a second human-to-computer interaction between the user and the automated assistant; following the second human-to-computer interaction between the user and the automated assistant, in response to receiving a transition command from the user to transition from the first third-party application to the automated assistant, retrieve one or more stored first parameters associated with the first third-party application; and using the retrieved first parameters to resume the first human-to-computer interaction between the user and the automated assistant.

[0023] In various implementations, the method may further include storing one or more first parameters of the first human-computer interaction between the user and the automated assistant in a computer memory. In various implementations, retrieving the stored one or more first parameters may include retrieving the one or more first parameters from the computer memory. In various implementations, the computer memory is located in a remote server.

[0024] In various implementations, storing the one or more first parameters may include creating a configuration file for the one or more first parameters and storing the configuration file in computer memory. In various implementations, the one or more first parameters indicate a context of the first human-to-computer interaction. In various implementations, the one or more first parameters may include one or more intentions of a user in the first human-to-computer interaction and / or one or more slot values ​​associated with one or more intentions of the user in the first human-to-computer interaction.

[0025] In various implementations, associating the one or more stored first parameters of the first human-to-computer interaction with the first third-party application may include creating a first token callable by the one or more processors in response to receiving a transition command to transition from the first third-party application to the automated assistant to retrieve the one or more stored first parameters associated with the first third-party application.

[0026] In various implementations, the first token may include a link between the one or more stored first parameters and the first third-party application. In various implementations, the method may further include associating the one or more second parameters of the second human-to-computer interaction between the user and the automated assistant with a second third-party application accessible by the user from the second human-to-computer interaction between the user and the automated assistant. In various implementations, in response to receiving a transition command from the user for transitioning from the second third-party application to the automated assistant, the method may include retrieving the one or more stored second parameters associated with the second third-party application and using the retrieved second parameters to resume the second human-to-computer interaction between the user and the automated assistant.

[0027] In various implementations, associating the one or more stored second parameters of the second human-computer interaction with the second third-party application may include creating, by the one or more processors, a callable second token in response to receiving a transition command to transition from the second third-party application to the automated assistant to retrieve the one or more stored second parameters associated with the second third-party application. In various implementations, the second token may include a link between the one or more stored second parameters and the second third-party application.

[0028] Additionally, some implementations include one or more processors of one or more computing devices, the one or more processors operable to execute instructions stored in associated memory, the instructions configured to cause performance of any of the aforementioned methods. Some implementations also include one or more non-transitory computer-readable storage media storing computer instructions executable by the one or more processors to perform any of the aforementioned methods.

[0029] It should be understood that all combinations of the foregoing concepts and additional concepts described in more detail herein are considered to be part of the subject matter disclosed herein, for example, all combinations of claimed subject matter appearing at the end of this disclosure are considered to be part of the subject matter disclosed herein. [Brief explanation of the drawings]

[0030] [Figure 1] FIG. 1 is a block diagram of an example environment in which implementations disclosed herein may be implemented. [Figure 2A] FIG. 1 illustrates an exemplary interaction between a user, another user, and an automated assistant using the user's computing device, according to various implementations. [Figure 2B] FIG. 1 illustrates an exemplary interaction between a user, another user, and an automated assistant using the user's computing device, according to various implementations. [Figure 2C] FIG. 1 illustrates an exemplary interaction between a user, another user, and an automated assistant using the user's computing device, according to various implementations. [Figure 2D] FIG. 1 illustrates an exemplary interaction between a user, another user, and an automated assistant using the user's computing device, according to various implementations. [Figure 3A]1A-1C illustrate exemplary interactions between a user and an automated assistant using a standalone interactive speaker, according to various implementations. [Figure 3B] 1A-1C illustrate exemplary interactions between a user and an automated assistant using a standalone interactive speaker, according to various implementations. [Figure 3C] 1A-1C illustrate exemplary interactions between a user and an automated assistant using a standalone interactive speaker, according to various implementations. [Figure 3D] 1A-1C illustrate exemplary interactions between a user and an automated assistant using a standalone interactive speaker, according to various implementations. [Figure 4] 1 is a flowchart illustrating an exemplary method according to implementations disclosed herein. [Figure 5] FIG. 1 illustrates an exemplary architecture of a computing device. [Figure 6] FIG. 1 is a block diagram of an exemplary computing device. DETAILED DESCRIPTION OF THE INVENTION

[0031] Turning now to Figure 1, an exemplary environment in which the techniques disclosed herein may be implemented is shown. The exemplary environment includes multiple client computing devices 1061-1066. N Each client device 106 may run a respective instance of an automated assistant client 118. One or more cloud-based automated assistant components 119, such as a natural language understanding engine 135, communicate with the client devices 1061-1066 via one or more local and / or wide area networks (e.g., the Internet), shown generally at 110. N The present invention may be implemented on one or more computing systems (collectively referred to as a "cloud" computing system) communicatively coupled to the

[0032] In some implementations, multiple client computing devices 1061-106 N (also referred to herein simply as "client devices") may be associated with one another in various ways to facilitate performance of the techniques described herein. For example, in some implementations, multiple client computing devices 1061-106 N may be associated with each other by being members of a collaborative ecosystem of client devices 106 operated by one or more users (e.g., individuals, families, employees of an organization, other predefined groups, etc.).

[0033] In some implementations, an instance of an automated assistant client 118, through interaction with one or more cloud-based automated assistant components 119, can form what appears from a user's perspective to be a logical instance of an automated assistant 120 with which the user may engage in human-computer interactions. Two instances of such an automated assistant 120 are shown in FIG. 1. A first automated assistant 120A, enclosed in dashed lines, provides services to a first user (not shown) operating a first client device 1061 and includes an automated assistant client 1181 and one or more cloud-based automated assistant components 119. A second automated assistant 120B, enclosed in dash-dot lines, provides services to a first user (not shown) operating a first client device 1061. N and provides services to a second user (not shown) who operates an automated assistant client 118. N and one or more cloud-based automated assistant components 119.

[0034] Thus, it should be understood that each user who engages with an automated assistant client 118 running on a client device 106 may actually engage with the user's own logical instance of the automated assistant 120. For brevity and simplicity, the term "automated assistant," as used herein as "serving" a particular user, refers to the combination of an automated assistant client 118 running on a client device 106 operated by the user and one or more cloud-based automated assistant components 119 (which may be shared among multiple automated assistant clients 118). It should also be understood that in some implementations, the automated assistant 120 may respond to requests from any user, regardless of whether the user is actually "served" by that particular instance of the automated assistant 120.

[0035] Client devices 1061-106 N may include, for example, one or more of a desktop computing device, a laptop computing device, a tablet computing device, a mobile phone computing device, a computing device in the user's vehicle (e.g., an in-car communication system, an in-car entertainment system, an in-car navigation system), a stand-alone interactive speaker, a smart appliance such as a smart television, and / or a user wearable device that includes a computing device (e.g., a user's watch with a computing device, a user's glasses with a computing device, a virtual or augmented reality computing device). Additional and / or alternative client computing devices may be provided.

[0036] Client Computing Devices 1061-106 N Each of the message exchange clients 1071 to 107 NThe message exchange clients 1071 to 1077 may operate various different applications, such as a corresponding one of the message exchange clients 1071 to 1077. N may take various forms, including the client computing devices 1061-106 N The client computing devices 1061-1066 may vary across and / or may have multiple forms. N In some implementations, the message exchange clients 1071-107 N One or more of the messaging clients 1071-1077 may be in the form of a short messaging service ("SMS") and / or multimedia messaging service ("MMS") client, an online chat client (e.g., instant messenger, Internet Relay Chat, or "IRC"), a messaging application associated with a social network, a personal assistant messaging service dedicated to conversations with the automated assistant 120, etc. In some implementations, the messaging clients 1071-1077 may be in the form of a short messaging service ("SMS") and / or multimedia messaging service ("MMS") client, an online chat client (e.g., instant messenger, Internet Relay Chat, or "IRC"), a messaging application associated with a social network, a personal assistant messaging service dedicated to conversations with the automated assistant 120, etc. N One or more of may be implemented via a web page or other resource rendered by a web browser (not shown) or other application of the client computing device 106.

[0037] As described in more detail herein, the automated assistant 120 may communicate with one or more client devices 1061-1066. N In some implementations, the automated assistant 120 interacts with one or more users via the user interface input and output devices of the client devices 1061-1066. NThe automated assistant 120 may engage in a human-to-computer interaction session with the user in response to user interface input provided by the user via one or more user interface input devices of one of the messaging clients 1071-1072. In some of these implementations, the user interface input is explicitly directed to the automated assistant 120. For example, the messaging clients 1071-1072 may N One of the messaging clients 1071-1077 may be a personal assistant messaging service dedicated to conversations with the automated assistant 120, and user interface input provided through the personal assistant messaging service may be automatically provided to the automated assistant 120. Also, for example, the user interface input may be automatically provided to the messaging clients 1071-1077 based on a particular user interface input indicating that the automated assistant 120 should be invoked. N For example, the particular user interface input may be one or more typed characters (e.g., (@AutomatedAssistant), user interaction with hardware and / or virtual buttons (e.g., tap, long tap), a verbal command (e.g., "Hey, automated assistant"), and / or other particular user interface input.

[0038] In some implementations, the automated assistant 120 can engage in a human-computer interaction session in response to user interface input, even when the user interface input is not explicitly directed to the automated assistant 120. For example, the automated assistant 120 can examine the content of the user interface input and engage in an interaction session in response to the presence of particular terms in the user interface input and / or based on other cues. In many implementations, a user can utter a command, search, etc., and the automated assistant 120 can utilize speech recognition to convert the utterance into text and respond to the text accordingly, for example, by providing search results, general information, and / or by taking one or more responsive actions (e.g., playing media, launching a game, ordering food, etc.). In some implementations, the automated assistant 120 can additionally or alternatively respond to the utterance without converting the utterance to text. For example, the automated assistant 120 can embed, convert the speech input into entity representations (indicating entities present in the speech input) and / or other “non-text” representations and operate on such non-text representations. Thus, implementations described herein as operating on text converted from speech input may additionally and / or alternatively operate directly on the speech input and / or on other non-textual representations of the speech input.

[0039] Client Computing Devices 1061-106 N , and each of the computing devices operating the cloud-based automated assistant component 119 may include one or more memories for storing data and software applications, one or more processors for accessing data and executing applications, and other components that facilitate communication over a network. NOperations performed by one or more of and / or by automated assistant 120 may be distributed across multiple computer systems. Automated assistant 120 may be implemented, for example, as a computer program running on one or more computers at one or more locations coupled together via a network.

[0040] As described above, in various implementations, the client computing devices 1061-106 N may operate an automated assistant client 118. In various embodiments, each automated assistant client 118 may include a corresponding speech capture / text-to-speech (“TTS”) / speech-to-text (“STT”) module 114. In other implementations, one or more aspects of the speech capture / TTS / STT module 114 may be implemented separately from the automated assistant 118. In various implementations, the speech capture / TTS / STT module 114 may generate a speech recognition output based on the voice query.

[0041] Each speech capture / TTS / STT module 114 may be configured to perform one or more functions: capturing a user's speech, e.g., via a microphone; converting the captured speech to text (and / or other representations or embeddings); and / or converting text to speech. For example, in some implementations, client devices 106 may be relatively constrained in terms of computing resources (e.g., processor cycles, memory, battery, etc.), so the speech capture / TTS / STT module 114 local to each client device 106 may be configured to convert a finite number of different spoken phrases, particularly certain phrases that invoke the automated assistant 120, to text (or other forms, such as lower-dimensional embeddings). Other speech input may be sent to a cloud-based automated assistant component 119, which may include a cloud-based TTS module 116 and / or a cloud-based STT module 117.

[0042] The cloud-based STT module 117 may be configured to leverage the virtually unlimited resources of the cloud to convert voice data captured by the speech capture / TTS / STT module 114 into text (which may then be provided to the natural language processor 122). The cloud-based TTS module 116 may be configured to leverage the virtually unlimited resources of the cloud to convert text data (e.g., natural language responses formulated by the automated assistant 120) into computer-generated speech output. In some implementations, the TTS module 116 may provide the computer-generated speech output to the client device 106, for example, to be output directly using one or more speakers. In other implementations, text data (e.g., natural language responses) generated by the automated assistant 120 may be provided to the speech capture / TTS / STT module 114, which may then convert the text data into computer-generated speech that is output locally.

[0043] The automated assistant 120 (and in particular the cloud-based automated assistant component 119) may include a natural language understanding engine 135, the aforementioned TTS module 116, the aforementioned STT module 117, and other components described in more detail below. In some implementations, one or more of the engines and / or modules of the automated assistant 120 may be omitted, combined, and / or implemented in a component separate from the automated assistant 120. In some implementations, to protect privacy, one or more of the components of the automated assistant 120, such as the natural language understanding engine 135, the speech capture / TTS / STT module 114, etc., may be implemented at least partially on the client device 106 (e.g., excluding the cloud).

[0044] In some implementations, the automated assistant 120 may communicate with the client devices 1061-1066 during a human-computer interaction session with the automated assistant 120. N The automated assistant 120 may generate response content in response to various inputs generated by a user of one of the client devices 1061-1062. The automated assistant 120 may provide the response content (e.g., over one or more networks if separate from the user's client device) for presentation to the user as part of an interaction session. For example, the automated assistant 120 may N The system may generate responsive content in response to free-form natural language input provided via one of the following: As used herein, free-form natural language input is input that is formulated by a user and is not constrained to a group of options presented for selection by the user.

[0045] As used herein, an "interaction session" may include a logically self-contained exchange of one or more messages between a user and automated assistant 120 (and, possibly, other human participants). Automated assistant 120 may distinguish between multiple interaction sessions with a user based on various signals, such as the passage of time between sessions, changes in user context (e.g., location, before / during / after a scheduled meeting, etc.) between sessions, detection of one or more intervening interactions between the user and a client device other than the interaction between the user and the automated assistant (e.g., a user switches applications for a while and later navigates away and returns to a standalone voice-activated product), locking / sleeping a client device between sessions, changes in client devices used to interface with one or more instances of automated assistant 120, changes in input / output ("I / O") modalities employed by the user, etc.

[0046] The natural language processor 122 of the natural language understanding engine 135 is connected to the client devices 1061 to 1066.N and generate annotated output (e.g., in text format) for use by one or more other components of automated assistant 120. For example, natural language processor 122 may process natural language free-form input generated by a user via one or more user interface input devices of client device 1061. The generated annotated output includes one or more annotations of the natural language input and, optionally, one or more (e.g., all) of the terms of the natural language input.

[0047] In some implementations, the natural language processor 122 is configured to identify and annotate various types of grammatical information within the natural language input. For example, the natural language processor 122 may include a morphological engine that may split individual words into morphemes and / or annotate the morphemes, for example, with their class. The natural language processor 122 may also include a part-of-speech tagger configured to annotate terms with their grammatical roles. For example, the part-of-speech tagger may tag each term with its part of speech, such as “noun,” “verb,” “adjective,” “pronoun,” etc. Also, for example, in some implementations, the natural language processor 122 may additionally and / or alternatively include a dependency parser (not shown) configured to determine syntactic relationships between terms within the natural language input. For example, the dependency parser may determine which terms modify other terms, the subject and verb of a sentence, etc. (e.g., a parse tree), and annotate such dependencies.

[0048] In some implementations, the natural language processor 122 may additionally and / or alternatively include an entity tagger (not shown) configured to annotate entity references in one or more segments, such as references to people (e.g., including literary characters, celebrities, notable figures, etc.), organizations, places (real and fictional), etc. In some implementations, data about the entities may be stored in one or more databases, such as a knowledge graph (not shown). In some implementations, the knowledge graph may include nodes that represent known entities (and, in some cases, entity attributes) and edges that connect the nodes and represent relationships between the entities. For example, a “banana” node may be connected (e.g., as a child) to a “fruit” node, which may be connected (e.g., as a child) to “produce” and / or “food” nodes. As another example, a restaurant called “Hypothetical Cafe” may be represented by a node that also includes attributes such as its address, the types of food served, business hours, contact information, etc. In some implementations, the "Hypothetical Cafe" node may be connected by edges (e.g., representing a child-parent relationship) to one or more other nodes, such as a "Restaurant" node, a "Business" node, a node representing the city and / or state in which the restaurant is located, etc.

[0049] The entity tagger of the natural language processor 122 may annotate references to entities at a high level of granularity (e.g., allowing for identification of all references to an entity class such as people) and / or at a low level of granularity (e.g., allowing for identification of all references to a particular entity such as a particular person). The entity tagger may rely on the content of the natural language input to resolve particular entities and / or may optionally communicate with a knowledge graph or other entity database to resolve particular entities.

[0050] In some implementations, the natural language processor 122 may additionally and / or alternatively include a coreference resolver (not shown) configured to group or "cluster" references to the same entity based on one or more contextual cues. For example, a coreference resolver may be utilized to resolve the term "there" to "Hypothetical Cafe" in the natural language input "I liked Hypothetical Cafe last time we ate there."

[0051] In some implementations, one or more components of the natural language processor 122 may rely on annotations from one or more other components of the natural language processor 122. For example, in some implementations, a named entity tagger may rely on annotations from a coreference resolver and / or a dependency parser when annotating all references to a particular entity. Also, for example, in some implementations, a coreference resolver may rely on annotations from a dependency parser when clustering references to the same entity. In some implementations, when processing a particular natural language input, one or more components of the natural language processor 122 may use related previous input and / or other relevant data other than the particular natural language input to determine one or more annotations.

[0052] The natural language understanding engine 135 may also include an intent matcher 136 configured to determine the intent of a user engaged in a human-computer interaction session with the automated assistant 120 based on the annotated output of the natural language processor 122. Although shown separately from the natural language processor 122 in FIG. 1 , in other implementations, the intent matcher 136 may be an integral part of the natural language processor 122 (or, more generally, of a pipeline that includes the natural language processor 122). In some implementations, the natural language processor 122 and the intent matcher 136 may collectively form the aforementioned “natural language understanding” engine 135.

[0053] The intent matcher 136 may use various techniques to determine the user's intent. In some implementations, the intent matcher 136 may access one or more databases 137 that include, for example, multiple mappings between grammars and response actions (or, more generally, intents). Additionally or alternatively, in some implementations, the one or more databases 137 may store one or more machine learning models trained to generate output indicative of the user's intent based on the user's input.

[0054] A grammar may be selected, formulated (e.g., manually), and / or learned over time to represent, for example, the most common intents of users. <artist>One grammar, "(playing <artist>)" is used on the client device 106 operated by the user. <artist>Another grammar, "[weather|forecast]today", may match user queries such as "what's the weather today" and "what's the forecast for today?". <artist>As can be seen in the example grammar for ", some grammars have slots (e.g., <artist>) Slot values ​​can be determined in a variety of ways. Often, the user actively provides slot values. For example, "Order me a <topping>For the grammar "order me a sausage pizza," the user may say the phrase "order me a sausage pizza," in which case the slot <topping>are automatically filled. Additionally or alternatively, if a user invokes a grammar that includes slots to be filled with slot values ​​without actively providing slot values, the automated assistant 120 may solicit those slot values ​​from the user (e.g., "what type of crust do you want on your pizza?").

[0055] In contrast to many grammars (which may be created manually), machine learning models can be trained automatically, for example, using logs of interactions between a user and an automated assistant. Machine learning models can take various forms, such as neural networks. They can be trained in various ways to predict a user's intent from the user's input. For example, in some implementations, training data including individual training examples can be provided. Each training example can include, for example, free-form input from a user (e.g., in text or non-text format) and can be labeled (e.g., manually) with an intent. The training examples can be applied as inputs to a machine learning model (e.g., a neural network) to generate an output. The output can be compared to the label to determine an error. This error can be used to train the model using techniques such as gradient descent (e.g., stochastic, batch, etc.) and / or backpropagation to adjust weights associated with the model's hidden layer. Once such a model has been trained with several (typically large) training examples, it can be used to generate an output that predicts intent from unlabeled, free-form natural language input.

[0056] In some implementations, automated assistant 120 may facilitate (or “broker”) transactions between a user and a third-party application. These third-party applications may or may not run on a separate computing system from the one running, for example, cloud-based automated assistant component 119. Thus, one type of user intent that may be identified by intent matcher 136 is to engage a third-party application. For example, automated assistant 120 may provide access to an application programming interface (“API”) to a pizza delivery service. A user may invoke automated assistant 120 and provide a command such as “I’d like to order a pizza.” Intent matcher 136 may map this command to a grammar (which may possibly be added to database 137 by the third party) that triggers automated assistant 120 to engage the third-party pizza delivery service. The third-party pizza delivery service may provide automated assistant 120 with a minimum list of slots that need to be filled to fulfill the pizza delivery order. The automated assistant 120 may generate and provide to the user (via the client device 106) natural language output requesting parameters for the slot.

[0057] The fulfillment engine 124 may be configured to receive the intent output by the intent matcher 136, as well as any associated slot values ​​(whether actively provided or requested by the user), and fulfill the intent. In various implementations, fulfilling the user's intent may cause, for example, the fulfillment engine 124 to generate / obtain various fulfillment information. As described below, the fulfillment information may, in some implementations, be provided to a natural language generator ("NLG" in some figures) 126, which may generate natural language output based on the fulfillment information.

[0058] Because intent can be fulfilled in various ways, fulfillment information can take various forms. Suppose a user requests pure information, such as, "Where were the outdoor shots of 'The Shining' filmed?" The user's intent can be determined to be a search query, for example, by intent matcher 136. The intent and content of the search query can be provided to fulfillment engine 124, which can communicate with one or more search engines 150 configured to search a corpus of documents and / or other data sources (e.g., a knowledge graph, etc.) for responsive information, as shown in FIG. 1. Fulfillment engine 124 can provide data indicative of the search query (e.g., the text of the query, a dimensionality-reduced embedding, etc.) to search engine 150. Search engine 150 can provide responsive information, such as GPS coordinates, or other more explicit information, such as "Timberline Lodge, Mt. Hood, Oregon." This response information may form part of the fulfillment information generated by the fulfillment engine 124 .

[0059] Additionally or alternatively, the fulfillment engine 124 may be configured to receive a user intent and any slot values ​​provided by the user or determined using other means (e.g., the user's GPS coordinates, user preferences, etc.), for example, from the natural language understanding engine 135, and trigger a responsive action. The responsive action may include, for example, ordering a product / service, starting a timer, setting a reminder, initiating a call, playing media, sending a message, etc. In some such implementations, the fulfillment information may include a slot value associated with the fulfillment, an acknowledgment response (which may in some cases be selected from predetermined responses), etc.

[0060] As described above, the natural language generator 126 may be configured to generate and / or select natural language output (e.g., spoken words / phrases designed to mimic human speech) based on data obtained from various sources. In some implementations, the natural language generator 126 may be configured to receive as input fulfillment information associated with the fulfillment of an intent by the fulfillment engine 124 and generate the natural language output based on the fulfillment information. Additionally or alternatively, the natural language generator 126 may receive information from other sources, such as a third-party application (e.g., that requested a slot), which may be used to formulate the natural language output for the user.

[0061] The dialogue context engine 138 can be configured to maintain and make available a dialogue context between the user and the automated assistant 120. In some implementations, the dialogue context engine 138 is configured to communicate with the client devices 1061-1066. N The dialogue context engine 138 may maintain (e.g., store) the dialogue context in one or more databases 139 that are remote from the client device 106. Additionally or alternatively, the dialogue context may be stored in memory that is local to one or more client devices 106, for example, to maintain user privacy and / or so that the techniques described herein may be performed even when the client device 106 is offline. In some such implementations, the dialogue context engine 138 may be implemented in whole or in part on the client device 106.

[0062] In various implementations, the dialogue context generated during a human-to-computer interaction and managed by the dialogue context engine 138 may include various information. In some implementations, the dialogue context may include one or more user intents, as detected by, for example, the intent matcher 136, based on various signals, such as free-form natural language input provided by the user and / or natural language output generated by the natural language generator 126. In some implementations, the dialogue context may also include one or more references to known entities (e.g., people, places, things) mentioned (by the user and / or by the automated assistant) during the human-to-computer interaction session and detected by the aforementioned entity tagger.

[0063] In some implementations, the dialogue context may further include one or more slot values ​​that can be used to fulfill the user's intent, i.e., by filling one or more slots associated with the user's intent. The slot values ​​may be obtained from various sources, as previously described. The dialogue context may be saved by the dialogue context engine 138 in various ways. In some implementations, the entire transcript of the human-computer interaction is saved in the database 139 or in memory local to the client device 106, e.g., so that the dialogue context can be reconstructed on the fly and / or as needed, e.g., by detecting intents, slot values, entities, etc., within the saved transcript. In some implementations, particularly those in which the user interacts with the automated assistant 120 using the messaging client 107, this may provide the added advantage that the user may be able to visually review the transcript. Additionally or alternatively, in some implementations, only core elements of the dialogue context, such as detected intents, slot values, mentioned entities, etc., may be saved in various formats, such as JavaScript Object Notation (“JSON”) or other similar formats.

[0064] 1 , such as the dialogue context engine 138, may be configured to perform selected aspects of the present disclosure for preserving multiple semantically distinct dialogue contexts generated during a human-to-computer dialogue between a user and the automated assistant 120. Additionally, techniques are described herein for enabling a user to transition to and / or between multiple dialogue contexts to resume a previous conversation with the automated assistant. For example, in some implementations, a user can transition or “toggle” through a previous conversation with the automated assistant 120 in an intuitive manner without losing the previous thread and / or thought process. In some implementations where a touchscreen, capacitive touchpad, or other visual input / output (“I / O”) mechanism is available on the client device 106, the user may be presented with a selectable element, such as a “back button,” that the user can manipulate to transition to a previous dialogue context. Additionally, or alternatively, in a scenario where a user operates client device 106 to verbally engage in a human-to-computer interaction with an automated assistant, the user may be able to utter one or more commands, such as "Hey Assistant, let's go back to <previous conversation>," to return the automated assistant 120 to a previous interaction context.

[0065] 2A, 2B, 2C, and 2D illustrate exemplary interactions between a user, another user, and an automated assistant 120 implemented at least partially on a computing device 206 operated by the user. These figures illustrate various aspects of the present disclosure, particularly the user's ability to transition between multiple different past dialogue contexts to resume past conversations. In FIGS. 2A-2D, the client device 206 includes a touchscreen 240, various user interface buttons 281, 282, 283, a text entry field 288, and a microphone icon 289. The microphone icon 289 may be operable to allow the user to provide voice input, which may be speech recognized, for example, by the speech capture / TTS / STT module 114, and used, for example, to enter text into the text entry field 288.

[0066] 2A , a user (“You”) operating client device 206 provides the following free-form natural language input (e.g., using input field 288): “SHOP FOR WOMEN'S DRESS SHOES IN SIZE 7.” In some implementations, this statement starts a new topic of conversation, so a breakpoint associated with the topic of shoe shopping may be created and associated with a duplicate, as indicated by label “BP1,” which is included in the figure for illustrative purposes and may or may not actually be rendered on touchscreen 240. The automated assistant asks, “DO YOU WANT TO SHOP FOR A SPECIFIC COLOR SHOE OR SPECIFIC BRAND?” as a way of finding a slot value for the predicted intent of SHOE_SHOPPING. The user provides a slot value by replying, “BLACK COLOR SHOES.”

[0067] At this point, another user named Joe sends the user a message that appears as part of the transcript and reads, "WANT TO GO TO DINNER TONIGHT AT 7:00 AT VINCENZO'S?" At this point, another break point, BP2 in FIG. 2A, may be inserted into the transcript to indicate the start of a new conversational context: dinner tonight. The user responds, "YES," and in response, the automated assistant interrupts, "WANT TO BOOK A RESERVATION FOR 2 FOR DINNER TONIGHT AT 7:00 AT VINCENZO'S?"

[0068] The conversation continues in Figure 2B. The user responds affirmatively ("YES") to the automated assistant's offer to make a reservation. When the intent (to make a dinner reservation with parameters 7 PM and "Vincenzo's") is fulfilled, the automated assistant generates and outputs the text "I HAVE BOOKED A DINNER RESERVATION FOR YOU AND JOE FOR TONIGHT AT 7:00 AT VINCENZO'S," e.g., as natural language output generator 126 in Figure 1. Joe then interrupts again with "THANKS. SEE YOU THEN."

[0069] At this point, there are two dialog contexts active during this conversation between the user and the automated assistant: one associated with BP1, which is related to shoe shopping, and the other associated with BP2, which is related to making dinner reservations. Data indicative of these dialog contexts is maintained by the dialog context engine 138, for example, in database 139. The user may then be able to step back through these dialog contexts (and possibly other dialog contexts that occurred before BP1) so that the user can resume their past conversations without starting from the beginning.

[0070] The user may return through these dialogue contexts in a variety of ways. In some implementations, the user may toggle backward through past dialogue contexts by operating a back button, such as graphical element 281, which may or may not be provided by the operating system of client device 206. In FIG. 2B, the user instead uses free-form natural language input, i.e., "HEY ASSISTANT - BACK TO SHOE SHOPPING," to return to the shoe shopping dialogue context. In some implementations, the same breakpoint BP1 may be inserted again in this portion of the transcript, for example, so that different portions of the transcript associated with this breakpoint can later be reconstructed together as relating to a single dialogue context.

[0071] When returning to a previous dialogue context, it may be beneficial to remind the user of what they were discussing with the automated assistant. Thus, in various implementations, after returning to a previous dialogue context, the automated assistant 120 may generate a natural language output that conveys at least one or more of the intents of the transitioned dialogue context and one or more slot values ​​of the transitioned dialogue context. The automated assistant 120 may then cause this natural language output to be presented on the same computing device or on a different computing device operated by the user. For example, in FIG. 2B , the automated assistant 120 summarizes the past dialogue context by saying, "YOU WERE INTERESTED IN BLACK COLOR WOMEN'S DRESS SHOES IN SIZE 7."

[0072] FIG. 2C illustrates an alternative dialogue that may occur after Joe says, "THANKS. SEE YOU THEN." Instead of the user proactively returning to shoe shopping, the user instead asks the automated assistant 120, "HEY ASSISTANT - WHAT WE TALKING ABOUT?" The automated assistant 120 responds by outputting an enumerated list of past dialogue contexts: "WE HAVE TALKED ABOUT: THE WEATHER FORECAST FOR TONIGHT; SHOE SHOPPING; AND DINNER RESERVATIONS FOR TONIGHT." As shown in FIG. 2C, each of these past topics may be associated with an interruption point. In some implementations, the interruption points may actually be rendered on the touchscreen 240, for example, so that the user can select one to return to the respective dialogue context.

[0073] 2D, however, the user instead says, "BACK TO SHOE SHOPPING. WHERE DID WE LEAVE OFF?" At this point, the automated assistant 120 provides a natural language output summarizing the past conversation: "YOU WERE INTERESTED IN BLACK COLOR WOMEN'S DRESS SHOES IN SIZE 7. DID YOU WANT TO SHOP FOR A SPECIFIC BRAND?" The user then replies, "ALL BRANDS THAT ARE ON SALE," and the shoe shopping conversation resumes.

[0074] 3A, 3B, 3C, and 3D show an exemplary interaction between a user 301 and an automated assistant 120 as a standalone interaction speaker 306 operated by the user 301, according to various implementations. The interaction begins with the user 301 asking the standalone interaction speaker 306, automated assistant 120, to "SHOP FOR WOMEN'S DRESS SHOES IN SIZE 7." The references to "SHOP" and "SHOES" allow the automated assistant 120 to identify a first interaction context of "SHOE SHOPPING." In addition to identifying the first interaction context, the automated assistant 120 may associate a first breakpoint BP1 with the first interaction context. While the breakpoint is represented in FIG. 3A as "BP1," this is for illustrative purposes only and is unlikely to be actually spoken by the user 301.

[0075] The automated assistant 120 may ask the user 301, "DO YOU WANT TO SHOP FOR A SPECIFIC COLOR SHOE OR SPECIFIC BRAND?" The user 301 may respond by limiting the search to "BLACK COLOR SHOES." At this point, the automated assistant 120 may interrupt the conversation to provide the following statement: "REMINDER: DINNER TONIGHT AT 7:00 AT VINCENZO'S WITH JOE. WANT TO MAKE A RESERVATION?" This may have been a reminder that the user 301 previously set for themselves to be delivered at this specific time, or a reminder triggered by someone else, such as Joe. Regardless, the references to "DINNER" and "RESERVATION" allow the automated assistant 120 to identify a second dialogue context of "DINNER RESERVATION." In addition to identifying the second dialogue context, the automated assistant 120 may associate a second break point BP2 with the second dialogue context.

[0076] Instead of addressing the reminder directly, user 301 changes the subject again by responding, "WHAT IS THE WEATHER FORECAST FOR TONIGHT?" This may be because, for example, the user prefers to sit outside when the weather is nice. The reference to "THE WEATHER FORECAST FOR TONIGHT" allows automated assistant 120 to identify a third dialogue context of "THE WEATHER FORECAST FOR TONIGHT." In addition to identifying the third dialogue context, automated assistant 120 may associate a third breakpoint BP3 with the third dialogue context. Automated assistant 120 may respond to the weather forecast by stating, "WARM AND CLEAR SKIES."

[0077] This favorable weather forecast may influence the user's decision to make a dinner reservation. For example, rather than choosing to eat indoors, user 301 may respond with, "MAKE THE DINNER RESERVATION FOR TONIGHT AT VINCENZO'S WITH PATIO SEATING." In response to user 301, automated assistant 120 may interact with a third-party application to make the dinner reservation and alert the user that there is a "DINNER RESERVATION BOOKED FOR TONIGHT AT 7:00 AT VINCENZO'S."

[0078] The conversation continues in Figure 3B. User 301 may request from his / her automated assistant 120, "MESSAGE JOE: I BOOKED A DINNER RESERVATION FOR TONIGHT AT 7:00 AT VINCENZO'S WITH PATIO SEATING." Further, automated assistant 120 may confirm the request by responding, "MESSAGE SENT TO JOE." Assume user 301 has forgotten the previous dialogue context. As noted above, there are various techniques user 301 may use to return to the previous dialogue context.

[0079] 3B, user 301 may return to a previous dialogue context by saying, "HEY ASSISTANT - WHAT WE TALKING ABOUT?" In response, automated assistant 120 may provide an enumerated list of previous dialogue contexts, such as, "WE HAVE TALKED ABOUT: 1) SHOE SHOPPING; 2) DINNER RESERVATION; AND 3) THE WEATHER FORECAST FOR TONIGHT." If user 301 specifies "SHOE SHOPPING," automated assistant 120 may return to the first dialogue context by returning to the dialogue associated with the first interruption point BP1. If user 301 specifies "DINNER RESERVATION," automated assistant 120 may return to the second dialog context by returning to the dialog associated with second breakpoint BP2. If user 301 specifies "THE WEATHER FORECAST FOR TONIGHT," automated assistant 120 may return to the third dialog context by returning to the dialog associated with third breakpoint BP3.

[0080] If user 301 chooses to return to one of the listed previous dialogue contexts, automated assistant 120 may provide a summary of the particular listed dialogue context. For example, user 301 may say, "BACK TO SHOE SHOPPING. WHERE DID WE LEAVE OFF?" Automated assistant 120 may respond, "YOU WERE INTERESTED IN BLACK COLOR WOMEN'S DRESS SHOES IN SIZE 7. DID YOU WANT TO SHOP FOR A SPECIFIC BRAND?" User 301 may respond by saying something like, "ALL BRANDS THAT ARE ON SALE."

[0081] 3C illustrates an alternative dialogue to that shown in FIG. 3B. In some implementations, as shown in FIG. 3C, user 301 can return to the most recent dialogue context by saying, "HEY ASSISTANT - WHAT WE TALKING ABOUT?" In response, automated assistant 120 can provide the most recent dialogue context, "WE WERE TALKING ABOUT THE WEATHER FORECAST FOR TONIGHT." However, user 301 may want to return to an earlier dialogue context and can indicate this by saying, "BEFORE THAT." In response, automated assistant 120 can provide the earlier dialogue context, "WE WERE TALKING ABOUT YOUR DINNER RESERVATION TONIGHT WITH JOE." Additionally, user 301 may want to return to an even earlier dialogue context (e.g., toggle further) and may indicate this by saying "BEFORE THAT." In response, automated assistant 120 may provide the even earlier dialogue context, "WE WERE TALKING ABOUT SHOE SHOPPING." In this way, user 301 may arrive at the desired dialogue context.

[0082] In some implementations, as shown in FIG. 3D , user 301 may return to a previous dialogue context by specifying, "BACK TO SHOE SHOPPING. WHERE DID WE LEAVE OFF?" In response, automated assistant 120 may provide a summary of the previous dialogue context: "YOU WERE INTERESTED IN BLACK COLOR WOMEN'S DRESS SHOES IN SIZE 7. DID YOU WANT TO SHOP FOR A SPECIFIC BRAND?" The user may then provide a spoken output that limits the search to "ALL BRANDS THAT ARE ON SALE." In response, the automated assistant 120 may determine that "BRAND X AND BRAND Y ARE BOTH HAVING A SALE ON BLACK COLOR WOMEN'S DRESS SHOES IN SIZE 7." Interaction based on the previous context may continue, or the user 301 may narrow the search sufficiently and say "THANKS."

[0083] 3A-3D, the user 301 transitioned between dialogue states using voice commands. However, this is not a limitation. Alternatively, the user 301 could interact with a back button 381 on the stand-alone dialogue speaker 306. For example, the stand-alone dialogue speaker 306 may include a basic display, e.g., having one or more LEDs disposed below a capacitive touchpad, that may operate to render a selectable back button. Additionally or alternatively, some stand-alone dialogue speakers may include an attached touchscreen display that may be used, for example, to render a back button or even to render text summarizing past dialogue context, list past dialogue context (and the listed items may be manually selected by the user), etc.

[0084] 4 is a flowchart illustrating an example method 400 according to implementations disclosed herein. For convenience, the operations of the flowchart are described with reference to a system that performs the operations. This system may include various components of various computer systems, such as one or more components of a computing system that implements the automated assistant 120. Furthermore, although the operations of method 400 are shown in a particular order, this is not meant to be limiting. One or more operations may be reordered, omitted, or added.

[0085] At block 452, the system may receive one or more instances of free-form natural language input from a user during a human-to-computer interaction session between the user and automated assistant 120. This free-form input may be typed and / or spoken (e.g., using messaging client 107) and then speech-recognized into text. At block 454, the system may generate a first dialogue context based on the one or more instances of the free-form natural language input. The first dialogue context may include, for example, one or more intents of the user and one or more slot values ​​associated with the one or more intents.

[0086] In block 456, the system may receive one or more additional instances of free-form natural language input from the user during the same human-computer interaction session or a different human-computer interaction session between the user and automated assistant 120. These additional instances of free-form natural language input may relate to a different subject matter than that of block 452. In block 458, the system may generate a second dialogue context based on the one or more additional instances of free-form natural language input. The second dialogue context may include one or more additional intents of the user and may be considered semantically different from the first dialogue context generated in block 454.

[0087] At block 460, the system may receive a transition command from the user to return the automated assistant 120 from the second dialog context to the first dialog context. This command may be spoken or provided using another input modality, for example, using a back button. At block 462, the system may generate a natural language output based on the first dialog context that conveys at least one or more of the intents of the first dialog context and one or more of the slot values ​​of the first dialog context. At block 464, the system may cause the natural language output to be presented on the same computing device or on a different computing device operated by the user. For example, the cloud-based automated assistant component 119 may send text of the natural language output to the speech capture / TTS / STT module 114, which may convert the text to speech and output it through one or more speakers of the client device 106.

[0088] In some implementations, a user may be removed from a conversation with automated assistant 120 to engage with a separate software application, which may in some cases be a third-party application accessible by the user from the human-to-computer interaction with automated assistant 120. This may occur, for example, when automated assistant 120 presents the user with a so-called “deep link” selectable by the user to launch the separate software application. Such a deep link may be presented by automated assistant 120 based on, for example, one or more intents or slot values ​​detected during the human-to-computer interaction. In some such implementations, the separate software application may include a graphical user interface that includes a selectable element, such as a back button, selectable to return to an interface available for interacting with automated assistant 120, such that the interaction context that was active before selection of the deep link becomes active once again. In other implementations, the user may operate a back button not specifically associated with the separate software application, such as a back button rendered by the underlying operating system, to return.

[0089] 5 illustrates an exemplary method 500 for implementing selected aspects of the present disclosure, according to various embodiments. For convenience, the operations of the flowchart are described with reference to a system that performs the operations. This system may include various components of various computer systems. Furthermore, although the operations of method 500 are shown in a particular order, this is not meant to be limiting. One or more operations may be reordered, omitted, or added.

[0090] In block 552, the system may receive, at one or more input components of a computing device operated by the user, one or more instances of free-form natural language input from the user during a first human-to-computer interaction between the user and an automated assistant implemented at least in part by one or more processors. This operation may be similar to block 452 of FIG. 4.

[0091] At block 554, the system may associate one or more stored first parameters of the first human-to-computer interaction between the user and the automated assistant with a first third-party application accessible by the user from the first human-to-computer interaction between the user and the automated assistant. These parameters may be stored, for example, at a remote server and / or in computer memory located locally at the client device. In some implementations, storing the one or more first parameters may include creating a configuration file for the one or more first parameters and storing the configuration file in computer memory. In some cases, the configuration file may be created in a format such as JSON and may, in effect, represent a dialogue context. As an illustrative use case, when a deep link to a first third-party application (e.g., a shoe shopping application) is presented as part of the first human-to-computer interaction, a dialogue context may be associated with the first third-party application, which may include one or more intents, slot values, mentioned entities, etc. (e.g., desired shoe size, style, color, etc.).

[0092] In block 556, the system may receive one or more additional instances of free-form natural language input from the user during a second human-to-computer interaction between the user and the automated assistant at one or more input components of the computing device. This operation may be similar to block 456 of FIG. 4.

[0093] In response to receiving a transition command from the user to transition from the first third-party application to the automated assistant following a second human-to-computer interaction between the user and the automated assistant, in block 558, the system may retrieve, for example from computer memory, one or more stored first parameters associated with the first third-party application. In block 560, the system may use the retrieved first parameters to resume the first human-to-computer interaction between the user and the automated assistant. In this manner, the user can resume the conversation with the automated assistant before the user selected the deep link to the third-party application.

[0094] In some implementations, associating the one or more stored first parameters of the first human-computer interaction with the first third-party application may include creating a callable token, for example, in response to receiving a transition command to transition from the first third-party application to the automated assistant, to retrieve the one or more stored first parameters associated with the first third-party application. In some such implementations, the token may take the form of a link between the one or more stored first parameters and the first third-party application.

[0095] 6 is a block diagram of an example computing device 610 that may optionally be utilized to perform one or more aspects of the techniques described herein. In some implementations, one or more of the client computing device, the user-controlled resource engine 130, and / or other components may comprise one or more components of the example computing device 610.

[0096] Computing device 610 typically includes at least one processor 614 that communicates with several peripheral devices via a bus subsystem 612. These peripheral devices may include a storage subsystem 624, including, for example, a memory subsystem 625 and a file storage subsystem 626, a user interface output device 620, a user interface input device 622, and a network interface subsystem 616. The input and output devices enable user interaction with computing device 610. The network interface subsystem 616 provides an interface to external networks and is coupled to corresponding interface devices in other computing devices.

[0097] The user interface input devices 622 may include a keyboard, a pointing device such as a mouse, trackball, touchpad, or graphics tablet, a scanner, a touchscreen integrated into a display, an audio input device such as a voice recognition system, a microphone, and / or other types of input devices. In general, use of the term "input device" is intended to include all possible types of devices and methods for inputting information into the computing device 610 or onto a communications network.

[0098] The user interface output devices 620 may include a display subsystem, a printer, a fax machine, or other non-visual displays, such as an audio output device. The display subsystem may include a cathode ray tube (CRT), a flat panel display such as a liquid crystal display (LCD), a projection device, or some other mechanism for creating a visible image. The display subsystem may also provide a non-visual display, such as via an audio output device. In general, use of the term "output device" is intended to encompass all possible types of devices and methods for outputting information from the computing device 610 to a user or to another machine or computing device.

[0099] Storage subsystem 624 stores programming and data constructs that provide the functionality of some or all of the modules described herein. For example, storage subsystem 624 may include logic for performing selected aspects of the methods of Figures 4 and 5, as well as implementing various components shown in Figure 1.

[0100] These software modules are generally executed by the processor 614 alone or in combination with other processors. The memory 625 used within the storage subsystem 624 may include several memories, including a main random access memory (RAM) 630 for storing instructions and data during program execution and a read-only memory (ROM) 632 in which fixed instructions are stored. The file storage subsystem 626 may provide persistent storage for program and data files and may include a hard disk drive, a floppy disk drive with associated removable media, a CD-ROM drive, an optical drive, or a removable media cartridge. Modules that implement the functionality of a particular implementation may be stored within the storage subsystem 624 or within another machine accessible by the processor 614 via the file storage subsystem 626.

[0101] The bus subsystem 612 provides a mechanism for allowing the various components and subsystems of the computing device 610 to communicate with each other as intended. Although the bus subsystem 612 is shown schematically as a single bus, alternative implementations of the bus subsystem may use multiple buses.

[0102] The computing device 610 can be of various types, including a workstation, a server, a computing cluster, a blade server, a server farm, or any other data processing system or computing device. Due to the ever-changing nature of computers and networks, the description of the computing device 610 shown in Figure 6 is intended only as a specific example for purposes of describing some implementations. Many other configurations of the computing device 610 can have more or fewer components than the computing device shown in Figure 6.

[0103] In situations where particular implementations discussed herein may collect or use personal information about users (e.g., user data extracted from other electronic communications, information about the user's social network, the user's location, the user's time, the user's biometric information, and user activity and demographic information, relationships between users, etc.), users are provided with one or more opportunities to control whether information is collected, whether the personal information is stored, whether the personal information is used, and how information about the user is collected, stored, and used. That is, the systems and methods discussed herein collect, store, and / or use a user's personal information only if they receive explicit authorization to do so from the associated user.

[0104] For example, a user may be provided with control over whether a program or feature collects user information about that particular user or other users associated with the program or feature. Each user from whom personal information is collected may be provided with one or more options to allow them to control information collection associated with that user, to provide permission or approval for whether information is collected or what portions of information are collected. For example, a user may be provided with one or more such control options over a communications network. Additionally, certain data may be processed in one or more ways before being stored or used so that personally identifiable information is removed. As one example, a user's identification information may be processed so that personally identifiable information cannot be determined. As another example, a user's geographic location may be generalized to a larger area so that the user's specific location cannot be determined.

[0105] While several implementations have been described and illustrated herein, various other means and / or structures for performing the functions and / or obtaining the results and / or one or more of the advantages described herein may be available, and each such variation and / or modification is considered to be within the scope of the implementations described herein. More generally, all parameters, dimensions, materials, and configurations described herein are exemplary, meaning that the actual parameters, dimensions, materials, and / or configurations will depend on the particular application in which the teachings are used. Those skilled in the art will recognize or be able to ascertain using no more than routine experimentation, many equivalents to the specific implementations described herein. Accordingly, the foregoing implementations are presented by way of example only, and it should be understood that, within the scope of the appended claims and their equivalents, implementations may be practiced otherwise than as specifically described and claimed. Implementations of the present disclosure are directed to each individual feature, system, article, material, kit, and / or method described herein. In addition, any combination of two or more such features, systems, articles, materials, kits, and / or methods is included within the scope of the present disclosure, if such features, systems, articles, materials, kits, and / or methods are not mutually inconsistent. [Explanation of symbols]

[0106] 106 client devices 1061~106 N Computing Device, Client Device, Client Computing Device 107 Message Exchange Client 1071~107 N Conversation Client 110 Local and / or Wide Area Networks 114 Speech Capture / Text to Speech ("TTS") / Speech to Text ("STT") Module, Speech Capture / TTS / STT 116 Cloud-based TTS Module 117 Cloud-based STT module 118, 1181, 118 N Automated Assistant Client 119 Cloud-Based Automated Assistant Components 120A First Automated Assistant 120B Second Automated Assistant 122 Natural Language Processor 124 Fulfillment Engine 126 Natural Language Generator 130 User-controlled resource engine 135 Natural Language Understanding Engine 136 Intention Matcher 137 databases 138 Dialogue Context Engine 139 databases 150 search engines 206 Computing Devices, Client Devices 240 Touchscreen 281 User Interface Buttons, Graphical Elements 282 User Interface Buttons 283 User Interface Buttons 288 Text entry field, input field 289 Microphone Icon 306 Standalone Interactive Speaker 381 Back button 610 Computing Devices 612 Bus Subsystem 614 processor 616 Network Interface Subsystem 620 User Interface Output Device 622 User Interface Input Devices 624 Memory Subsystem 625 Memory Subsystem, Memory 630 Main Random Access Memory (RAM) 632 Read-Only Memory (ROM)< / topping> < / topping> < / artist> < / artist> < / artist> < / artist>

Claims

1. 1. A method implemented using one or more processors, comprising: receiving, at one or more input components of a computing device operated by a user, one or more instances of free-form natural language input from the user during a first human-to-computer interaction between the user, an automated assistant implemented at least in part by the one or more processors, and a third-party application, wherein the automated assistant mediates a transaction between the user and the third-party application based on the one or more instances of free-form natural language input; storing one or more first parameters of the first human-computer interaction in association with a first interaction context for subsequent use, the one or more first parameters including an intent and one or more slot values; receiving, at one or more of the input components of the computing device, one or more further instances of free-form natural language input from the user during a second human-to-computer interaction between the user and the automated assistant, the second human-to-computer interaction occurring after the first human-to-computer interaction and semantically different from the first human-to-computer interaction; storing one or more second parameters of the second human-computer interaction between at least the user and the automated assistant in association with a second interaction context, wherein the first and second interaction contexts have inconsistent intents; following the second human-to-computer interaction between the user and the automated assistant, retrieving the one or more stored first parameters associated with the third-party application, and in response to a request from the user, using the retrieved first parameters to generate a natural language summary of the first human-to-computer interaction rendered by one or more output components of the computing device, the natural language summary including the previously stored user-selectable intent and one or more slot values ​​of the first human-to-computer interaction. method.

2. The method of claim 1 , wherein the obtaining step is performed in response to receiving a request from the user.

3. The method of claim 2 , wherein the request from the user includes one or more words that match one or more of the first parameters.

4. The method of claim 2 , wherein the request from the user includes one or more words that match one or more topics associated with the first dialog context.

5. The method of claim 1 , wherein the first and second dialogue contexts do not include any overlapping intents.

6. The method of claim 1 , wherein the one or more first parameters and the one or more second parameters are stored in a computer memory local to the computing device operated by the user.

7. The method of claim 1 , wherein the third-party application comprises a ride-sharing application.

8. The method of claim 1 , wherein the third-party application comprises a reservation application.

9. 1. A system comprising one or more processors and a memory storing instructions, the system comprising: one or more processors configured to: receiving, at one or more input components of a computing device operated by a user, one or more instances of free-form natural language input from the user during a first human-to-computer interaction between the user, an automated assistant implemented at least in part by the one or more processors, and a third-party application, wherein the automated assistant mediates a transaction between the user and the third-party application based on the one or more instances of free-form natural language input; storing one or more first parameters of the first human-computer interaction in association with a first interaction context for subsequent use, the one or more first parameters including an intent and one or more slot values; receiving, at one or more of the input components of the computing device, one or more further instances of free-form natural language input from the user during a second human-to-computer interaction between the user and the automated assistant, the second human-to-computer interaction occurring after the first human-to-computer interaction and semantically different from the first human-to-computer interaction; storing one or more second parameters of the second human-computer interaction between at least the user and the automated assistant in association with a second interaction context, wherein the first and second interaction contexts have inconsistent intents; following the second human-to-computer interaction between the user and the automated assistant, retrieve the one or more stored first parameters associated with the third-party application, and, in response to a request from the user, use the retrieved first parameters to generate a natural language summary of the first human-to-computer interaction rendered by one or more output components of the computing device, the natural language summary including the previously stored user-selectable intent and one or more slot values ​​of the first human-to-computer interaction. system.

10. The system of claim 9 , wherein the one or more first parameters are obtained in response to receiving a request from the user.

11. The system of claim 10 , wherein the request from the user includes one or more words that match one or more of the first parameters.

12. The system of claim 10 , wherein the request from the user includes one or more words that match one or more topics associated with the first dialogue context.

13. The system of claim 9 , wherein the first and second dialogue contexts do not include any overlapping intents.

14. 10. The system of claim 9, wherein the one or more first parameters and the one or more second parameters are stored in a computer memory local to the computing device operated by the user.

15. The system of claim 9 , wherein the third-party application comprises a ride-sharing application.

16. The system of claim 9 , wherein the third-party application comprises a reservation application.

17. A computer-readable storage medium storing instructions, the ... receiving, at one or more input components of a computing device operated by a user, one or more instances of free-form natural language input from the user during a first human-to-computer interaction between the user, an automated assistant implemented at least in part by the one or more processors, and a third-party application, wherein the automated assistant mediates a transaction between the user and the third-party application based on the one or more instances of free-form natural language input; storing one or more first parameters of the first human-computer interaction in association with a first interaction context for subsequent use, the one or more first parameters including an intent and one or more slot values; receiving, at one or more of the input components of the computing device, one or more further instances of free-form natural language input from the user during a second human-to-computer interaction between the user and the automated assistant, the second human-to-computer interaction occurring after the first human-to-computer interaction and semantically different from the first human-to-computer interaction; storing one or more second parameters of the second human-computer interaction between at least the user and the automated assistant in association with a second interaction context, wherein the first and second interaction contexts have inconsistent intents; following the second human-to-computer interaction between the user and the automated assistant, retrieve the one or more stored first parameters associated with the third-party application, and, in response to a request from the user, use the retrieved first parameters to generate a natural language summary of the first human-to-computer interaction rendered by one or more output components of the computing device, the natural language summary including the previously stored user-selectable intent and one or more slot values ​​of the first human-to-computer interaction. A computer-readable storage medium.

18. 20. The computer-readable storage medium of claim 17, wherein the one or more first parameters are obtained in response to receiving a request from the user.

19. 20. The computer-readable storage medium of claim 18, wherein the request from the user includes one or more words that match one or more of the first parameters.

20. 20. The computer-readable storage medium of claim 18, wherein the request from the user includes one or more words that match one or more topics associated with the first dialog context.

Images