Context-aware human-machine dialog

Abstract

This disclosure relates to context-aware human-to-computer dialog. Methods, devices, and computer-readable media are described that involve utilizing the context of an ongoing human-to-computer dialog to enhance the ability of an automated assistant to interpret and respond when a user abruptly transitions between different domains (topics). In various implementations, natural language input from a user can be received during an ongoing human-to-computer dialog with an automated assistant. A grammar can be selected to parse the natural language input. The selection can be based on a topic that is stored as part of a context data structure associated with the ongoing human-to-computer dialog. The natural language input can be parsed based on the selected grammar to generate a parse. Based on the parse, a natural language response can be generated and output to the user by using an output device. Any topic raised by the parse or the natural language response can be identified and added to the context data structure.

Description

[0001] Case Analysis

[0002] This application is a divisional application of Chinese invention patent application 201710891157.1, filed on September 27, 2017. Technical Field

[0003] This application relates to context-aware human-computer dialogue. Background Technology

[0004] Users are increasingly capable of interacting with computers using natural language, a situation referred to as "human-computer dialogue" in this paper. For example, many mobile computing devices (such as smartphones, tablets, smartwatches, standalone smart speakers, etc.) include software programs called "automation assistants" (also known as "interactive assistant modules," "mobile assistants," etc.). Automation assistants can be configured to parse and interpret natural language input (e.g., spoken expressions first, then converted to text, or initially received as text) and provide responsive output (e.g., answers to questions, task initiation, etc.). Existing automation assistants often struggle to switch between domains of conversation. For example, if a user and automation assistant have already exchanged conversations about objects in one topic or domain (e.g., playing a game), and then the user suddenly shifts the conversation to another topic in an unrelated domain (e.g., the weather), the automation assistant may not fully respond and / or may require additional dialogue to respond appropriately. Summary of the Invention

[0005] This paper describes techniques for enhancing the ability of automated assistants (or more generally, interactive voice response "IVR" components) to interpret and respond when a user abruptly switches between different domains in a human-computer dialogue, leveraging the context of the ongoing dialogue. Among various implementations, a so-called "context data structure" can be used to maintain various "topics" related to the ongoing dialogue. For example, these topics can be used to select one or more grammars for parsing the user's natural language input.

[0006] Based on the parsing generated by the chosen grammar, and in some cases, further based on a dynamic dialogue tree representing the ongoing human-computer dialogue, an automated assistant can generate and provide responses (e.g., natural language responses, task initiation, etc.). Simultaneously, any new topics added to the ongoing dialogue by the user and / or the automated assistant can be added to the context data structure. Topics that have not been mentioned or referenced within a certain period can be discarded. In this way, if the user provides topics that are not directly related to some previous topics in the ongoing dialogue but are relevant to the natural language input, the automated topic can potentially seamlessly transition back to the previous topic. Therefore, the automated assistant is better able to understand what the user is saying at a given moment and is thus better able to provide the information or services the user needs, even in the absence of clear statements.

[0007] In some implementations, the techniques described herein can be implemented on a system that includes a parser subsystem and a response subsystem. The parser subsystem may be responsible for interpreting natural language input and providing the interpretation (e.g., "parse" and / or topic) to the response subsystem. The response subsystem may be responsible for receiving the interpretation and responding appropriately by generating and outputting a natural language response or by taking some other response action (e.g., launching a specific application).

[0008] In some implementations, the parser subsystem can be configured to select one or more grammars to parse received natural language input based on one or more topics stored in the aforementioned context data structure associated with the ongoing human-computer dialogue. For example, in some implementations, each of multiple grammars (e.g., a potentially very large grammar library) can be stored in association with one or more applicable topics. If one or more of the associated topics of a grammar are currently relevant (and / or sufficiently relevant) to the human-computer dialogue, that grammar can be selected / applied; otherwise, it can be left unapplied. Thus, when a user provides natural language input, only those grammars associated with the currently relevant topics are selected and applied. This technique is occasionally referred to as “context filtering” in this paper. Context filtering offers the technical advantage of avoiding grammars that are meaningless in the context of a conversation, potentially leading to meaningless output. In some cases, this can reduce the number of grammars applied to each natural language input received from a human (as opposed to applying all grammars from a potentially large library to every natural language input). Reducing the number of grammars applied decreases the utilization of computational resources such as processor cycles, memory, and / or network bandwidth. This could be particularly advantageous for human-computer dialogue, as minimizing the latency of automated assistants (i.e., ensuring that their automated assistants respond to user input as quickly as possible) can be crucial for the user. If the human-computer dialogue has just begun (and therefore, there is no topic yet), the parser subsystem can process the received natural language input using conversational techniques. The parser subsystem can then begin filling in the context data structure based on topics proposed by the human or the automated assistant during the human-computer dialogue.

[0009] The interaction between the parser subsystem and the response subsystem is the aforementioned context data structure. The context data structure can take various forms of data structures stored in memory and can be used to maintain topics relevant to the ongoing human-computer dialogue or topics that have recently been relevant to the ongoing dialogue. When a topic is raised by the user or by any of the automated assistants, for example, the topic can be added to the context data structure via the response subsystem. If a topic that already exists in the context data structure is raised, that topic can be "triggered," for example, brought back to the forefront of the conversation.

[0010] Therefore, in various implementations, each topic that persists in the context data structure can be associated with a measure of the topic's relevance to the ongoing human-computer dialogue. For example, in some implementations, the measure of relevance associated with each topic can be determined at least in part based on the count of rounds in the ongoing human-computer dialogue since the topic was last raised. The more rounds since the topic was raised (e.g., added or triggered), the lower the measure of relevance for that topic. Suppose a user starts a human-computer dialogue with a question about the weather (allowing the topic "weather" to be added), but then the dialogue covers various topics unrelated to weather. The more rounds in the dialogue since the topic of weather was raised, the more the relevance score associated with the topic of weather decreases. In some implementations, if the measure of a topic's relevance falls below a threshold, that topic can be completely discarded from the context data structure. Discarding "invalid" topics from the context data structure can provide various technical advantages. The more syntax selected by the parser subsystem (i.e., context filtering as described above) and applied to each natural language input received from the user, the more computational resources are consumed. By discarding invalid topics, the number of grammars applied to each natural language input by the parser subsystem can be reduced, thereby lowering computational resource consumption. Furthermore, by focusing the parser subsystem on grammars related to topics relevant to the current human-computer dialogue, off-topic or other meaningless parsing is less likely (which could lead to similarly meaningless output from an automated assistant). Therefore, the number of human-computer dialogue rounds required to achieve the user's specific goals (and thus, the user-provided natural language input) can be reduced, allowing the user to benefit from the limited physical or contextual capabilities of providing multiple inputs.

[0011] The measure of relevance can also be based on other factors. For example, in some implementations, the measure of relevance associated with each topic can be determined at least in part based on a measure of the relevance (e.g., semantics) between the topic and one or more other topics in the context data structure. If the first topic has not been proposed for a period of time, but a semantically related second topic is proposed later, the relevance measure of the first topic can be increased.

[0012] In some implementations, the context data structure can take the form of an undirected graph, comprising multiple nodes and multiple edges connecting those nodes. Each node in the undirected graph can represent a given topic from one or more topics stored as part of the context data structure. In some implementations, each node can also store a count of the number of rounds of human-computer dialogue in progress since the last time the given topic was raised. In some implementations, each edge connecting two nodes can represent a measure of relevance (e.g., semantics) between the two topics represented by those two nodes. Of course, other data structures are also considered in this paper.

[0013] In some implementations, each grammar can be associated with a topic and that topic's threshold relevance score. If a topic persists in the context data structure but its relevance score does not meet the threshold, a grammar can be omitted. This allows for fine-tuning of when to apply and when not to grammars.

[0014] In various implementations, the response subsystem may receive interpretations (or "parses") from the grammar selected and applied by the parser subsystem. In some implementations, for example, the parser subsystem may also provide topics to the response subsystem as part of and / or alone as parsing / interpretation. In some implementations, these topics may already exist in the context data structure and may have already been triggered by the associated grammar of these topics that have been applied. In various implementations, the response subsystem may employ a dialogue tree to manipulate the ongoing human-computer dialogue between seemingly unrelated topics. Essentially, the response subsystem receives one or more parsings (or interpretations) and topics from the parser subsystem and manipulates the session along the dialogue tree based on the parsings and topics.

[0015] For example, in some implementations, each node in the dialogue tree represents a natural language process. The root node of the dialogue tree can be configured to process any natural language input (either by initiating a process or by disambiguating from a user request) and initiate one or more child nodes corresponding to processes initiated in response to natural language input from the user. Sub-processes themselves can add additional sub-processes to process each aspect of the dialogue within themselves. Because each process can be a modular process configured to implement the techniques described herein, this simplifies code maintenance and reuse.

[0016] Whenever the response subsystem generates a response for the user (e.g., a natural language response, a response action, or a task), it can add any relevant topics to the context data structure. In some implementations, each node in the dialogue tree can be associated with one or more topics (e.g., these topics can be chosen by the developer of the process below the node). As mentioned above, each grammar can also be associated with one or more topics. Therefore, by adding these topics to the context data structure, the response subsystem effectively increases the number of grammars that can be applied by the parser subsystem at any given time. Thus, if a user talks about a topic, changes the process, and then returns to the original topic (or a semantically related topic), the grammar associated with the original topic remains applicable because the associated topics of the grammar still persist in the context data structure. However, as mentioned above, if the human-computer dialogue deviates from a given topic for a sufficiently long time, in some implementations, for example, the topic can be discarded from the context data structure to prevent the parser subsystem from applying more than a number of grammars to each natural language input, which, as mentioned above, can become computationally expensive.

[0017] As an example, suppose a user initiates a dialogue using the phrase "Let's play a game together." The response subsystem can launch a dialogue tree at the root node. The parser subsystem can parse / interpret the user's natural language input and provide the response subsystem with the parser / interpretation and any proposed topics (e.g., games). Based on the received parser and topics(s), the response subsystem can provide dialogue (such as, "Okay, what game do you want to play?") to initiate a game subprocess below the root node. The response subsystem can also add topics (such as "games" and "game selection") (which can be determined by the parser subsystem based on the received input and / or by the response subsystem based on the topics associated with the game subprocess) to the context data structure. A game subprocess can be associated with one or more topics (e.g., by its developer), and each topic can be associated with one or more topics related to the game (such as "games", "game selection"). For example, one grammar that can be associated with such a topic could be the list_available_games grammar. Another grammar that can be associated with such a topic could be the number_of_players grammar. Suppose the user now says, "What games do you have?". Because its topic (games, game selection) is currently in a context data structure with relatively high relevance (since the topic was recently introduced), the parser subsystem can choose the `list_available_games` grammar. The `list_available_games` grammar can parse the user's statements and provide their interpretations (e.g., the command "LIST GAMES") to the response subsystem. Therefore, the automation assistant (e.g., with the help of the response subsystem) can list available games, such as, "I have Sports Trivia and Historical Trivia". The response subsystem can also add topics (such as "mini-games", "sports", and "history") to the context data structure and can add subprocess nodes corresponding to the SportsTrivia and Historical Trivia games to the dialogue tree.

[0018] Now, suppose a user suddenly changes the topic, for example, by saying "What's the weather forecast today?". There might not currently be any topics related to the topic "weather" stored in the context data structure. In some implementations, the parser subsystem can identify (e.g., context filtering from a large store of grammars) one or more grammars associated with each topic (such as "weather") proposed by the utterance. The parser subsystem can provide the response subsystem with one or more parsings obtained from one or more identified grammars and their corresponding topics. The response subsystem can attempt to associate (e.g., match) existing dialogue tree nodes with the parsings / topics returned by the parser subsystem. In some cases, the parsings / topics provided by the parser subsystem can associate (e.g., match) a dialogue tree node if (i) the topic associated with the dialogue tree node matches the topic of the parser / topic, and (ii) the dialogue tree node is capable of processing the parser / topic. If one or both conditions are not met, control can be passed back to the root node of the dialogue tree. This is likely the case in this example, as the topic "weather" might not match any of the aforementioned topics already added to the dialogue tree. Therefore, the response subsystem can load subprocess nodes associated with the topic "weather" and can provide responses such as "30% chance of rain in cloudy weather." The response subsystem can also add the topic "weather" to the context data structure and, in some cases, reduce the measurement of associations with weather-unrelated topics already included in the context data structure (such as "games," "game selection," "knowledge quiz," "sports," and "history").

[0019] Suppose the user now says "Sports Trivia". Although the user recently changed the topic of the dialogue to "weather", slightly reducing the relevance measure, the topics "games", "game selection", "knowledge quiz", "sports", and "history" are still stored in the context data structure. Therefore, the parser subsystem can still select (e.g., context filtering from a large store of grammars) one or more grammars associated with these topics to parse the user's utterance. The parser subsystem can then apply the selected grammars to the input and provide the resulting parsing and associated topics to the response subsystem. The response subsystem can then pass control to the SportsTrivia subprocess node previously added to the dialogue tree.

[0020] The techniques described in this paper offer a variety of additional technical advantages beyond those already mentioned. For example, the techniques described in this paper enable smooth changes to the conversation topic without requiring (e.g., a separate IVR process) significant resources from individual developers to handle such topic changes. Additionally or alternatively, because the grammar is only applicable when it is relevant to a specific topic (and in some cases, must have a measure of relevance that satisfies various thresholds), using the techniques described in this paper also limits the number of generated parsings / interpretations to those that are only meaningful in the current dialogue context. This can save computational resources such as memory, processor cycles, and network bandwidth.

[0021] In some implementations, the techniques described in this paper facilitate the modular design of IVR dialogues to be implemented by automated assistants. This modular design (including the hierarchical structure of the dialogue tree described in this paper) is easier to maintain and also allows for reuse of simpler code. An additional technical advantage of some implementations is that the dialogue tree described in this paper allows all business logic to be easily developed in a single programming language (such as C++). Therefore, development time is reduced compared to solutions involving several programming languages.

[0022] In some implementations, a method is provided that is performed by one or more processors, including the following operations: receiving natural language input from a user as part of an ongoing human-computer dialogue between the user and an automated assistant operated by one or more processors; selecting one or more grammars to parse the natural language input, wherein the selection is based on one or more topics stored in memory as part of a context data structure associated with the ongoing human-computer dialogue; parsing the natural language input based on the selected one or more grammars to generate one or more parsings; generating a natural language response based on one or more parsings; outputting the natural language response to the user using one or more output devices; identifying one or more topics proposed by one or more parsings or the natural language response; and adding the identified one or more topics to the context data structure.

[0023] Alternatively, these techniques and other implementations disclosed herein may include one or more of the following features.

[0024] In various implementations, one or more grammars are selected from a plurality of grammars, and each of the plurality of grammars is stored in association with one or more applicable topics. In various implementations, each topic stored in memory as part of a context data structure can be associated with a measure of the topic's relevance to the ongoing human-computer dialogue. In various implementations, the measure of relevance associated with each topic can be determined at least in part based on the count of the number of rounds of human-computer dialogue in progress since the last topic was raised.

[0025] In various implementations, the measure of association associated with each topic can be determined at least in part based on a measure of the correlation between the topic and one or more other topics in the context data structure.

[0026] In various implementations, the context data structure may include an undirected graph comprising multiple nodes and multiple edges connecting those nodes. In various implementations, each node of the undirected graph may represent a given topic from one or more topics stored as part of the context data structure, and a count of ongoing human-computer dialogue rounds since the last time the given topic was raised. In some implementations, each edge connecting two nodes may represent a measure of relevance between two topics represented by those two nodes.

[0027] In various implementations, the method may further include: generating a dialogue tree with one or more nodes, each node representing one or more interactive voice processes invoked during an ongoing human-computer dialogue. In various implementations, one or more nodes may be associated with one or more topics. In various implementations, the selection may include: selecting one or more grammars from one or more grammars associated with one or more topics.

[0028] Additionally, some implementations include one or more processors of one or more computing devices, wherein the one or more processors are operable to execute instructions stored in associated memory, and wherein the instructions are configured to perform any of the aforementioned methods. Some implementations also include one or more non-transitory computer-readable storage media storing computer instructions executable by one or more processors to perform any of the aforementioned methods.

[0029] It should be apparent to you that all combinations of the foregoing and additional concepts described in more detail herein are considered part of the subject matter disclosed herein. For example, all combinations of the claimed concepts appearing at the end of this disclosure are considered part of the subject matter disclosed herein. Attached Figure Description

[0030] Figure 1 This is a block diagram of an example environment that can implement the methods disclosed in this article.

[0031] Figure 2 Example syntaxes that can be applied during human-computer dialogue, based on various implementation methods, are described.

[0032] Figure 3 , Figure 4 ,and Figure 5 Example scenario data structures based on various implementations are described.

[0033] Figure 6 An example dialogue tree is depicted based on the implementation disclosed in this article.

[0034] Figure 7A , Figure 7B ,and Figure 7C Examples of how the dialogue tree can be dynamically grown during human-computer dialogue are depicted, depending on various implementation methods.

[0035] Figure 8 Example processes for practicing the various techniques described in this article are depicted.

[0036] Figure 9 The illustration shows an example architecture of a computing device. Detailed Implementation

[0037] Now refer to Figure 1 The illustration shows an example environment in which the techniques disclosed herein can be implemented. The example environment includes multiple client computing devices 106. 1-N And Automation Assistant 120. Despite in Figure 1 The automation assistant 120 is illustrated with a client computing device 106. 1-N However, in some implementations, all or more aspects of the automation assistant 120 can be handled by the client device 106. 1-N One or more implementations of the automation assistant 120. For example, client device 1061 may implement one or more instances of the automation assistant 120, and client device 106 N Individual instances of one or more aspects of the automation assistant 120 can also be implemented. In these instances, one or more aspects of the automation assistant 120 are remotely controlled by the client computing device 106. 1-N In an implementation using one or more computing devices, the client computing device 106 1-N Those aspects of the automation assistant 120 can communicate via one or more networks (such as local area networks (LANs) and / or wide area networks (WANs) (e.g., the Internet)).

[0038] For example, client computing device 106 1-N This may include one or more of the following devices: desktop computing devices, laptop computing devices, tablet computing devices, mobile phone computing devices, computing devices in the user's vehicle (e.g., in-vehicle communication systems, in-vehicle entertainment systems, in-vehicle navigation systems), and / or wearable devices of the user that include computing devices (e.g., the user's watch with computing devices, the user's glasses with computing devices, virtual or augmented reality display computing devices). Additional and / or alternative client computing devices may be provided. In some implementations, a given user may communicate with the automation assistant 120 using multiple client computing devices from a coordinated "ecosystem" shared by the computing devices. However, for the sake of brevity, some examples described in this specification will focus on the user operating a single client device 106.

[0039] Client computing device 106 1-N Each of these can operate various different applications, such as message exchange clients 107. 1-N The corresponding one in the list. Message exchange client 107 1-N It can be done in various forms, and these forms can be used across client computing devices 106. 1-N The variations and / or multiple forms can be implemented on the client computing device 106. 1-N It runs on a single device. In some implementations, the message exchange client 107 1-N One or more of these may take the form of: a Short Message Service (“SMS”) and / or Multimedia Messaging Service (“MMS”) client, an online chat client (e.g., instant messaging tools, internet relay chat, or “IRC”, etc.), a messaging application associated with a social network, a personal assistant messaging service dedicated to conversations with the automation assistant 120, etc. In some implementations, the message exchange client 107 1-N One or more of these can be implemented via web pages or other resources presented through a web browser (not depicted) or other applications of client device 106.

[0040] As described in more detail herein, the automation assistant 120 is via one or more client computing devices 106 1-N The user interface input and output devices participate in a human-computer dialogue session with one or more users. In some implementations, in response to a user via a client computing device 106 1-NOne or more user interface input devices provide user interface input, allowing the automation assistant 120 to participate in a conversational session with the user. In some of these implementations, the user interface input is explicitly directed to the automation assistant 120. For example, message exchange client 107 1-N One of these could be a personal assistant messaging service dedicated to conversing with the automation assistant 120, and user interface inputs provided via this personal assistant messaging service could be offered to the automation assistant 120. Alternatively, for example, the user interface inputs could be explicitly directed to the message exchange client 107 based on specific user interface inputs indicating that the automation assistant 120 will be invoked. 1-N One or more automation assistants 120 are included. For example, specific user interface input can be one or more types of characters (e.g., @AutomatedAssistant), user interaction with hardware and / or virtual buttons (e.g., tap, long tap), verbal commands (e.g., “Hello, Automation Assistant”), and / or other specific user interface input. In some implementations, the automation assistant 120 can participate in a conversational session in response to user interface input even when the user interface input is not explicitly directed to it. For example, the automation assistant 120 can examine the content of the user interface input and participate in a conversational session in response to specific items present in the user interface and / or based on other prompts. In many implementations, the automation assistant 120 can participate in interactive voice response (“IVR”), enabling the user to utter commands, search, etc., and the automation assistant can utilize one or more grammars to convert speech into text and can respond accordingly to that text.

[0041] Client computing device 106 1-N Each of the automation assistants 120 may include one or more memories for storing data and software applications, one or more processors for accessing data and executing applications, and other components that facilitate communication over a network. This is achieved by the client computing device 106. 1-N One or more of the operations performed by the automation assistant 120 can be distributed across multiple computer systems. For example, the automation assistant 120 can be implemented as a computer program running on one or more computers in one or more locations coupled to each other via a network.

[0042] The automation assistant 120 may include a parser subsystem 130, a response subsystem 140, a dialogue context engine 150, and an application engine 160. In some implementations, one or more of the engines and / or subsystems of the automation assistant 120 may be omitted, combined, and / or implemented in components separate from the automation assistant 120. The automation assistant 120 may be implemented via an associated client computing device 106. 1-N Engage in human-computer dialogue sessions with one or more users to obtain information (e.g., answer questions, obtain search results, etc.), initiate tasks (phone apps, calendar apps, miscellaneous apps, etc.), participate in conversations, etc.

[0043] As used herein, a “dialogue session” may include a logically self-contained exchange of one or more messages between a user and an automation assistant 120 as part of a human-computer dialogue. The automation assistant 120 may distinguish between multiple dialogue sessions with the user based on various signals such as: the time elapsed between sessions, changes in the user’s context between sessions (e.g., location before / during / after a scheduled meeting), detection of one or more intermediate interactions between the user and the client other than the dialogue between the user and the automation assistant (e.g., the user switching applications over a period of time, the user leaving a standalone voice-activated product and later returning to it), locking / sleeping the client device between sessions, changing the client device used to connect to one or more instance interfaces of the automation assistant 120, etc.

[0044] In some implementations, when the automation assistant 120 provides a prompt requesting user feedback, the automation assistant 120 may preemptively activate one or more components of the client device (via which the prompt is provided), which are configured to receive user interface input processing in response to the prompt. For example, if user interface input is to be provided via the microphone of the client device 1061, the automation assistant 120 may provide one or more commands to: preemptively "turn on" the microphone (eliminating the need to hit an interface element or say a "hot word" to turn on the microphone), preemptively activate the local voice of the text processor of the client device 1061, preemptively establish a communication session between the client device 1061 and the remote voice to the text processor, and / or present a graphical user interface (e.g., an interface including one or more selectable elements that can be selected to provide feedback) on the client device 1061. This allows user interface input to be provided and / or processed more quickly compared to if components are not preemptively activated.

[0045] The parser subsystem 130 of the automation assistant 120 can be accessed via the client computing device 106. 1-NThe system processes user-generated natural language input and can generate an output of annotations, which may include "parsing" or "interpretation," for use by one or more other components of the automation assistant 120 (such as the response subsystem 140), based on one or more grammars selected using context filtering. For example, the parser subsystem 130 can process free-form natural language input generated by the user via one or more user interface input devices of the client device 1061. The generated annotated output includes one or more annotations (e.g., one or more topics) of the natural language input and optionally includes one or more items (e.g., all) of the natural language input.

[0046] In some implementations, the parser subsystem 130 may be configured to recognize and annotate various types of syntactic information in the natural language input. For example, the parser subsystem 130 may include a part-of-speech tagger configured to annotate items using their syntactic roles. For example, the part-of-speech tagger may use its part of speech (such as "noun", "verb", "adjective", "pronoun", etc.) to tag each item. Additionally, for example, in some implementations, additionally and / or alternatively, the parser subsystem 130 may include a dependency parser configured to determine syntactic relationships between these items in the natural language input. For example, the dependency parser may determine which items modify other items in the sentence, the subject, and verbs, etc. (e.g., a dialogue tree)—and may annotate such dependencies.

[0047] In some implementations, additionally and / or alternatively, the parser subsystem 130 may include an entity annotator configured to annotate entity references in one or more fragments, such as references to people (including, for example, literary characters), organizations, locations (real and fictional), topics, etc. The entity annotator may annotate entity references at a high-level granularity (e.g., capable of identifying all references to entity categories (such as people)) and / or a lower-level granularity (e.g., capable of identifying all references to a specific entity (such as a specific person)). The entity annotator may rely on the content of the natural language input to parse specific entities or topics, and / or may optionally communicate with a knowledge graph or other entity database to parse specific entities and / or topics. In some implementations, additionally and / or alternatively, the parser subsystem 130 may include a coreference parser configured to group or “cluster” references to the same entity based on one or more contextual cues.

[0048] In some implementations, one or more components of the parser subsystem 130 may depend on annotations from one or more other components of the parser subsystem 130. For example, in some implementations, a named entity annotator may depend on annotations from referential parsers and / or dependent parsers in all references to a particular entity. Additionally, for example, in some implementations, a coreference parser may depend on annotations from dependent parsers in clusters of references to the same entity. In some implementations, when processing a particular natural language input, one or more components of the parser subsystem 130 may use relevant previous inputs and / or other relevant data (e.g., topics maintained by the dialogue context engine 150) outside the particular natural language input to determine one or more annotations.

[0049] In various implementations, the dialogue context engine 150 can be configured to maintain one or more topics raised during the current human-computer dialogue session between the user of client device 106 and automation assistant 120. In some implementations, the dialogue context engine 150 can maintain a so-called "context data structure 152" in computer memory. The context data structure 152 can take the form of various data structures stored in memory and can be used to maintain topics relevant to the ongoing human-computer dialogue session or those recently relevant to the ongoing human-computer dialogue session. When a topic is raised by either the user or the automation assistant 120, the topic can be added to the context data structure, for example, through the dialogue context engine 150 or through another component of the automation assistant 120. If a topic that already exists in the context data structure 154 is raised, the topic can be "triggered," for example, effectively bringing the topic back to the forefront of the conversation. In other words, the topics existing in the context data structure 152 can represent topics that the user is likely to be interested in during a particular human-computer dialogue session (or, in some implementations, across different human-computer dialogue sessions that occur relatively close in time). By maintaining these topics in the context data structure 152 at least temporarily, the automation assistant 120 is better able to interpret natural language input from the user that is unrelated to the recently discussed topics, and is therefore better able to adapt to users who change the topics of human-computer dialogue.

[0050] Therefore, each topic present in context data structure 152 can be associated with a measure of relevance to the topic in an ongoing human-computer dialogue. For example, in some implementations, the measure of relevance associated with each topic can be determined at least in part based on the count of rounds in the ongoing human-computer dialogue since the topic was last raised. The more rounds counted since the topic was raised (e.g., added or triggered), the lower the measure of relevance for that topic. Suppose a user's question about the weather starts a human-computer dialogue (adding the topic "weather"), but the dialogue then covers a wide range of topics unrelated to weather. The more rounds counted since the weather topic was raised, the more the relevance score associated with the weather topic decreases. In some implementations, if the measure of relevance for a topic decreases below a threshold, that topic can be completely discarded from context data structure 152.

[0051] The measure of relevance can also be based on other factors. For example, in some implementations, the measure of relevance associated with each topic in context data structure 152 can be determined at least in part based on a measure (e.g., semantics) of the relevance between the topic and one or more other topics in context data structure 152. If a first topic is not proposed at some point in time, but a semantically relevant second topic is proposed later, the measure of relevance of the first topic can be increased or enhanced.

[0052] In some implementations, the context data structure 152 may take the form of an undirected graph comprising multiple nodes and multiple edges connecting those nodes. Each node in the undirected graph may represent a given topic from one or more topics stored as part of the context data structure 152. In some implementations, each node may also store a count of the number of rounds of ongoing human-computer dialogue since the last time the given topic was raised. In some implementations, each edge connecting two nodes may represent a measure of relevance (e.g., semantics) between the two topics represented by those two nodes. Of course, other data structures are considered in this paper.

[0053] In various implementations, parser subsystem 130 may include grammar engine 132. Grammar engine 132 may have access to a grammar database 134 containing multiple grammars. In various implementations, each grammar stored in grammar database 134 may be associated with a topic and a threshold relevance score for that topic. If a topic persists in context data structure 152 but its relevance score does not meet the threshold, the grammar may not be selected by parser subsystem 130. This allows for fine-tuning of when and how grammars are applied.

[0054] For example, it is assumed that the following syntax rules are valid:

[0055]

[0056] These grammar rules can be specified for situations where users can use various forms of natural language input to restart the game. The top-level rules only require that the topic "game" is somewhat relevant. For example, the top-level rules can remain valid as long as the topic "game" has a measure of relevance that satisfies some minimum threshold (e.g., >0). This could be true at any point while the user is playing the game, or when the game is launched in the background. Because natural language input that satisfies the top-level grammar ("Please restart the game") is a highly regarded and unambiguous statement, it makes sense to make the top-level grammar widely applicable.

[0057] In contrast, the underlying rules require the topic "game_restart_confirmation" to be highly relevant, for example, through a measure of relevance that meets some relatively strict thresholds. For instance, the topic "game_restart_confirmation" can only be raised when the user makes statements such as, "Please restart the game," "We should probably reset the game," or "I want to restart." Therefore, if the user is only highly focused on restarting the game, the underlying rules allow the user to restart the game by providing simple affirmative responses in various forms (e.g., $PT_Yes could include "Yes," "Okay," "OK," etc.). The user provides this affirmative input in response to a request from the automation assistant 120 (e.g., "Are you sure you want to restart the game?"). If the user says "Yes" without first raising the topic, the automation assistant 120 can provide outputs such as "Sorry, I don't understand" or "What do you agree with?" While not mandatory, generally, a higher topic relevance requirement is associated with the grammar expected to be more relevant to natural language input.

[0058] In some implementations, for example, when various topics are relevant to the current human-computer dialogue, the grammar rules stored in the grammar database 134 can be applied in a variety of situations. For instance, simple grammar rules for interpreting positive responses (e.g., "yes," "okay," "definitely," "affirmative," etc.) can be applied to a wide range of situations. The following grammar rules demonstrate some example scenarios in which such grammar rules can be applied:

[0059] (allow_if(high pizza_order_confirmation))

[0060] (allow_if(high quiz_quit_game_question))

[0061] (allow_if(high quiz_add_player_question))

[0062] (allow_if(high phone_call_retry_question))

[0063] (rule$common_yes($PT_Yes)

[0064] (=(event_name yes)))

[0065] Using this syntax, users can provide simple affirmative responses (e.g., "Yes", "Okay", "Correct", "OK", etc.) whenever any of the topics "pizza_order_confirmation", "quiz_quit_game_question", "quiz_add_player_question", or "phone_call_retry_question" have a highly relevant metric. This syntax requires a high degree of attention to these topics (i.e., relatively high-relevance metrics) for applicability. This is because each of these topics is only raised when the automation assistant 120 asks a question seeking a simple yes or no response. More than one of these topics will not have a sufficiently high relevant metric to be applied in any given context. Even if more than one of these topics is currently persistently present in the context data structure 152, only the most recently raised topic is likely to meet the strict relevance requirements.

[0066] In some implementations, the grammar rules stored in the grammar database 134 can be associated with multiple topics. For example, the following grammar rules may be used:

[0067] (allow_if(medium person)(medium person))

[0068] (rule$were_married(were they ever married)

[0069] (=(event_name were_married)))

[0070] This rule can be applied when multiple distinct individuals have at least a moderate measure of relevance. Therefore, for example, the following human-computer dialogue could be made possible:

[0071] USER: Who is the lead singer of Iron Lady?

[0072] AA: Bruce Dickinson.

[0073] USER: Who played the song Paparazzi?

[0074] AA: Lady Gaga.

[0075] USER: Were they married?

[0076] AA: No, they didn't.

[0077] Two distinct instances of the topic "person" are presented: one is "Bruce Dickinson," and the other is "Lady Gaga." Therefore, when a user asks the question "Are they married?", the automation assistant 120 can apply the above grammatical rules to obtain information indicating that the two people are not actually married.

[0078] In some implementations, syntax rules can be applied to topics that can be proposed by sources other than the user or the automation assistant 120. It is assumed that the following syntax rules are valid:

[0079] (allow_if(point_of_interest))

[0080] (rule$who_built(google who builtit)

[0081] (=(event_name who_built)))

[0082] Regardless of whether the relevance metric is high or low, this syntax rule is applicable as long as any point of interest currently remains in the context data structure 152. Furthermore, the "point_of_interest" topic can be added by the user or by the automation assistant 120, or by a component in the user's computing device (such as a location coordinate sensor, e.g., a Global Positioning System, or "GPS")). Suppose a user stands in front of the Eiffel Tower (adding the topic "point_of_interest" to the context data structure 152) and asks, "Who built the Eiffel Tower?". The automation assistant 120 can apply the above syntax rule to parse the user's natural language input and can return the answer, "The Eiffel Tower was built by Gustave Eiffel."

[0083] The response subsystem 140 can receive parsings and topics from the syntax selected and applied by the parser subsystem 130. In various implementations, the response subsystem 140 may include a dialogue engine 142 that employs a dialogue tree 144 to steer the ongoing human-computer dialogue between seemingly unrelated topics. Essentially, the response subsystem receives one or more parsings (or interpretations) and topics from the parser subsystem and manipulates the session along the dialogue tree based on these parsings and topics.

[0084] For example, in some implementations, each node in the dialogue tree 144 represents a natural language process. The root node of the dialogue tree 144 can be configured to handle any natural language input (either by initiating a process or by disambiguating from a user request) and is configured to initiate one or more child nodes corresponding to the process, in response to natural language input from the user. Sub-processes themselves can be additionally added to handle various aspects of their own internal dialogue. Because each process can be a modular process configured to implement the techniques described herein, this simplifies code maintenance and allows for reuse.

[0085] Whenever the response subsystem 140 generates a response for the user (e.g., a natural language response, a response action, or a task), the response subsystem 140 can add any relevant topics to the context data structure 152. In some implementations, each node of the dialogue tree can be associated with one or more topics (e.g., these topics can be selected by the developer of the process below the node). As mentioned above, each grammar can also be associated with one or more topics. Therefore, by adding these topics to the context data structure, the response subsystem effectively increases the number of grammars that can be applied by the parser subsystem 130 at any given time. Thus, if the user talks about a topic, changes the process, and then returns to the original topic (or a semantically related topic), the grammar associated with that original topic remains applicable because the associated topics of the grammar still exist in the context data structure 152. However, as mentioned above, if the human-computer dialogue deviates from a given topic for a sufficiently long time, in some implementations, for example, the topic can be discarded from the context data structure 152 to prevent the parser subsystem 130 from applying too many grammars to each natural language input, which, as mentioned above, can become computationally expensive.

[0086] In addition to simply conversing with the user using the techniques described herein, or alternatively, the automation assistant 120 can also initiate one or more tasks based on natural language input provided by the user. Therefore, the application engine 160 can maintain the application library 162 or otherwise have access to it. Applications in the application library 162 can be installed on one or more client devices 106, can be processes available to the automation assistant 120 (e.g., web processes, cloud processes, etc.), and / or can be processes built into the automation assistant 120. More generally, applications in the application library 162 can be processes available to the automation assistant 120 during human-computer interaction.

[0087] As will be described in more detail below, the response subsystem 140 can maintain a node graph corresponding to the interactive natural language processes initiated and / or accessed by the automation assistant 120 during the human-computer dialogue session, for example, as a dialogue tree 144. When the automation assistant 120 accesses a new process, the nodes representing these processes can be added as child nodes to the aforementioned root node. In some implementations, additional child node processes can also be added as needed.

[0088] In some implementations, each child node procedure can be compatible with the techniques and frameworks described herein. For example, each child node procedure can be associated with one or more topics. These topics effectively associate each child node with a grammar that is also associated with the same topic. If natural language input is received that does not satisfy the most recently added child node procedure (i.e., the natural language input proposes a topic that does not apply to any topic of the most recently added child node procedure), it is possible that other previously added child node procedures can have associated topics that can be applied to (match) the newly proposed topic. In this way, the user can seamlessly return to the previous topic of the human-computer dialogue session without requiring the automation assistant 120 to trace the state machine associated with the previously proposed child node procedure.

[0089] Figure 2A simplified example is depicted illustrating how one or more grammars 270 are applied during a human-computer dialogue session 272 between a user and an automation assistant 120 (represented as "AA" in the figure). In this example, the top-level grammar 270 may be applicable if the topic "human" has a metric considered "high" relevance (e.g., meeting a threshold). The dialogue begins with a user providing natural language input in the form of the question "What is this music called?" (assuming there is music being played by the automation assistant 120 or detected as ambient sound). As indicated by bold, in some implementations, for example, the parser subsystem 130 and / or the dialogue context engine 150 may analyze the user's natural language input to determine which topic "music" should be added to the topics present in the current context data structure 152. At this point in time, because the topic has just been added, the metric of relevance associated with the topic "music" may be relatively high / strong.

[0090] The automation assistant 120 responds by stating, "This is Chopin's Nocturne in C-sharp minor." For example, the automation assistant 120 may determine this answer because it itself initiated music playback, or by utilizing various sound processing techniques to identify music based on one or more audible features. As indicated in bold, in various implementations, topics (such as "person," "artist," "composer," and / or "Chopin") can be added to the topics currently existing in the context data structure 152. At that point in time, because the topic has just been added, the measure of relevance associated with the topics "person" and / or "Chopin" can be relatively high / strong. In some implementations, for example, because one round has passed since the music topic was presented, the measure of relevance associated with the previously added "music" topic can be reduced / weakened. However, in other implementations, because the theme "Chopin" (composer) has a close semantic relationship with the theme "music", the measure of the relevance of the theme "music" to the human-computer dialogue cannot be reduced or weakened.

[0091] Next, the user provides natural language input, which, among other things, includes the question "Where was he born?". Because the topic "person" was just raised in the previous round (and thus added to the context data structure 152), the topic "person" can still have a strong or high degree of relevance. Therefore, for example, the parser subsystem 130 can apply grammar 270 to determine that the natural language input "Where was he born?" should trigger an event called "where_person_born", which causes the parser subsystem 130 to provide parsing and / or interpretation to the response subsystem 140, which in turn causes the response subsystem 140 to determine the birthplace of the most recently referenced person during the dialogue (Chopin) and return the response "His birthplace is...". "Wola". Meanwhile, the response subsystem 140 can also handle topics (such as "city" and / or "...") Wola) is added to the context data structure 152 or the dialogue context engine 150 adds the topic (such as "city" and / or "Wola") to the context data structure 152 or makes the conversation context engine 150 add the topic (such as "city" and / or "Wola") to the context data structure 152 or make ... Wola”) is added to the context data structure 152. As previously mentioned, for example, the measure of relevance to the topics (e.g., “music”, “people”, “Chopin”) proposed in previous rounds may or may not be reduced based on whether the topics (e.g., “music”, “people”, “Chopin”) are semantically related to the newly added topic.

[0092] Next, the user provides natural language input asking the question "Where?". It should be noted that, without benefiting from any prior dialogue context, the user initiates a new human-computer dialogue session with the automation assistant 120 by asking such a vague question. The automation assistant 120 may no longer be able to provide a response to this question without requiring further deambiguity information from the user. However, because the topic "city" was just added to the context data structure 152 in the previous round, the topic "city" has a relatively high relevance metric, allowing the parser subsystem 130 to apply the second grammar in grammar 270 to the user's natural language input. The parser subsystem 130 can provide the resulting parsing or interpretation (e.g., event_name = "where_city_located") to the response subsystem 140 along with any new applied topic. Based on this parsing (and topic), the response subsystem 140 can (e.g., from various sources such as knowledge graphs, searchable documents, etc.) obtain response information and provide natural language output.

[0093] Wola is a city in the Masovian region of Poland. Similar to the previous example, topics (such as "region", "country", and / or "Poland") can be added to the context data structure 152 by the response subsystem 140 and / or the dialogue context engine 150.

[0094] The user then provides natural language input, which asks the question, “How far is it from the capital?” Furthermore, it is impossible for the automated assistant 120 to answer such a question without prior context and without needing to disambiguate information requested by the user. However, this utilizes a relatively “new” (i.e., still relatively relevant metric) topic in this particular human-computer dialogue session. Given "Wola" and "Poland," the automation assistant 120 can answer the question more directly. For example, the parser subsystem 130 can apply a third grammar down to the user's natural language input to trigger an event "identify_capitol" that returns the capital of the currently relevant country or region (in this example, "Warsaw"). Alternatively, in some implementations, the topic (such as "capital") can be added to the context data structure 152. The parser subsystem 130 can then apply a fourth grammar down to the combination of the user's natural language input and the identified capital. The resulting parsing or interpretation (e.g., event_name = "distance_between_cities") can be provided to the response subsystem 140. Based on this parsing, the response subsystem 140 can obtain response information and provide the natural language input "distance is 39 miles".

[0095] The next two rounds of the human-computer dialogue 272 specifically highlight a technical advantage provided by the technique described herein: the sudden change in the domain of the conversation. The user provides natural language input that asks, “Do you know any similar artists?”. As mentioned above, the topic of “artist” is added to the context data structure 152 relatively early during the human-computer dialogue 272 and is not raised in several dialogue rounds. Therefore, the measure of relevance associated with the topic “artist” can be significantly reduced / weakened. Nevertheless, the underlying grammar of grammar 270 specifies that that grammar can still be applied even if the topic “artist” is associated with a relatively low measure of relevance. Therefore, parser subsystem 130 can apply that grammar to generate a parser for response subsystem 140. In response to this parser, response subsystem 140 can then trigger an event “find_similar_artists” that initiates a search for other similar artists (e.g., those who create aesthetically similar art, are demographically similar, operate within similar time periods, etc.). The response subsystem 140 can then provide natural language output: "Of course, I recommend Franz Liszt, Ludwig van Beethoven, or Franz Schubert."

[0096] Figure 3 An example illustrating how the context data structure 152 can be logically implemented is provided. In this example, the context data structure 152 is formed as an undirected graph 364 having multiple nodes and multiple edges connecting these nodes. Each node of these undirected graphs 364 can represent a given topic among one or more persistent topics that are part of the context data structure 152. In some implementations, each node can also store a count of the number of rounds of human-computer dialogue that have been ongoing since the last time the given topic was raised.

[0097] Undirected graph 364 includes and in Figure 2 During the human-computer dialogue 272, the nodes corresponding to the topics in the context data structure 152 may have already been added. The square brackets before each topic indicate the count of the last round in which that topic was raised; these brackets also... Figure 2 The instructions were given. Therefore, for example, during the first ([0]) round (although this is not required, it includes the user's natural language input and responses from the automation assistant 120), the topics "music" and "artist" were presented. When the automation assistant 120 mentioned "..." When the user asked for the distance to the capital, the topic “city” was presented during the second round ([1]). When the automation assistant 120 mentioned “Poland” and “Masovian region” respectively, the topics “country” and “region” were presented during the third round ([2]). When the user asked for the distance to the capital, the topics “distance” and “city” were presented during the fourth round ([3]). When the automation assistant 120 provided three examples of artists similar to Chopin, the topic “artist” was presented three times during the fifth round ([4]).

[0098] In some implementations, the relevance of a particular topic can be simply determined by subtracting the number of its associated dialogue rounds (if there are recent dialogue rounds representing multiple nodes of the topic) from the current round's count. The smaller the result, the more relevant the topic is to the current dialogue. However, as mentioned above, other implementations consider other factors, such as topic relevance. For example, in some implementations, nodes representing topics raised in the current dialogue round (whether newly added or updated) can be identified, such as node A. The shortest distance d to any node belonging to A can be calculated for each node in the directed graph 364. i This is used to determine the relevance of a given topic during the current dialogue round. Additionally, the difference 'a' between the current dialogue round number and the node's round number can be calculated. i The measure of a node's (and therefore, topic's) relevance can be set as a i and d i The minimum value in.

[0099] Figure 4 A similar undirected graph 464 (e.g.) is depicted Figure 3 (Depicted). The assumption for this example is: Figure 2 The human-computer dialogue 272 depicted in the figure is now entering its sixth ([5]) dialogue round, and a shaded node (“the artist”) has been proposed during the count of the current dialogue round ([5]). Each node in the undirected graph 464 includes, on the left side of the square brackets, the difference d between the current dialogue round number ([5]) and the most recent dialogue round in which the topic was proposed. i For each node in the undirected graph 464, the shortest path a between that node and the most recently added node A is also included on the right side of the square brackets. i . Figure 5 Depicting based on a i and d iThe minimum value of the measure of relevance determined for each node / topic. Determining the measure of relevance in this way can take into account scenarios where a particular topic has not been mentioned recently, but has still been mentioned in close relational contexts. In such scenarios, the particular topic can still be at least indirectly related to the human-computer dialogue, and therefore, it may be meaningful to reconcile the measure of how much relevance is reduced to the particular topic.

[0100] As mentioned above, the response subsystem 140 (e.g., via dialogue engine 142) can control how the automation assistant 120 engages in human-computer dialogue with the user. In some implementations, the parsing (or interpretation) of the user's natural language input can be provided to a hierarchical decision-making process implemented by dialogue engine 142 (e.g., via dialogue tree 144). In some cases, the hierarchical decision-making process may include a process hierarchy (e.g., the root and child process nodes of dialogue tree 144), in which larger processes manage smaller processes.

[0101] Figure 6 A simplified example of a dialogue tree 644 that can be used during human-computer dialogue 670 is illustrated. This example shows how the more available contexts (i.e., the more topics persist in context data structure 152), the less the user needs to say to achieve certain goals (e.g., to get a response from the automation assistant 120, to have the automation assistant initiate a task, etc.). Assume the following example: the following syntax rules are defined for GAME A, which may be a mini-game:

[0102] (allow_always)

[0103] (rule$play_game(I want to play Game A for three players)

[0104] (=(event_name play_game_A)(players 3))

[0105] (allow_if(medium game_A))

[0106] (rule$player_choice(I want three players)

[0107] (=(event_name player_choice)(number 3))

[0108] (allow_if(high player_question))

[0109] (rule$number($PT_Number)

[0110] (=(event_name number)(value 3))

[0111] It is evident from these grammatical rules that a user can always explicitly state at the outset that he or she wants to play GAME A with three players. This statement includes all the parameters required by GAME A to begin playing the game. If the topic "game_A" has at least an intermediate measure of relevance (e.g., the player says, "I want to play GAME A," but provides no additional information), then the player can say "I need <non-zero integer> players" to begin playing the game. Additionally, if the automation assistant 120 has just asked, "How many players?", then the topic "player_question" can have a sufficiently high measure of relevance that allows the user to simply provide a numerical response.

[0112] In human-computer dialogue 672, the user begins by stating "I want to play GAME A". The topic "game_A" (and potentially other topics such as "game") can be added to the context data structure 152, and child nodes for GAME A can be added to the root node. Child nodes for GAME A can represent child procedures associated with GAME A. When developing such child procedures (or specifying multiple child procedures for a dialogue specific to an application), the developer can define (and in some implementations, store in the syntax database 134) syntax rules such as those described above. In this example, GAME A includes three child procedures: GAME MODE (controlling whether the implementation is single-player or multiplayer), PLAYER (controlling the number of players if multiplayer mode is implemented), and MULTIPLAYER LOOP (controlling multiple players playing the game).

[0113] Based on the GAME MODE sub-procedure, the automation assistant 120 replies, “Okay, single-player or multiplayer?” Multiplayer responds, “Multiplayer.” In some implementations, the sub-procedure associated with the GAME MODE node can interpret the user's input (e.g., using the syntax rules provided associated with the GAME MODE node) and provide parsing / interpretation to the procedure represented by the GAME A node. Next, based on the sub-procedure associated with the PLAYERS node, the automation assistant 120 can ask, “How many players?” The players respond, “Three.” In some implementations, the sub-procedure associated with the PLAYERS node can interpret the user's input (e.g., using the syntax rules provided associated with the PLAYERS node) and provide parsing / interpretation to the procedure represented by the GAME A node. Once the procedure associated with the GAME A node knows the game mode (multiplayer) and the number of players (three), it can then pass control to the MULTIPLAYER LOOP node, which controls the dialogue during the game.

[0114] In some implementations, each child node procedure of the dialogue tree can be designed to perform three functions: handling a specific user interpretation (i.e., parsing), communicating with its nearest parent node, and communicating with its nearest child node. For example, a child node procedure can perform the function of handling a specific user interpretation by associating with one or more topics that are applicable in the current context (e.g., stored in a context data structure). Figure 6 In this context, the node associated with GAME A may be able to receive parameters required to play the game from multiple sources (such as, directly from the user (e.g., by providing natural language input that is interpreted by the syntax associated with the GAME A node)) and / or from one or more child nodes (e.g., receiving the game mode from the GAME MODE node, receiving the number of players from the PLAYERS node).

[0115] Figures 7A-7C This demonstrates how to dynamically generate a dialogue tree during the following human-computer interaction: 744

[0116] USER: I want to play GAME A

[0117] AA: Okay, how many players are there?

[0118] USER: First, tell me the current time.

[0119] AA: It's 7:15 p.m. now.

[0120] When does the USER:Roses game start today?

[0121] AA: It will start at 8 p.m., 45 minutes later.

[0122] USER: Okay. I need three players.

[0123] AA: Okay, let's start playing the game. The first question...

[0124] As mentioned above, since there are no other topics in the context data structure, the root node interprets the user's first statement ("I want to play GAME A"). Figure 7A The description shows that a user's request can initiate the child node process GAME A, and, in some cases, the MULTIPLAYER LOOP. In some implementations, the automation assistant's response ("Okay, how many players?") can cause one or more additional child node processes, such as PLAYERS, to be added below node GAME A.

[0125] Figure 7B Dialogue tree 744 is depicted after the user abruptly changes the topic by saying, "First, tell me the current time." When the user provides this statement, the topic "time" has not yet been added to the context data structure 152. Therefore, control is passed back to the root node, which processes the natural language input if no topic associated with any active sub-process node is applicable (e.g., matches) to the newly proposed topic. The root node is able to interpret the user's request and initiate another sub-process node called TIME, which can be configured to return the current time.

[0126] Figure 7C Dialogue tree 744 is depicted after the user abruptly changes the topic again. This time, the user asks, “When does Roses’ game start today?” Potentially applicable topics (such as “TV,” “series,” etc.) do not persist in context data structure 152. Therefore, control is again passed back to the root node, which interprets the user’s natural language input and responds: “Starts in 45 minutes, at 8 PM.” The user then manipulates the topic back to GAME A, saying, “Okay, I need three players.” Because the topic “GAME A” still persists in context data structure 152, the syntax associated with the topic “GAME A” can be applied to the procedures of child nodes of GAME A.

[0127] From the above explanation and examples, it should be clear that in some implementations, child procedures in the dialogue tree are associated with topics. The topics existing in the context data structure 152 are based on the context of the ongoing human-computer dialogue, which changes over time. Therefore, the child procedures existing in the dialogue tree can also be based on the context of the ongoing human-computer dialogue. Thus, as the context of the ongoing human-computer dialogue changes, the dialogue tree also changes dynamically over time.

[0128] In some implementations, child procedures can be implemented as categories in various programming languages ​​(e.g., C++, Java, etc.). Each child procedure can be configured to exchange information with its parent and child (if any), respond to the parsing / interpretation of user-provided natural language input, persist in certain states, and in some cases, communicate with remote procedures, such as remote procedure calls that can return several pieces of information (like weather, sports ratings, news, TV programs, general information (e.g., information associated with entities in a knowledge graph)).

[0129] In some implementations, child node procedures may include methods of statically defined interfaces. In simplex communication, this response is not expected. Only the called party exposes the interface, as follows:

[0130] The sample code is shown below:

[0131] void SomeChildClass::set_difficulty(intlevel);

[0132] On the other hand, in full-duplex communication, the receiver must be able to return the result to the caller. Therefore, both parties can expose their interfaces:

[0133] void SomeChildClass::ask_for_game_settings();

[0134] void ParentClass::game_settings_callback(const GameSettings&settings);

[0135] If template programming is used (for example, in C++), it may be possible to ensure that the parent procedure implements the interface required by its child procedures, for example, to enable full-duplex communication. Therefore, errors caused by inconsistencies in the communication protocol between the calling and called parties can be detected at compile time.

[0136] Such a well-defined communication interface can facilitate code reuse. For example, suppose process A is intended to use logic provided by process B. If process B does not return any data to its parent process, process A can simply add process B as a child process. On the other hand, if process B is configured to return data to its parent process (e.g., via interface IB), then process A can be configured to implement interface IB, and then process B can be added as a child process.

[0137] In some implementations, each category of the implementation process can implement a method responsible for parsing / interpreting natural language input received from the user. The state of a category instance can be maintained within the subject object associated with that category instance. Each category can have access to inherited components that provide interfaces to the outside world for making various calls (e.g., remote procedure calls).

[0138] Figure 8 This is a flowchart illustrating an example method 800 according to the implementation disclosed herein. For convenience, the operations in the flowchart are described with reference to the system performing these operations. This system may include various components of various computer systems, such as one or more components of the automation assistant 120. Furthermore, although the operations of method 800 are shown in a specific order, this is not intended to be limiting. One or more operations may be reordered, omitted, or added.

[0139] At box 802, the system can receive natural language input from a user. Natural language input can be received in various forms via various types of input devices. For example, in some implementations, the input device can be a microphone, and the natural language input can be audible speech provided by the user. In some such implementations, tokens and other information can be extracted from the audio signal, and the tokens and other information can be used to generate text corresponding to the spoken input. In other implementations, for example, natural language input can be received as typed or spoken input provided by the user in a message exchange thread using message exchange client 107.

[0140] At box 804, the system can select one or more grammars to parse the natural language input. This selection can be based on one or more topics that persist as part of the context data structure 152, and / or on a measure of relevance associated with these topics. In an implementation where the dialogue tree is adopted by the response subsystem 140, control can be passed to the root node if no topic persists in the context data structure 152, or if no topic currently associated with any subprocess node is applicable (e.g., a match) to the most recently proposed topic. The root node can then parse the natural language input and react accordingly (e.g., obtain information and provide a natural language response, initiate one or more applications / processes, etc.). If the user's natural language input is not understood at the root node, the automation assistant 120 can request clarification from the user.

[0141] At box 806, the system can parse the natural language input received at box 802 based on the grammar selected at box 804. In some implementations, multiple grammars are applicable, and therefore, a parsing / interpretation of the natural language input can be generated. The system can select from these multiple (potentially conflicting) interpretations in various ways. In some implementations, the system can select the parsing generated by the grammar that requires the most attention to the subject it is associated with (e.g., associated with the highest measure of relevance). For example, suppose a first grammar is associated with the topic "games" but only requires the topic "games" to have a moderate or even slight measure of relevance. Further suppose a second grammar, also associated with the topic "games," requires a relatively high measure of relevance. Both grammars can be applicable if the topic "games" is currently associated with a high measure of relevance—e.g., because the subject was recently proposed or the topic is semantically related to other recently proposed topics. However, due to the high attention threshold of the second grammar, the system can choose the parsing generated by the second grammar. In other implementations, the system can select from multiple parsings / interpretations based on other signals, such as the probability that each parser represents the user's true intent. For example, in some implementations, logs, relevance metrics, or other similar data points can be used to calculate the probability that each of the multiple parsings matches the user's true intent. Where multiple parsings appear to have similar probabilities of being correct, the system can seek additional clarification and / or disambiguation from the user.

[0142] In some implementations, at box 808, the system can generate a natural language response based on one or more parsings generated at box 806. In some such implementations, the natural language response may accompany the system initiating processes (such as the aforementioned mini-games). Furthermore, while it is mentioned that the mini-games provided as examples above are integrated with the automation assistant (i.e., described as seamlessly integrated into an IVR application within an ongoing human-computer dialogue), this is not intended to be limiting. In various implementations, the system can initiate tasks that do not involve IVR, such as opening applications that do not necessarily require control through spoken or typed natural language input (e.g., phone apps, calendar apps, social networking apps, graphical video games, fitness apps, etc.). At box 810, the system can output the natural language response generated at box 808. For example, the automation assistant can answer user questions, look for another parameter in a process, engage in a dialogue with the user, provide output indicating that the automation assistant is initiating another application, etc.

[0143] At box 812, the system can identify topics proposed by the parsing generated at box 806 and / or the natural language response generated at box 808. At box 814, the system can update the context structure based on the identified topics. If topics have not yet been added to the context data structure, they can be added. If topics already exist as part of the context data structure, they can be "triggered," for example, by increasing the measure of relevance associated with these topics. Additionally, in some implementations, other topics in context data structure 152 that are semantically related to the newly proposed topic can also be "triggered" (e.g., to the same extent as the added topic, or in some cases, to a lesser extent).

[0144] Figure 9 The block diagram of the example computing device 910 may be used to perform one or more aspects of the techniques described herein. In some implementations, one or more of the client computing device, automation assistant 120, and / or other components may include one or more of the example computing device 910.

[0145] Computing device 910 typically includes at least one processor 914 that communicates with several peripheral devices via a bus subsystem 912. These peripheral devices may include a storage subsystem 924, which includes, for example, a memory subsystem 925 and a file storage subsystem 926; a user interface output device 920; a user interface input device 922; and a network interface subsystem 916. The input and output devices allow users to interact with computing device 910. The network interface subsystem 916 provides an interface to an external network and is coupled to corresponding interface devices in other computing devices.

[0146] User interface input device 922 may include a keyboard, pointing device (such as a mouse, trackball, touchpad, or drawing tablet), scanner, touchscreen integrated into the display, audio input device (such as a speech recognition system, microphone), and / or other types of input device. Generally, the term "input device" is used to refer to all possible types of means and methods for inputting information into computing device 910 or into a communication network.

[0147] User interface output device 920 may include a display subsystem, a printer, a fax machine, or a non-visual display such as an audio output device. The display subsystem may include a cathode ray tube (CRT), a flat panel device (such as a liquid crystal display (LCD)), a projector device, or other mechanisms for creating visual images. The display subsystem may also provide a non-visual display such as via an audio output device. Generally, the term "output device" is used to encompass all possible types of means and methods for outputting information from computing device 910 to a user or to another machine or computing device.

[0148] Storage subsystem 924 provides programming and data construction capabilities for some or all of the modules described herein. For example, storage subsystem 924 may include logic components to perform... Figure 8 The aspects of the method chosen and its implementation Figure 1 The components depicted in the text.

[0149] These software modules are typically executed by processor 914 alone or in combination with other processors. The memory subsystem 925 used in storage subsystem 924 can include several memories, including main random access memory (RAM) 930 for storing instructions and data during program execution and read-only memory (ROM) 932 for storing fixed instructions. File storage subsystem 926 provides permanent storage for program and data files and can include hard disk drives, floppy disk drives with associated removable media, CD-ROM drives, optical drives, or removable media cartridges. Modules implementing specific functionalities can be stored in storage subsystem 924 or in other machines accessible by processor 914 by file storage subsystem 926.

[0150] The bus subsystem 912 provides a mechanism for enabling various components and subsystems of the computing device 910 to communicate with each other as intended. Although the bus subsystem 912 is schematically shown as a single bus, alternative implementations of the bus subsystem may use multiple buses.

[0151] The computing device 910 can be of different types, including workstations, servers, computing clusters, blade servers, server groups, or any other data processing systems or computing devices. Due to the constantly evolving nature of computers and networks, for the purpose of illustrating some implementation methods, [the following is omitted as it is not part of the main text]. Figure 9 The description of the computing device 910 depicted is intended only as a specific example. Figure 9 Compared to the computing device depicted in the diagram, many other configurations of the computing device 910 can have more or fewer components.

[0152] In the specific implementations discussed herein, personal information about users (e.g., user data extracted from other electronic communications, information about the user's social networks, the user's location, the user's time, the user's biometric information, and the user's activity and demographic information) may be collected or used. Users are provided with one or more opportunities to control whether information is collected, whether personal information is stored, whether personal information is used, and how information about the user is collected, stored, and used. That is, the systems and methods discussed herein collect, store, and / or use user personal information only after receiving explicit authorization from the relevant user. For example, users are provided with control over whether a program or feature collects user information about that particular user or other users associated with that program or feature. For each user for whom personal information is to be collected, one or more options are presented to allow control over the collection of information related to that user, providing permission or authorization regarding whether information is collected and which parts of the information are to be collected. For example, users may be provided with one or more such control options over a communication network. Furthermore, certain data may be processed in one or more ways before being stored or used, such that personally identifiable information is removed. As an example, a user's identity may be processed so that personally identifiable information cannot be determined. As another example, a user's geographic location can be generalized to a larger area, making it impossible to determine the user's specific geographic location.

[0153] While several implementations have been described and illustrated herein, various other means and / or structures may be utilized for performing functions and / or obtaining results and / or one or more of the advantages described herein, and each of these variations and / or modifications is considered to be within the scope of the implementations described herein. More generally, all parameters, dimensions, materials, and configurations described herein are intended to be exemplary, and actual parameters, dimensions, materials, and / or configurations will depend on the specific application for which this teaching is used. Those skilled in the art will recognize or be able to determine several implementations equivalent to the specific implementations described herein using conventional experimentation. Therefore, it is to be understood that the foregoing implementations are presented by way of example only, and within the scope of the appended claims and their equivalents, implementations may be practiced in other ways as specifically described and claimed. Implementations of this disclosure relate to each individual feature, system, article, material, toolkit, and / or method described herein. Furthermore, any combination of two or more such features, systems, articles, materials, kits, and / or methods is included within the scope of this disclosure if such features, systems, articles, materials, kits, and / or methods do not conflict with each other.

Claims

1. A method implemented using one or more processors, comprising: receiving a natural language input from a user as part of an ongoing human-to-computer dialog between the user and an automated assistant operated by one or more of the processors, wherein one or more previously raised topics during the ongoing human-to-computer dialog are stored in a memory as part of a context data structure associated with the ongoing human-to-computer dialog; contextually filtering a subset of one or more grammars associated with the previously raised one or more topics from a superset of grammars associated with a plurality of respective topics, wherein the contextual filtering is based on one or more respective measures of relevance of the previously raised one or more topics to the ongoing human-to-computer dialog; parsing the natural language input based on the contextually filtered one or more grammars to generate one or more parses after the contextual filtering; generating a natural language response based on one or more of the parses; outputting the natural language response to the user using one or more output devices; identifying one or more new topics raised by one or more of the parses or the natural language response; adding the identified one or more new topics to the context data structure; and boosting the measure of relevance of a given topic of the previously raised topics to the ongoing human-to-computer dialog based on a measure of semantic relevance between the given topic and the one or more new topics.

2. The method of claim 1, wherein, the measure of relevance associated with each given topic of the one or more topics is further determined based at least in part on a count of turns of the ongoing human-to-computer dialog since the given topic was last raised, wherein the count of turns since the given topic was last raised inversely relates to the given topic's relevance to the ongoing human-to-computer dialog.

3. The method of claim 1, wherein, the context data structure comprises an undirected graph comprising a plurality of nodes and a plurality of edges connecting the plurality of nodes.

4. The method of claim 3, wherein, each node of the undirected graph represents a given topic of the previously raised one or more topics stored as part of the context data structure and a count of turns of the ongoing human-to-computer dialog since the given topic was last raised.

5. The method of claim 4, wherein, each edge connecting two nodes represents a measure of semantic relevance between two topics represented by the two nodes, respectively.

6. The method of claim 1, further comprising generating a dialog tree having one or more nodes representing one or more interactive voice processes that have been invoked during the ongoing human-to-computer dialog.

7. The method of claim 6, wherein, one or more of the nodes are associated with one or more topics.

8. A system comprising one or more processors and memory operatively coupled with the one or more processors, wherein, the memory stores instructions that, in response to execution of the instructions by the one or more processors, cause the one or more processors to: receiving a natural language input from a user as part of an ongoing human-to-computer dialog between the user and an automated assistant operated by one or more of the processors, wherein one or more previously raised topics during the ongoing human-to-computer dialog are stored in memory as part of a context data structure associated with the ongoing human-to-computer dialog; contextually filtering a subset of one or more grammars associated with the previously raised one or more topics from a superset of grammars associated with a plurality of respective topics, wherein the contextual filtering is based on one or more respective measures of relevance of the previously raised one or more topics to the ongoing human-to-computer dialog; after the contextual filtering, resolving the natural language input based on the contextually filtered one or more grammars to generate one or more resolutions; generating a natural language response based on one or more of the resolutions; outputting the natural language response to the user using one or more output devices; identifying one or more new topics raised by one or more of the resolutions or the natural language response; adding the identified one or more new topics to the context data structure; and promoting, based on a measure of semantic relevance between a given topic of the previously raised topics and the one or more new topics, the measure of relevance of the given topic to the ongoing human-to-computer dialog.

9. The system of claim 8, wherein, the measure of relevance associated with each given topic of the one or more topics is further determined based at least in part on a count of turns of the ongoing human-to-computer dialog since the given topic was last raised, wherein the count of turns since the given topic was last raised inversely relates to the relevance of the given topic to the ongoing human-to-computer dialog.

10. The system of claim 8, wherein, the context data structure comprises an undirected graph comprising a plurality of nodes and a plurality of edges connecting the plurality of nodes.

11. The system of claim 10, wherein, each node of the undirected graph represents a given topic of the previously raised one or more topics stored as part of the context data structure and a count of turns of the ongoing human-to-computer dialog since the given topic was last raised.

12. The system of claim 11, wherein, each edge connecting two nodes represents a measure of semantic relevance between two topics respectively represented by the two nodes.

13. The system of claim 8, further comprising generating a dialog tree having one or more nodes representing one or more interactive voice processes that have been invoked during the ongoing human-to-computer dialog.

14. The system of claim 13, wherein, one or more of the nodes are associated with one or more topics.

15. At least one non-transitory computer-readable medium comprising instructions that, in response to execution of the instructions by one or more processors, cause the one or more processors to perform the following operations: receiving a natural language input from a user as part of an ongoing human-to-computer dialog between the user and an automated assistant operated by one or more of the processors, wherein, storing one or more previously raised topics during the ongoing human-to-computer dialog in memory as part of a context data structure associated with the ongoing human-to-computer dialog; contextually filtering a subset of one or more grammars associated with the one or more previously proposed topics from a superset of grammars associated with a plurality of respective topics, wherein the contextual filtering is based on one or more respective measures of relevance of the one or more previously proposed topics to the ongoing human-to-computer dialog; after the contextual filtering, resolving the natural language input based on the one or more grammars of the contextual filtering to generate one or more resolutions; generating a natural language response based on one or more of the resolutions; outputting the natural language response to the user using one or more output devices; identifying one or more new topics proposed by one or more of the resolutions or the natural language response; adding the identified one or more new topics to the contextual data structure; and promoting, based on a measure of semantic relevance between a given topic of the previously proposed topics and the one or more new topics, the measure of relevance of the given topic to the ongoing human-to-computer dialog.

16. The at least one non-transitory computer-readable medium of claim 15, wherein, the measure of relevance of each given topic of the one or more topics is further determined based at least in part on a count of turns of the ongoing human-to-computer dialog since the given topic was last proposed, wherein the count of turns since the given topic was last proposed is inversely related to the relevance of the given topic to the ongoing human-to-computer dialog.

17. The at least one non-transitory computer-readable medium of claim 15, wherein, the contextual data structure comprises an undirected graph comprising a plurality of nodes and a plurality of edges connecting the plurality of nodes.

18. The at least one non-transitory computer-readable medium of claim 17, wherein, each node of the undirected graph represents a given topic of the one or more previously proposed topics stored as part of the contextual data structure and a count of turns of the ongoing human-to-computer dialog since the given topic was last proposed.

19. The at least one non-transitory computer-readable medium of claim 18, wherein, each edge connecting two nodes represents a measure of semantic relevance between the two topics represented by the two nodes, respectively.

Images