Clarifying questions to rewrite ambiguous user utterances

Abstract

A method for disambiguating a user query comprising a set of user utterances in a multi-turn conversation comprises: using a predefined language model to identify ambiguous entities in unresolved user utterances from the multi-turn conversation. The method further comprises: outputting a clarifying question about the ambiguous entity, and receiving a clarifying user utterance. The method further comprises: identifying a disambiguating entity in the clarifying user utterance. The method further comprises: rewriting the unresolved user utterance into a rewritten utterance that replaces the ambiguous entity with the disambiguating entity, and outputting the rewritten utterance to one or more query answering machines.

Description

Background Art

[0001] Automated assistant systems are used to interact with users via natural language, for example, to answer queries. However, user utterances may include ambiguities that may make processing the user utterances difficult or impossible. Summary of the invention

[0002] This summary is provided to introduce a series of concepts in a simplified form, which are further described below in the detailed description. This summary is neither intended to identify the key features or essential features of the claimed subject matter, nor is it intended to be used to limit the scope of the claimed subject matter. In addition, the claimed subject matter is not limited to embodiments that address any or all of the disadvantages mentioned in any part of this disclosure.

[0003] A user query comprising a set of user utterances in a multi-turn dialogue is disambiguated using a predefined language model that identifies ambiguous entities in unresolved user utterances from the multi-turn dialogue. Clarifying questions about the ambiguous entities are output, and clarifying user utterances are received. Ambiguous entities are identified in the clarifying user utterances, and the unresolved user utterances are rewritten to replace the ambiguous entities with the disambiguating entities. The rewritten utterances are output to one or more query answering machines. BRIEF DESCRIPTION OF THE DRAWINGS

[0004] Figure 1A Shown is an architecture for processing user utterances.

[0005] Figure 1B An exemplary runtime state format for an architecture for processing user utterances is shown.

[0006] Figure 2A-2M Exemplary runtime states are shown for exemplary conversations associated with different speech acts.

[0007] Figure 3 A method for resolving ambiguous entities in a user utterance is shown.

[0008] Figure 4 A method for resolving suggested entities or corrected entities in a user utterance is shown.

[0009] Figure 5 A method for dissolving suggestion intent in user utterances is shown.

[0010] Figure 6 A method for clarifying ambiguous user utterances is shown.

[0011] Figure 7 An exemplary computer system is shown. DETAILED DESCRIPTION

[0012] The automated agent system can enable a user to interact with a computer device via natural language. For example, a user can issue a command or query via a natural language utterance, and the automated agent system can control the computer device to respond to the natural language command / query.

[0013] The automated agent system may be configured to process natural language utterances in the form of computer-readable text, parse trees, or other computer-readable data structures. In some examples, a user may interact with the automated agent system by sending utterance text (e.g., text entered at a keyboard or any other appropriate input device). In some examples, a user may interact with the automated agent system through audible speech, which may be received, for example, at a microphone associated with the automated agent system and processed by any appropriate language-to-text technology. Similarly, the automated agent system may output utterances to convey information to the user in any appropriate form, such as by outputting text to a display device and / or by outputting audible speech via a speaker.

[0014] In some examples, the user utterance defines a query, representing a question to be answered by the automated agent system. The automated agent system can answer a variety of different questions, including domain-specific queries (e.g., sports questions, movie questions, geopolitical news questions) and / or domain-independent queries (e.g., web searches). For example, the automated agent system can be trained to recognize a specific domain for a query (e.g., sports) and use domain-specific methods to answer the query (e.g., find relevant scores and highlights from sporting events). In other examples, the automated assistant system may not recognize a specific domain for the query and may use a general domain-independent method (e.g., sending the query text to a web search engine to receive web search results related to the query text) to answer the query.

[0015] In some examples, a user utterance may define a query by including a query intent and one or more query entities. For example, a query may represent a question about one or more query entities, wherein the question is defined by the query intent. As used herein, "entity" may include people, places, things, companies, concepts (e.g., "weather"), animals (e.g., animal species or specific animals), etc. In some examples, a query intent may include one or more attributes of a query entity, such as the expected content of an answer to a query. For example, if a user asks "Who founded Microsoft?", the query contains an entity: "Microsoft" and a query for the relevant intent of the "founder of the company" attribute of the "Microsoft" entity. The query intent may be associated with any related words from the user utterance (e.g., the word "founder" is associated with the "founding" attribute). The query intent may be represented by verbs, adjectives, and / or nouns of the user utterance in various ways, for example, the same query intent about the "Microsoft" entity may be defined by different queries: "Who is the founder of Microsoft?"

[0016] In some examples, a user utterance may be an utterance of a multi-turn conversation between one or more users and an automated agent. The user utterance may be contextually related to other utterances in the computer-accessible conversation history of the multi-turn conversation (e.g., previous utterances of (multiple) users and / or automated agents). Therefore, without placing the information provided by other utterances in the computer-accessible conversation history in context, the user utterance may be ambiguous. For example, the user utterance may refer to a previously discussed entity via a coreference such as a pronoun or a partial name. In the example described above, in response to the query "Who founded Microsoft?", the automated agent may respond "Bill Gates and Paul Allen". If the user then issues another query "What is Bill's net worth?", the automated agent may use the method of the present disclosure to infer that the user wants to know Bill Gates' net worth. However, without the context provided by the first query ("Who founded Microsoft?") and the automated agent's reply ("Bill Gates and Paul Allen"), the query "What is Bill's net worth?" will be ambiguous, i.e., because the partial name "Bill" is not specific enough about which "Bill" the user is asking.

[0017] Thus, FIG. 1 illustrates an exemplary data flow architecture 100 for disambiguating user utterances in a multi-turn conversation. The data flow architecture 100 conceptually works by dissolving any coreferences in the user utterances 106 in the coreference resolution machine 102, classifying the speech acts represented by the user utterances, and delegating further processing of the user utterances to one or more downstream machines based on the detected speech acts. However, such a data flow is by no means limiting, and other data flows may be used. In some examples, processing the detected speech acts also includes: when one or more ambiguities from the user utterances exist even after the resolution of the coreferences, dissolving such ambiguities. For example, an ambiguous suggestion to discuss a new topic can be resolved into a fully resolved query. In some examples, the resolution of ambiguities is fully automated based on the context provided by the computer-accessible conversation history of the multi-turn conversation. In some examples, the resolution of ambiguities includes asking the user clarifying questions and receiving clarifying replies. In any case, after resolving ambiguity in the user utterance (e.g., via coreference resolution, resolving ambiguous suggestions to discuss a new topic, or asking clarifying questions), the rewritten utterance may be output to one or more downstream query answering machines for further processing. The rewritten utterance is a fully resolved query that defines all information needed to process the query, such that further processing of the computer-accessible conversation history may not be necessary to process the rewritten utterance. The dataflow architecture 100 and / or other suitable dataflow architectures may be used in any suitable scenario involving natural language interaction with a user, such as to provide an automated agent.

[0018] One or more query answering machines may return multiple different answers to a given query. In some examples, data flow architecture 100 may be configured to obtain such multiple different answers and select one or more answers for providing to the user via an automated proxy reply. For example, different answers may each have an associated confidence value evaluated by the query answering machine, and one answer may be selected based on having the highest confidence value, or two answers may be selected and presented as possibilities.

[0019] The components of the data flow architecture 100 can be implemented using any suitable combination of existing and / or future machine learning and / or natural language techniques. In some examples, the decisions, scores, and any other calculations described herein can be implemented via a predefined language model. The predefined language model can be implemented via any combination of machine learning, artificial intelligence, and / or statistical models (e.g., via discriminative and / or generative models). Figure 7to further describe an exemplary model. The predefined language model can be trained using any appropriate process with respect to any appropriate objective function (e.g., with respect to user satisfaction with responses to user utterances). For example, the predefined language model can be trained via supervised training using labeled data examples from historical sessions and / or reinforcement training based on user feedback data. The predefined language model can be trained in an "end-to-end" manner so that all components described herein are simultaneously optimized with respect to the combined functionality of the data flow architecture 100 and / or (multiple) automated agent systems implemented using the data flow architecture 100. In some examples, the predefined language model may include one or more knowledge bases, databases, and / or search engine tools configured to dynamically retrieve information related to entities, intents, and / or natural language phrases. For example, a predefined language model may include a knowledge base containing information about people, organizations, and the like and including entity attributes, relationships between entities, and the like.

[0020] The data flow architecture 100 includes a speech act classifier machine 110 configured to detect a plurality of different predefined speech acts in the user utterance 106 with respect to the conversation history 202. As each new user utterance 106 is processed, the conversation history 202 is updated to reflect the conversation so far. Prior to classification by the speech act classifier machine 110, the user utterance 106 may be pre-processed by the coreference resolution machine 102 to resolve any ambiguous coreferences (e.g., ambiguous pronouns, partial names, etc.) present in the user utterance 106. Reference Figure 2C-2F and Figure 3 The coreference resolution machine 102 is further described below.

[0021] For example, Figure 1A As shown in , the speech act classifier machine 110 can use a predefined language model to detect predefined speech acts including the following items: "greeting", "confirmation", "rejection", "fully resolved query", "entity suggestion", "entity correction", "intent suggestion", "clarification" and "ambiguity". Based on the detection of specific predefined speech acts, the speech act classifier machine 110 is configured to delegate further processing of the user's utterance to one or more downstream operators, including a greeting / confirmation / rejection reply machine 112, an entity suggestion / correction resolver machine 118, an intent suggestion resolver machine 120, a clarification technology machine 116, a clarification resolution machine 122, and (multiple) query answering machines 114. These downstream operators, as well as the coreference resolution machine 102 and the speech act classifier machine 110, can be implemented on any appropriate computer device or combination of computer devices. For example, the machine described herein can be implemented in an integrated computing device having a single housing, as a separate processing node in a cloud computing scenario, or in any other appropriate manner.

[0022] "Greeting", "confirmation", and "rejection" are exemplary speech acts in which a user greets an automated agent, confirmation of a reply is provided by the automated agent (e.g., an OK), or a reply is rejected by the automated agent. In response to detecting one of these speech acts, speech act classifier machine 110 may delegate processing of user utterance 106 to greeting / confirmation / rejection reply machine 112, described below with reference to Figure 2B to further describe it.

[0023] A "fully resolved query" speech act includes any user utterance that fully defines a query for processing by a downstream query answering machine. In other words, a "fully resolved query" speech act is a user utterance that does not have any ambiguity and does not rely on the context of the conversation history for processing. In response to detecting a fully resolved query, the speech act classifier 110 can delegate the processing of the user utterance 106 to a query answering machine 114 that is configured to return an answer to the query in the user utterance 106. The query answering machine 114 can be configured to respond to fully defined queries that only explicitly mention any relevant entities, query intent, etc. The query answering machine 114 may include domain-specific technology 114S, such as domain-specific technology for sports queries or domain-specific technology for celebrity queries. The query answering machine 114 may alternatively or additionally include domain-independent technology, such as web search technology for answering arbitrary queries via web search.

[0024] "Entity suggestion" and "entity correction" speech acts are utterances in which a user intends to change the topic to discuss a different entity with respect to a query intent established by a previous user or agent utterance. An "entity suggestion" or "entity correction" utterance is not, by itself, a fully defined query, as the change of topic requested is with respect to a context established in the conversation history 202. In response to detecting an entity suggestion or entity correction, the speech act classifier machine 110 may delegate processing of the user utterance 106 to an entity suggestion / correction resolver machine 118, which is configured to rewrite the query in an unambiguous manner that incorporates any relevant information from the conversation history 202, and in turn delegate processing of the rewritten query to the query answering machine 114. References Figure 2G , Figure 2H , Figure 4 and Figure 5 Entity suggestion / correction resolver machine 118 is further described below.

[0025] An “intent suggestion” speech act is an utterance in which a user wants to change the topic to discuss a different query intent with respect to an entity mentioned in a previous user or agent utterance. An “intent suggestion” utterance is not itself a fully defined query because the requested change of topic is with respect to an entity previously discussed in the conversation history 202. In response to detecting an entity suggestion or entity correction, the speech act classifier machine 110 may delegate processing of the user utterance 106 to an intent resolver machine 118, which is configured to rewrite the query in an unambiguous manner that incorporates any relevant information from the conversation history 202, and in turn delegate processing of the rewritten query to the query answering machine 114.

[0026] An "ambiguous" speech act is any ambiguous utterance that cannot be disambiguated based on information in the user utterance 106 and / or the conversation history 202. For example, an ambiguous utterance may include a suggestion that cannot be resolved as a co-reference to any entity from the conversation history 202, a co-reference to two different entities corresponding to the conversation history 202, or a change in topic about a proposed entity and / or an intent that is ambiguous. In response to detecting an ambiguity, the speech act classifier 110 is configured to delegate further processing of the user utterance 106 to a clarification technique 116 configured to output clarification questions. Subsequent utterances received from the user may be classified as "clarifying" speech acts. In response to detecting a clarifying speech act, the speech act classifier machine 110 is configured to delegate processing of subsequent user utterances to a clarification resolution machine 122 configured to rewrite the original user utterance 106 so as to disambiguate the original user utterance 106 by incorporating relevant information from the clarifying speech act. Thus, if the rewritten utterance is completely disambiguated, the rewritten utterance may be further processed by the query answering machine 114. If remaining ambiguity remains after clarifying the speech act, the rewritten utterance may be further processed by the clarification technique machine 116 before ultimately delegating processing of the rewritten utterance to the query answering machine 114 to output one or more additional clarifying questions.

[0027] The methods described herein can enable a downstream query answering machine to return an answer to a user utterance containing ambiguity without requiring (multiple) downstream query answering machines to process disambiguation. These methods include providing an unambiguous fully formed rewritten utterance to (multiple) downstream query answering machines. The fully formed rewritten utterance can be processed by (multiple) downstream query answering machines in any appropriate manner. In some examples, the rewritten utterance is provided to multiple downstream query answering machines, which can all process the rewritten utterance (e.g., in parallel). Since disambiguation is performed before the rewritten utterance, the rewritten utterance can be processed before any additional disambiguation steps are performed, which can reduce the latency, memory requirements, and / or computational costs for the downstream query answering machine to return an answer.

[0028] In some examples, the data flow architecture 100 can be configured to provide multiple different possible rewritten queries to one or more query answering machines, and receive answers for each of the different possible rewritten queries. In some examples, the data flow architecture 100 can be configured to provide multiple different answers in a single agent discourse. For example, if a user asks "Who does Russell play for?", then the method of the present disclosure can be deduced based on the conversation history that the user is referring to "Russell Wilson" or "Russell Westbrook". Therefore, two different rewritten queries "Who does Russell Westbrook play for?" and "Who does Russell Wilson play for?" can be published to one or more query answering machines. Based on the results from (multiple) query answering machines, the data flow architecture 100 can determine similar confidence values ​​for the answers to these two different questions. Therefore, the data flow architecture 100 can be configured to output a discourse that incorporates answers to two rewritten queries, such as "Russell Wilson plays for the Seahawks; but if you mean Russell Westbrook, then he plays for the Thunder".

[0029] In some examples, the user utterance 106 can be placed in context by a computer-accessible conversation history 202 that represents a multi-turn conversation between a user and an automated agent. Thus, the computer-accessible conversation history 202 can include multiple utterances of a user and / or an automated agent. In the illustrated example, the user utterance 106 is the most recent user utterance, occurring after other user and / or automated agent utterances.

[0030] The user utterance 106 may mention one or more entities, for example to define a query topic. In some examples, the user utterance 106 may mention one or more entities using the full name of the entity, which can be easily identified by a downstream operator (e.g., by searching in a knowledge base of the entity based on the full name). However, in other examples, the entity may be mentioned by coreference. As used herein, "coreference" describes the portion of an utterance that mentions a particular entity. A single particular entity may be mentioned by multiple different coreferences in a user utterance and / or in a series of user utterances.

[0031] Figure 1B An exemplary runtime state format for a multi-turn dialogue 200 is shown. The runtime state format for a multi-turn dialogue 200 can be processed in any suitable manner according to the methods and examples of the present disclosure. For example, the runtime state format for a multi-turn dialogue 200 can be used to track Figure 1AThe runtime state of the data flow architecture 100. The runtime state format for the multi-turn dialogue 200 includes unresolved user utterances 216, which can be represented in any suitable format, such as a computer-readable text string and / or a computer-readable parse tree.

[0032] The runtime state format for multi-turn dialog 200 also includes a computer-accessible conversation history 202. In some examples, computer-accessible conversation history 202 can be empty, including zero utterances. In other examples, computer-accessible conversation history 202 can include a single user utterance and / or a single agent utterance. More generally, computer-accessible conversation history 202 can include any suitable number of user utterances and / or agent utterances. For example, in Figure 1B , computer-accessible conversation history 202 is shown with user utterance 204, agent utterance 206, user utterance 208, agent utterance 210, user utterance 230, agent utterance 232, and other user / agent utterances not shown. The user and / or agent utterances in computer-accessible conversation history 202 can be in chronological order, such as indicated by timestamps or in any other manner. For example, Figure 1B The user and agent utterances in the computer-accessible conversation history 202 may be arranged in a top-to-bottom order representing the temporal first-to-last order when the utterances occurred. The utterances in the computer-accessible conversation history 202 may be stored in any suitable format, such as as a collection data structure (e.g., a list, array, or collection data structure) that includes computer-readable text and / or a parse tree for each utterance. Although Figure 1B A computer-accessible conversation history 202 is shown in which the recorded utterances alternate between user utterances and agent utterances, but the utterances may appear in any other order, e.g., multiple user utterances in a row without agent utterances in reply, multiple agent utterances in response to a single user utterance, or any other order.

[0033] The computer-accessible conversation history 202 may also include a collection of entities 212 from the history. The collection of entities 212 may include any entities included in the discourse in the history, such as entities explicitly named by the user, entities that appear in answers by the agent, and the like. Each entity includes any appropriate details that may be relevant to the conversation. Non-limiting examples of entity details include an entity name, a type of entity (e.g., person, place, etc.), a subtype of the entity (e.g., a person entity may have subtypes such as "founder," "artist," etc.), relationships to other entities, and / or any other entity attributes. In some examples, entity attributes may be stored as key / attribute value pairs. Each entity in the collection of entities may be stored in any appropriate format, such as as a computer-readable grammatical description of a data interchange language (e.g., JSON, XML), according to a database schema, and the like. The collection of entities 212 may be stored in any appropriate format, such as as a database, a collection of data structures, a graphical knowledge base, and the like. For example, as Figure 2C As shown in , the collection of entities 212 from history includes Bill Gates and Paul Allen, organized by name and with attributes / types for each entity.

[0034] A collection of previously resolved entities 212 from history can be identified and / or tracked in any appropriate manner using a predefined language model and / or a query answering machine. As a non-limiting example, the query answering machine can be configured to record each newly mentioned entity that can be resolved unambiguously from any user utterance and / or agent utterance. In some examples, tracking a collection of previously resolved entities can be implemented at least in part by a query answering machine. For example, the query answering machine can be configured to identify entities from an utterance, and is configured to output each entity that appears in the utterance. The query answering machine can also be configured to track entities that appear in a reply to a query as a step included in determining a reply to the query. However, the method described herein is not so limited, and any method of tracking previously resolved entities can be adopted together with the method and / or data stream of the present disclosure. In some examples, a collection of previously resolved entities 212 from history can include previously resolved entities from possible rewritten utterances 218 and / or from possible agent utterances 222. These previously resolved entities may be tracked in any suitable manner, such as by sending possible rewritten utterances 218 to the query answering machine and / or by receiving possible proxy utterances 222 from the query answering machine and previously resolved entities 212 determined by the query answering machine to be relevant to possible proxy utterances 222. Although illustrated as part of computer-accessible conversation history 202, entities 212 from history may be tracked separately and stored on any computing device. As another non-limiting example, entities 212 from history may be tracked by and stored on the query answering machine, and the query answering machine may be consulted to retrieve entity data from the query answering machine and / or to retrieve specific results related to an entity (e.g., looking up an entity attribute by an entity name).

[0035] The runtime state format for the multi-turn dialogue 200 also includes other runtime state information related to processing the unresolved user utterance 216, including entity constraints 224 from the unresolved user utterance. The entity constraints may include entity attributes, names, and / or types, which may be automatically inferred from the unresolved user utterance 216 using a predefined language model according to the present disclosure. The runtime state format for the multi-turn dialogue 200 also includes detected speech acts 220, which may be automatically inferred from the unresolved user utterance 216 by a speech act classifier machine according to the present disclosure. The detected speech act may be one of a plurality of predefined speech acts, and may be represented in any appropriate format, such as being represented as a computer-readable integer index or a computer-readable text string. In some examples, the method of the present disclosure includes rewriting the unresolved user utterance 216 to produce a possible rewritten utterance 218. The possible rewritten utterance 218 is a complete, fully defined utterance that can be processed without reanalyzing the information in the computer-accessible conversation history 202. In some examples, the method of the present disclosure includes determining a possible agent utterance 222 to be output by an automated agent. Possible rewritten utterances 218 and possible agent utterances 222 can each be stored in any suitable format, such as as a computer-readable text string or a computer-readable parse tree. Computer-accessible conversation history 202 can be stored in any suitable location, such as locally on a computing device providing an automated agent, or remotely on a cloud computing server.

[0036] Reference Figure 2A-2M To describe various multi-turn dialogues. Figure 2A-2M In the multi-round dialogue 200, according to the Figure 1B The runtime state format of the multi-turn dialog 200 is shown. Figure 2A-2M Each of them is about a different multi-turn dialogue; however, similar components are used corresponding to Figure 1B The components of the runtime state format of the multi-turn dialog 200 are labeled with a numerical prefix and a letter suffix corresponding to the diagram. For example, Figure 2A A multi-turn dialogue 200A is shown, and Figure 2B A multi-turn dialog 200B is shown. When generally referring to similar components of multiple different multi-turn dialogs, the component may be referenced without an alphabetic suffix. Figure 2A-2M , relevant aspects of the runtime state format for the multi-turn dialog 200 are shown to explain an example usage scenario, while aspects not relevant to the example usage scenario are not shown.

[0037] In such Figure 2AIn some of the examples shown in , the only reference to an entity in the unresolved user utterance 216A is the full name ("Microsoft"), and therefore the coreference resolution machine 102 detects that the unresolved user utterance actually contains a fully resolved query. Therefore, the user utterance can be processed by the speech act classifier machine 110 to detect that the speech act 220A is a fully resolved query. Therefore, the fully resolved query can be forwarded to one or more query answering machines 114 to obtain an answer, which can be output by the agent as a possible agent utterance 222A. The computer-accessible session history 202A also indicates any related entity constraints 224A from the unresolved user utterance, indicating that the user is interested in entities having a "founder" type and founded Microsoft Corporation.

[0038] In some examples, such as Figure 2B As shown in , the unresolved user utterance 216B is a simple "thank you", which does not actually contain any reference to an entity. Therefore, the co-reference resolution machine 102 detects that the unresolved user utterance 216B can be sent directly to the speech act classifier machine 110, which in turn detects confirmation speech acts. Simple speech acts such as statements of confirmation, greeting, or confirmation / rejection can be processed by a greeting / confirmation / rejection reply machine 112 configured to provide replies to these simple speech acts. In addition to politely processing user utterances such as "thank you" or "hello", the greeting / confirmation / rejection reply machine 112 can similarly issue polite replies to other user utterances, such as affirming a proposed course of action, rejecting a proposed course of action, and so on.

[0039] although Figure 2B The unresolved user utterance 216B does not mention any entity, but the computer-accessible session history 202B may still track entities 212B from the history, such as Bill Gates and Paul Allen mentioned in the agent utterance 206B in response to the user utterance 204B. Related attributes for the entity 212B from the history are tracked along with the entity name, for example, tracking entity type(s), entity attributes, etc. Although "Bill Gates" is shown as having tracked "person" and "founder" types and tracked "gender" and "founded company" attributes, the entity may have other associated types and / or attributes associated with characteristics of the entity determined from a knowledge base, web search, query answering machine, etc. For example, for "Bill Gates", other types and / or attributes may be tracked, such as a "creator" type or a "founded company = Bill & Melinda Gates Foundation" attribute.

[0040] In other examples, such as Figure 2C As shown in , one or more coreferences within the unresolved user utterance 216C are ambiguous and / or require context from the computer-accessible conversation history 202. Figure 2C As shown in, the method of the present disclosure (ie, the following will refer to Figure 3 The method 300 discussed above may be used to infer that the partial name "Bill" refers to the entity "Bill Gates" tracked in entity 212C from history 202C. Unresolved user utterance 216C by itself is not sufficient to disambiguate "Bill Gates", (even though entity 212C includes "Bill Gates"), because "Bill Gates" is only referenced by the name "Bill" in unresolved user utterance 216. Thus, while the automated assistant system may be able to process the unresolved user utterance 216C in the context of the computer-accessible conversation history 202C, the unresolved user utterance 216C will not be suitable for processing by a downstream query answering machine that may not be configured to access the computer-accessible conversation history 202C. Note that "Bill Gates" as tracked in entity 212G from history is shown with only a list of types and / or attributes that may be relevant that are shown, e.g., Bill Gates may additionally have a "net worth" attribute defined to track his net worth in US dollars.

[0041] Thus, the coreference resolution machine 102 is configured to pre-process the user utterance 106 to resolve any potentially ambiguous coreferences in the user utterance to produce a rewritten utterance. The rewritten utterance is self-contained for speech act detection by the speech act classifier machine 110 and / or further processing by a downstream query answering machine, etc. For example, in Figure 2C In the example above, after rewriting, the detected speech act 220F is a completely resolved problem.

[0042] Examples of coreference include part names (e.g., Figure 2C ), full name and other names, such as nicknames, titles, etc. In some examples, coreference may also include misspelled / mispronounced names, etc. Thus, user utterances may refer to the same person in a variety of different ways, with each such reference being a coreference to the same entity. As a specific example, user utterances may refer to "Bill Gates" in any of the following various ways: 1) "William Henry Gates III," 2) "Bill Gates," 3) "Bill," 4) "Mr. Gates," or 5) "Gates," as well as other possibilities. Coreference also includes pronouns, for example, Bill Gates may be referred to as "he" or "him," and other nouns related to Bill Gates may be described using possessive pronouns, "his" referring to Bill Gates, for example "his wife" may refer to "Bill Gates' wife." Coreference may also include explicit references to entities from the list of entities from the computer-accessible session history 202, e.g., "first," "second," "third," or any other appropriate description to refer to an entity in the ordered list of entities within the utterance from the computer-accessible session history 202 provided by the user and / or by the agent.

[0043] Figure 2D and 2E Another example of a multi-turn dialog is shown in which an ambiguous entity from an unresolved user utterance 216 is resolved to refer to "Bill Gates." Figure 2D is characterized by a coreference to "Bill Gates" in the form of a reference to "the first" in the ordered list of "Bill Gates and Paul Allen" from agent utterance 206D. Figure 2E is characterized by coreference to "Bill Gates" in the form of the pronoun "he".

[0044] therefore, Figure 3 A method 300 is shown for disambiguating a user query comprising a set of user utterances in a multi-turn conversation by resolving coreference. The coreference machine 102 may be configured to implement the method 300 .

[0045] At 302, method 300 includes identifying ambiguous entities in unresolved user utterances from a multi-turn conversation using a predefined language model (e.g., any suitable predefined language model as described above). An ambiguous entity can be any natural language reference to an entity (e.g., a coreference to an entity as described above). For example, in Figure 2C In , "Bill" is an ambiguous entity. Figure 2D In , "first" is an ambiguous entity. Figure 2E In the sentence, “his” is a possessive pronoun referring to an ambiguous entity.

[0046] At 304, method 300 also includes identifying entity constraints for ambiguous entities using a predefined language model. Entity constraints can be derived from any words of the user utterance (e.g., coreference phrases referring to entities, verbs describing actions performed by entities, adjectives describing attributes of entities, etc.).

[0047] For example, in Figure 2C In the entity constraint, the dependency on the partial name "Bill" and the attribute "Net Assets" is included. Figure 2D , entity constraint 224D includes the relevance of the "net worth" attribute and an indication that the user is interested in the "first" item from the list of items (e.g., "Bill Gates" from the list of "Bill Gates and Paul Allen" in agent utterance 206D). Figure 2E , entity constraints 224E include "gender=male" (e.g., based on the masculine pronoun "he" from the unresolved user utterance 216E) and "defined attribute=founded company" based on the verb phrase "he founded...company" in the unresolved user utterance 216E.

[0048] At 306 , method 300 further includes searching, in the computer-accessible conversation history of the multi-turn dialog, a set of previously resolved entities for a candidate entity having an entity attribute corresponding to a highest confidence of an entity constraint of the ambiguous entity.

[0049] The confidence of the correspondence between the entity constraint and the entity attribute can be determined in any appropriate manner, for example, by measuring the number or proportion of matching entity constraints / attributes. In some examples, the confidence derived for the constraint / attribute match can be weighted differently for different constraints / attributes, for example, matching partial names can be weighted higher than matching gender, and vice versa. In some examples, the confidence value can be evaluated by a predefined language model and / or by one or more other machine learning, artificial intelligence and / or statistical models. More generally, determining the corresponding confidence can be based on the correspondence between any appropriate features of the entity constraint and the computer-accessible conversation history 202 and / or the entity 212 from the history. For example, the entity constraint can correspond to an entity type and / or an entity attribute, and the confidence can be based on the number and / or proportion of the corresponding entity attribute / type. The confidence can be additionally determined by the context provided by the computer-accessible conversation history 202, such as whether the entity appears in an ordered list of entities, and / or how long one of the entities 212 from the history is mentioned in the user speech or agent speech from the computer-accessible conversation history 202.

[0050] In some examples, searching for a candidate entity in a set of previously resolved entities includes resolving entity types based on entity constraints of the ambiguous entity, and thus searching for a candidate entity in a set of previously resolved entities includes finding a subset of previously resolved entities that have corresponding entity types. The corresponding entity types can be the same type or related types from a hierarchy of types. For example, the hierarchy of types can include a "person" type and a "founder" type that is a subtype of the "person" type. Thus, searching for a candidate entity in a set of previously resolved entities includes finding a subset of previously resolved entities that have corresponding entity types. The corresponding entity types can be the same type or related types from a hierarchy of types. For example, the hierarchy of types can include a "person" type and a "founder" type that is a subtype of the "person" type. Figure 2E In the example shown in , when searching based on the "person" type, a subset of previously resolved entities may have the "person" type, the "founder" type, or any other appropriate subtype of the "person" type (e.g., "actor" or "musician"). Figure 2E The example in shows only "Bill Gates" and "Paul Allen" among the tracked entities from history 202, but entities from history 212 may include other entities, such as a "Microsoft" company entity. However, searching for entities to resolve the "him" coreference from unresolved user utterance 216E may include searching for a subset of entities that match the "human" type such that the search is effectively limited to only consider related entities such as Bill Gates and Paul Allen and not consider entities with unmatched types, such as "Microsoft".

[0051] In some examples, candidate entities are resolved from user utterances (which may be original user utterances or rewritten user utterances) from a computer-accessible conversation history. Figure 2E , "he" from the unresolved user utterance 216E may be resolved to "Bill Gates" mentioned in the (rewritten) user utterance 218E from the computer-accessible conversation history 202E. In some examples, the candidate entity is resolved from one of the automated assistant utterances of the multi-turn conversation, e.g., Figure 2D , “first” from the unresolved user utterance 216D is resolved to “Bill Gates” from the agent utterance 206D.

[0052] In some examples, such as Figure 2C , the entity constraint of the ambiguous entity includes a partial name of the ambiguous entity, and thus, searching for a candidate entity in the set of previously resolved entities includes finding a subset of previously resolved entities having full names that are consistent with the partial names, where the candidate disambiguated entity is from the subset. Figure 2C , determining a higher confidence correspondence for “Bill Gates” can be based on the correspondence between the entity name “Bill Gates” and the “partial name = Bill” constraint and the correspondence between the “defined attribute = net worth” constraint and the “net worth” attribute of Bill Gates (e.g., and other attributes of Bill Gates not shown in historical entity 212C).

[0053] In some examples, such as Figure 2D , the ambiguous entity in the unresolved user utterance includes the ordinal reference to the ordered list of entities from the multi-turn dialogue, and wherein the candidate entity is the corresponding entity from the ordered list of entities. Figure 2D In addition to having a defined "net worth" attribute, the entity of interest from the unresolved utterance 216D is "first", which corresponds with high confidence to the first entry "Bill Gates" from the ordered list of "Bill Gates and Paul Allen".

[0054] In some examples, such as Figure 2E In , the ambiguous entity in the unresolved user utterance includes a pronoun, the entity constraint of the ambiguous entity includes a pronoun gender, and wherein searching the set of previously resolved entities for a candidate entity includes finding a subset of the previously resolved entities having a gender consistent with the pronoun gender. In some examples, in addition to considering pronoun gender, searching the set of previously resolved entities can be based on other pronoun features, such as plural, animality (e.g., pronouns used to refer to humans, such as 'he' or 'she', versus pronouns used to refer to objects, such as 'it' or places such as 'there'). Figure 2E, the entity constraints from the unresolved user utterance 216E include "gender = male" and "defined attributes = company founded". These constraints match both Bill Gates and Paul Allen (e.g., both are male, both have the "founder" type and the "company founded" attribute). However, "Bill Gates" was recently mentioned in the user utterance 210E. Therefore, the coreference resolution machine 102 can be configured to assess a relatively higher confidence that the user is more likely to ask a follow-up question about "Bill Gates" rather than a question about "Paul Allen". Therefore, "Bill Gates" has the highest confidence corresponding to the entity constraint of the ambiguous entity ("he") from the unresolved user utterance 216E.

[0055] In some examples, method 300 also includes using a predefined language model to identify a verb predicate derived from an unresolved user utterance, wherein the entity constraint of the ambiguous entity includes the verb predicate, and wherein searching for a candidate entity in a set of previously resolved entities includes finding a subset of previously resolved entities having a defined value for the verb predicate. For example, Figure 2E The unresolved user utterance 216E includes a verb predicate indicating that the entity of interest "founded" one or more companies. Thus, searching the set of previously resolved entities may include finding a subset of previously resolved entities having relevant attributes such as the "founded company" attribute.

[0056] In some examples, such as Figure 2F As shown in , the entity constraint of the ambiguous entity may include a dependency relationship between the ambiguous entity and the basic entity, and thus searching for candidate entities includes searching for the basic entity and automatically resolving the candidate entity using the dependency relationship between the ambiguous entity and the basic entity. For example, Figure 2F The unresolved user utterance 216F includes two distinct entities, "Bill" and "his wife". However, if the coreference "Bill" is resolved first, for example by looking up "Bill" in the knowledge base and finding the related entities, then the dependent entity "his wife" can be easily resolved by the downstream query answering machine. Therefore, the query can be resolved to the rewritten utterance 218F by replacing "Bill" with "Bill Gates", and the entity "his wife" does not need to be rewritten.

[0057] At 308, method 300 also includes rewriting the unresolved user utterance into a rewritten utterance that replaces the ambiguous entity with the candidate entity. Figure 2C-2F A rewritten user utterance 218 corresponding to the unresolved user utterance 216 is shown, wherein ambiguous entities are replaced by candidate entities selected from the computer-accessible conversation history.

[0058] In many cases, such as Figure 2C-2FAs shown in , the rewritten utterance can be completely resolved so that coreferences to entities are replaced with explicit references to the entities. Since the rewritten utterance is completely resolved, where any entity is mentioned in an unambiguous manner, the rewritten utterance can be processed by one or more query answering machines without access to context from a computer-accessible conversation history. Therefore, at 310, method 300 includes outputting the rewritten utterance to one or more query answering machines.

[0059] However, in many examples, even if (e.g., by the coreference resolution machine 102 of FIG. 1 , and / or according to Figure 3 After processing the coreference according to method 300 , the resulting rewritten query may not be a fully resolved query that can be processed by query answering machine 114 .

[0060] In some examples, even after all coreferences are resolved, the rewritten utterance may still require context from the computer-accessible conversation history 202 for query answering. Thus, in addition to sending a fully defined query to a query answering machine, different speech acts may be handled in a specialized manner depending on the speech act. For example, an utterance may be an entity suggestion speech act or an entity correction speech act, which is intended to change the conversation topic of a previously established intent to discuss a different entity, and thus the utterance may be handled by the entity suggestion / correction resolver machine 118 of FIG. 1 (hereinafter referred to as Figure 2G-2H and Figure 4 Similarly, an utterance may be an intent suggestion speech act that is intended to change the conversation topic to discuss a different intent with respect to a previously established entity, and thus the utterance may be processed by the intent suggestion resolver machine 120 of FIG. 1 (hereinafter referred to as Fig.2I and Figure 5 which is further described) process.

[0061] Figure 2G An example of an entity suggestion speech act in an unresolved user utterance 216G that has been pre-processed by the coreference resolution machine 102 and classified by the speech act classifier machine 110 as a "suggest entity" speech act is shown. In the entity suggestion speech act, the computer-accessible conversation history 202G has included a discussion of another entity with respect to the defined query intent. For example, the discussion may be about an attribute of interest of the other entity. Figure 2G, the computer-accessible conversation history 202G includes a previous discussion of Bill Gates' net worth, e.g., the conversation is related to a query intent defined by a "net worth" entity attribute about the entity "Bill Gates". Thus, the unresolved user utterance 216G establishes a change in topic to discuss the same query intent (e.g., defined by the "net worth" entity attribute) about a new entity "Jeff Bezos". The entity suggestion may include any appropriate indication of a change in topic, e.g., a natural language phrase such as "how" or "how about" or "about" may indicate an entity suggestion speech act, and thus such a natural language phrase may be detected by the speech act classifier machine 110, e.g., using a predefined language model as described above. If an entity suggestion speech act is detected, then the remainder of the user utterance may be evaluated to define a new entity that the user intends to discuss, i.e., "Jeff Bezos". Thereafter, based on the user's intent to discuss Jeff Bezos, the original user utterance 208G may be rewritten to incorporate the entity suggestion, e.g., by replacing "Bill Gates" with "Jeff Bezos" in the rewritten user utterance. Thus, after detecting the "suggest entity" speech act 220G, the unresolved user utterance 216G may be rewritten, for example, into a possible rewritten utterance 218G to arrive at a fully formed query.

[0062] Figure 2HAn example of an entity correction speech act in an unresolved user utterance 216H that has been preprocessed by the coreference resolution machine 102 and classified as a "correct entity" speech act by the speech act classifier machine 110 is shown. Although the entity 212H from history includes both Bill Gates and Paul Allen, in the depicted example, the user utterance 208H that mentions "his net worth" is resolved to refer to Bill Gates. For example, the coreference machine 102 can evaluate higher confidence for Bill Gates and the "net worth" attribute because Bill Gates is a living entity with a current "net worth" attribute, while Paul Allen is deceased and the "net worth" attribute may not be relevant. Therefore, the proxy utterance 210H is related to Bill Gates. However, the user may actually have been interested in Paul Allen's net worth, such as when he died. Therefore, the user can say "No, I mean Paul", which is detected as a "correct entity" speech act to change the topic to discuss Paul Allen instead. The coreference resolution machine 102 can be used to preprocess the unresolved user utterance to determine that "Paul" actually does refer to Paul Allen. Similar to entity suggestions, entity correction speech acts may be detected by the speech act classifier machine 110 (e.g., by detecting demonstrative words such as "I mean" or "no") in any appropriate manner. If a "correct entity" speech act is detected, then the remainder of the user utterance may be evaluated to define the entity that the user intended to discuss, namely "Paul Allen." Thereafter, based on detecting the "correct entity" speech act 220H and detecting the user's intent to discuss Paul Allen, the original user utterance 208H may be rewritten to incorporate entity suggestions, for example by replacing "Bill Gates" with "Paul Allen" in the rewritten user utterance. Although Figure 2G and 2H The entity suggestions / corrections in are based on finding entities based on partial or full name constraints, but the above Figure 2C-2F The coreference resolution techniques described with respect to method 300 may be similarly applied to searching for related entities in resolving entity suggestions / corrections.

[0063] therefore, Figure 4 A method 400 is shown for disambiguating a user query comprising a set of user utterances in a multi-turn conversation by resolving entity suggestions and / or corrections (e.g., in an entity suggestion / correction resolver machine 118 as shown in FIG. 1 ). The method 400 is described with respect to suggested entities, but the techniques described herein are similarly applicable to resolving corrected entities. The entity suggestion / correction machine 118 may be configured to implement the method 400.

[0064] At 402, method 400 includes using a predefined language model (e.g., any suitable predefined language model as described above) to identify suggested entities in unresolved user utterances from a multi-turn conversation. The suggested entities can be referred to in any suitable manner (e.g., via coreference). Detecting that an entity is being suggested can be based on any suitable features, such as based on detecting a relevant English phrase such as "how", detecting a relevant phrase in any other language, or in any other suitable manner.

[0065] At 404, method 400 includes using a predefined language model to identify entity constraints associated with the suggested entity. Entity constraints may be derived in any suitable manner from any term of the user utterance (e.g., a full or partial name, a coreference phrase referring to an entity, a verb describing an action performed by an entity, an adjective describing an attribute of an entity, etc.). For example, in Figure 2G , the suggested entity "Jeff Bezos" is referred to by the full name. Alternatively, "Jeff Bezos" can be referred to in any other way (e.g., "the founder of Amazon" or "Bezos"). The suggested entity can be resolved in any appropriate way (e.g., using the coreference resolution method described herein).

[0066] At 406, method 400 includes searching in a computer-accessible conversation history of a multi-turn dialogue a set of previously resolved entities for a candidate entity containing an entity attribute corresponding to the highest confidence of an entity constraint of a proposed entity, a candidate utterance including the candidate entity, and a candidate intent associated with the candidate entity. A candidate utterance may be any utterance that mentions a candidate entity, such as a user utterance or an agent utterance. Similar to resolving coreference, the entity constraint of a proposed entity may include a partial name of the proposed entity, and thus searching for a candidate entity in a set of previously resolved entities includes finding a subset of previously resolved entities with full names consistent with the partial name, wherein the candidate entity is among the subset. In other examples, an ambiguous entity in an unresolved user utterance may include an ordinal reference to an ordered list of entities from a multi-turn dialogue, and thus the candidate entity may be identified as a corresponding entry from the ordered list of entities. In some examples, the entity constraint of a proposed entity includes a dependency relationship between the proposed entity and a base entity, and thus searching for a candidate entity includes searching for the base entity, and automatically resolving the candidate entity using the dependency relationship between the proposed entity and the base entity. For example, a user utterance including an entity suggestion may be in any of the following forms: "How is she?" "How is the third one?" "How is his wife?".

[0067] The candidate intent can be an intent from a candidate utterance or any other query intent discussed about a candidate entity (e.g., an entity attribute of a candidate entity). In many examples, the candidate intent and the candidate entity appear in the same user utterance of a set of user utterances. For example, in Figure 2G , the candidate entity is Bill Gates, and the candidate intent is the "net worth" previously described with respect to Bill Gates. Searching for candidate entities based on entity constraints / entity attributes can be performed in any suitable manner, such as described above with respect to coreference resolution. For example, Jeff Bezos has many entity constraints that can be determined based on looking up Jeff Bezos in a knowledge base or web search. As depicted in this example, the entity constraints for Jeff Bezos closely correspond to the entity attributes of Bill Gates, for example, the two entities can have a defined "founded company" attribute and a "net worth" attribute, as well as similar entity types (e.g., "human being", "founder").

[0068] At 408, method 400 includes rewriting the candidate utterance into a rewritten utterance that includes the candidate intent and replaces the candidate entity with the suggested entity. Figure 2G , a possible rewritten utterance 218G is a user utterance 208G in which the candidate entity "Bill Gates" is replaced with the suggested entity "Jeff Bezos".

[0069] At 410, method 400 includes outputting the rewritten utterance to one or more query answering machines. As with coreference resolution, after resolving the suggested / corrected entities, the rewritten utterance may be suitable for downstream processing by a query answering machine without any further context from computer-accessible conversation history 202. For example, in Figure 2G In, the rewritten utterance 218G is a fully defined query about Jeff Bezos' net worth.

[0070] In some examples, method 400 also includes using a predefined language model to identify one or more entity attributes of a suggested entity in the unresolved user utterance, wherein the entity constraint associated with the suggested entity includes the one or more entity attributes of the suggested entity. Figure 2GIn the example shown in , in addition to determining the full name Jeff Bezos, the predefined language model can be operated to find attributes / types for Jeff Bezos, for example, to determine that Jeff Bezos has the types "person" and "founder" and has the gender "male". Alternatively, if instead of referring to "Jeff Bezos", the user has referred to "the founder of Amazon" in the unresolved user utterance 216G, then the predefined language model can be operated to find attributes / types for "the founder of Amazon", such as the "founded company = Amazon" attribute, the "person" type and / or the founder type. In some examples, method 400 also includes using the predefined language model to identify verb predicates in the unresolved user utterance, for example, "founder" implies the "founded company" predicate. Therefore, the entity constraint associated with the suggested entity includes a verb predicate, and wherein searching for a candidate entity in the set of previously resolved entities includes finding a subset of previously resolved entities that have defined values ​​for entity attributes that indicate a verb predicate. The relationship between predicate phrases, entity attributes, entity types, and entity constraints can be determined in any appropriate manner, for example, based on example data obtained from a human user. In some examples, the relationship between predicate phrases, entity attributes, entity types, and / or entity constraints can be represented as a computer-accessible dictionary, database, and / or knowledge base, including any appropriate mapping between relevant predicate phrases, entity attributes, entity types, and / or entity constraints. In some examples, such relationships can be determined by a machine learning function. For example, a machine learning function can be configured to evaluate a confidence value for a particular relationship. Alternatively or additionally, a machine learning function can be configured to receive a predicate phrase, an entity attribute, an entity type, or an entity constraint, and output one or more relevant predicate phrases, entity attributes, entity types, and / or entity constraints.

[0071] In some examples, a user may suggest talking about time and location by implicitly mentioning the current time and / or location. For example, the method of the present disclosure may infer that "What's the weather like?" is likely to ask about the weather at the current time and location. Therefore, if a user in Portland, Oregon asks "What's the weather like?" at 4:00 p.m. on January 1, 2020, the query may be rewritten by adding the current time and location to the query, as in "What's the weather like in Portland, Oregon at 4:00 p.m. on January 1, 2020?". However, in some examples, the method of the present disclosure may be configured to infer that different times / locations are associated with relatively high confidences, for example, if a particular time / location is mentioned in a previous utterance in the conversation history. For example, if a user asks "Is there a show at the Oregon Zoo tomorrow?" and then asks "What's the weather like?", the method of the present disclosure may assess a relatively high confidence that the time is "tomorrow" and the location is "Oregon Zoo" rather than the current time and the current location.

[0072] Fig.2I An example of a "suggestion intent" speech act in an unresolved user utterance 216I that has been pre-processed by the coreference resolution machine 102 and classified as a "suggestion intent" speech act by the speech act classifier machine 110 is shown. The computer-accessible conversation history 202I includes a discussion of Bill Gates' net worth. The unresolved user utterance 216I is a "suggestion intent" speech act in which the user establishes a new query intent to discuss about the same entity (Bill Gates). As with the suggestion / correction entity, the "suggestion intent" speech act can be detected by the speech act classifier machine 110 (e.g., as indicated by the word "how" from the unresolved user utterance, or as indicated by any other indicator word / phrase such as "how" or "about") in any appropriate manner. If the "suggestion intent" speech act is detected, then the remainder of the user utterance can be evaluated to define the user intent about the new topic (i.e., "annual income"). Thereafter, based on the user's intent to discuss "annual income", the original user utterance 208I may be rewritten to incorporate the intent suggestion, for example by replacing "net worth" with "annual income" in the rewritten utterance 218I. Note that the unresolved user utterance 216G "What about annual income?" does not mention any entity and is therefore not a fully formed query that can be answered without a first rewrite that mentions a related entity.

[0073] therefore, Figure 5 A method 500 is shown for disambiguating a user query comprising a set of user utterances in a multi-turn conversation by resolving entity suggestions and / or corrections. The entity intent resolver machine 120 may be configured to implement the method 500 .

[0074] At 502, method 500 includes using a predefined language model to identify a suggested intent in an unresolved user utterance from a multi-turn conversation. As with the detection of suggested entities, any suitable predefined language model can be used to detect the suggested intent based on any suitable criteria, such as by detecting phrases such as "how?" or "how about?"

[0075] At 504, method 500 includes identifying entity constraints associated with the suggested intent using a predefined language model. For example, the suggested intent can be defined with respect to entity attributes, e.g., Fig.2I , the intention is to inquire about the "annual income" entity attribute. Therefore, the entity constraints from the unresolved user utterance 216I may be related to the same entity attribute (eg, the constraint "defined attribute = annual income").

[0076] At 506, method 500 includes searching, in the computer-accessible conversation history of the multi-turn dialog, a set of previously resolved entities for a candidate entity that contains an entity attribute that corresponds to the entity constraint of the suggested intent with the highest confidence. Figure 3 ) to determine the corresponding confidence. Fig.2I In , Bill Gates has the "annual income" attribute defined by the intent, while Paul Allen (who is deceased) does not have this attribute. Therefore, Bill Gates has a higher confidence correspondence with the entity constraint of the proposed intent.

[0077] At 508, method 500 includes rewriting the unresolved user utterance into a rewritten utterance that includes the suggested intent and candidate entities. Fig.2I , the rewritten utterance 218I includes the suggested intent "annual income" and the candidate entity "Bill Gates".

[0078] At 510, method 500 includes outputting the rewritten utterance to one or more query answering machines. As with coreference resolution, after resolving the suggested intent, the rewritten utterance may be suitable for downstream processing by a query answering machine without any further context from computer-accessible conversation history 202. For example, in Fig.2I In , the rewritten utterance 218I is a fully defined query about the annual income of Bill Gates. In some examples, the candidate entity is previously resolved from one of the automated assistant utterances in the computer-accessible conversation history 202I, such as in Fig.2I , Bill Gates is first mentioned in agent utterance 206I. Alternatively or additionally, the candidate entity may be resolved from the user utterance, for example, an earlier user utterance may explicitly mention Bill Gates.

[0079] In some examples, identifying a suggested intent also includes identifying a predicate rewriting pattern associated with the suggested intent. For example, a predefined language model may be operable to determine one or more relevant predicate rewriting patterns for a given suggested intent, wherein rewriting an unresolved user utterance into a rewritten utterance includes applying the predicate rewriting pattern to a candidate entity. For example, a predefined language model may associate multiple predefined intents to a predicate rewriting pattern. In some examples, a predefined language model may include a computer-accessible dictionary, a database, or a knowledge base including multiple different predefined intents and one or more relevant predicate rewriting patterns for each predefined intent. In some examples, a predefined language model may include a machine learning function configured to receive a predefined intent and output a predicate rewriting pattern. As a specific example, for the "annual income" intent, a predefined language model may include a predefined pattern: "What is the annual income of <candidate entity>", where <candidate entity> can be replaced with a suitable candidate entity. This exemplary predicate rewriting pattern will result in Fig.2I The rewritten utterance 218I shown in FIG.

[0080] If the predefined learning model fails to find a predicate rewriting pattern for a given intent, the query can be rewritten according to general predefined rules. Non-limiting examples of general rules for rewriting unresolved user utterances include using a predefined concatenation string or replacing parts of a template string as defined herein. In some examples, the suggested intent includes an entity attribute of interest, and the entity constraint includes a defined value for the entity attribute of interest. Fig.2I , the suggested intent includes the attribute of interest "annual income", and Bill Gates has a defined value for the attribute. Thus, in some examples, rewriting the unresolved user utterance into a rewritten utterance includes concatenating the entity attribute of interest, the predefined connection string, and the candidate entity. For example, the predefined connection string can be "is", "of", or a space (""). Thus, Fig.2IThe rewritten utterance 218I in can alternatively be rewritten as, for example, "Bill Gates' annual income". Rules based on replacing portions of a template string can be defined in any appropriate manner for a variety of natural languages. In English, if the predicate phrase is a noun phrase, then a non-limiting example of a template string is "<predicate phrase> of <candidate entity>", where the bracketed portions are specified to be replaced with the candidate entity and predicate phrase, respectively. As another non-limiting example, the predicate phrase is a phrase of the form "<adjective> how" for a certain adjective of interest, after which the query can be the rewritten "<candidate entity> <adjective> how". Otherwise, more generally, the intent suggestion resolver machine 120 is configured to rewrite queries by replacing portions of the template string "<predicate phrase> <object entity>". It is believed that, similarly, general rules can be implemented for any natural language to facilitate rewriting of unresolved user utterances to include suggested intents and candidate entities.

[0081] In some examples, coreference resolution, intent suggestion resolution, and / or entity suggestion / correction resolution may not successfully resolve every possible ambiguity from an unresolved user utterance. For example, there may be one or more coreferences that cannot be resolved into a single unambiguous entity, either because the search for matching entities yielded zero entities with sufficient relevance (e.g., zero entities with confidence values ​​above a predefined threshold), or because the search yielded two or more matching entities with similar relevance (e.g., based on the same or similar confidence values). Alternatively, a suggested intent or entity may not be resolvable based on a computer-accessible conversation history, such as if the suggested intent is mentioned but the suggested intent is not relevant to any entity from the history, or if the suggested entity is mentioned but is not relevant to any intent from the history. Therefore, in order to reply to a user utterance, it may be necessary to obtain additional information about what entities / intents are relevant.

[0082] Thus, there may be inherent ambiguities that remain in the rewritten utterance that cannot be resolved even using the context provided by the computer-accessible conversation history. Thus, even given the rewritten utterance, the speech act classifier machine 110 may not be able to classify the rewritten utterance as a "fully resolved question", "greeting", "confirmation", "rejection", "suggesting entity", "correcting entity", or "suggesting intent" speech act. Therefore, to resolve the inherent ambiguity, the utterance may be processed by a clarification technique machine 116 configured to detect ambiguities and output clarifying questions to obtain clarifying responses from the user. Reference will be made below to Figure 2J-2M6 further describes the clarification technology machine 116. In addition, the user utterance can be a clarifying reply provided by the user in response to a clarifying question. Based on the clarifying reply appearing after the clarifying question, and / or based on the words appearing in the clarifying reply (e.g., "I mean"), the speech act classifier machine 110 can classify the clarifying reply as a "clarifying" speech act and delegate the clarifying reply to be processed by the clarification resolution machine 122 configured to disambiguate (multiple) previous user utterances based on the clarifying reply.

[0083] Figure 2J An example of an ambiguous user utterance detected as an ambiguity that cannot be fully resolved via coreference resolution, entity suggestion / correction resolution, and / or intent suggestion resolution is shown. Unresolved user utterance 216J uses the pronoun "he" and is associated with the entity that founded the company. However, these attributes can match any of the entities 212J from history, namely Bill Gates or Paul Allen. Since "he" cannot be resolved into a single unambiguous entity, and since there are no additional attributes that more closely correspond to Bill Gates or Paul Allen, the detected speech act 220J is "ambiguous".

[0084] Figure 2K An example of a clarifying question output in agent utterance 210K and a corresponding clarifying reply in unresolved user utterance 214K is shown. The speech act classifier machine 110 can be programmed and / or trained to detect clarifying speech acts, which can be used to disambiguate a previously ambiguous user utterance 208K. For example, the speech act classifier machine can be trained to detect clarifying speech acts based on words in a clarifying reply (e.g., "I mean") and / or based on having previously output a clarifying question. Return to Figure 2K In the example in , in the unresolved user utterance 214K, the user clarifies that they mean "the first one", which can be resolved (e.g., by the coreference resolution methods described herein) to refer to "Bill Gates" from the ordered list including "Bill Gates and Paul Allen". Any other suitable reference can be used for clarification and can be resolved to the entity from history 212K using any of the methods described herein, for example, a partial name such as "Bill" can be resolved using the coreference resolution methods described herein. After resolving the disambiguated entities, the original ambiguous user utterance (e.g., Figure 2K 208K) can be rewritten to replace the ambiguous entity (e.g., "he") with the disambiguating entity (e.g., "Bill Gates"), as in possible rewritten utterance 218K.

[0085] Figure 2JAnother example of an ambiguous user utterance detected as an ambiguity that cannot be fully resolved via coreference resolution, entity suggestion / correction resolution, and / or intent suggestion resolution is shown. Unresolved user utterance 216J uses the pronoun "she" and is associated with an entity that founded a company. However, the "female" gender of the "she" pronoun does not match the gender attribute for any of the entities 212J (i.e., Bill Gates or Paul Allen) from the conversation history 202J. Since "she" cannot be resolved into a single, unambiguous entity, the detected speech act 220J is "ambiguous." Although the user is interested in asking about companies founded by women, there is no such entity 212J in the history, so the question is ambiguous because the conversation history 202J does not include enough information to resolve the ambiguity.

[0086] Figure 2M An example of a clarifying question output in agent utterance 210M and a corresponding clarifying reply in unresolved user utterance 216M is shown. The clarifying reply is detected as a clarifying speech act, which can be used to disambiguate a previously ambiguous user utterance 208M. The user clarifies that they actually wanted to ask "Melinda Gates," even though she has not yet been tracked in entity 212M from history. As with the other examples, the clarifying entity can be resolved using any of the techniques described herein (e.g., via co-reference). For example, instead of stating "Oh, I mean Melinda Gates," the user can alternatively state "I mean Bill's wife" or "I mean Melinda," and based on the context already in the computer-accessible conversation history 202, these specifications can be resolved to "Melinda Gates," e.g., based on the established relationship between the Bill Gates and Melinda Gates entities. After resolving the disambiguating entity, the original ambiguous user utterance (e.g., Figure 2M 208M) can be rewritten to replace the ambiguous entity (e.g., "she") with the disambiguating entity (e.g., "Melinda Gates"), as in possible rewritten utterance 218M.

[0087] therefore, Figure 6 A method 600 is shown for disambiguating a user query including a set of user utterances in a multi-turn conversation by obtaining additional information from the user using clarifying questions. The clarification technique machine 116 and / or the clarification resolution machine 122 may be configured to implement the method 600 .

[0088] At 602, method 600 includes using a predefined language model to identify ambiguous entities in unresolved user utterances from a multi-turn conversation. For example, ambiguous entities can be defined by a coreference that is ambiguously resolved to two or more entities, a coreference that cannot be resolved to any entity, or a suggested intent that cannot be resolved to refer to any entity, or a suggested intent that may refer to two or more different entities.

[0089] At 604, method 600 includes attempting to resolve ambiguous entities by analyzing a computer-accessible conversation history of the multi-turn dialog. The analysis may include coreference resolution, suggested / corrected entity resolution, suggested intent resolution, or any other (multiple) techniques described herein. For example, the analysis may include identifying an initial unresolved entity in an unresolved user utterance using a predefined language model, identifying an entity constraint of the initial unresolved entity using the predefined language model, and searching a set of previously resolved entities in the computer-accessible conversation history of the multi-turn dialog for a resolved entity having an entity attribute corresponding to the entity constraint of the initial unresolved entity.

[0090] If the resolved entity can be found in the set of previously resolved entities, then the ambiguous entity can be disambiguated by analysis. Accordingly, the user utterance can be rewritten in a disambiguated form, such as described above with respect to coreference resolution, suggested / corrected entity resolution, and / or suggested intent resolution. Accordingly, no further clarification is required and at 614, the rewritten user utterance can be output to one or more query answering machines.

[0091] However, in some cases, such as Figure 2J and 2L In the example above, the resolved entity is not included in the set of previously resolved entities. Therefore, the ambiguous entity cannot be disambiguated by analysis (e.g., because there are zero candidate entities from the history that are suitable for disambiguation, or because there are two or more equally likely candidate entities).

[0092] In response to failing to disambiguate the ambiguous entity through the analysis, the method 600 further includes outputting a clarifying question about the ambiguous entity at 606. The method 600 further includes receiving a clarifying user utterance at 608. Thus, at 610, the method 600 further includes identifying the disambiguated entity in the clarifying user utterance using any of the methods of the present disclosure, such as by operating a predefined language model to identify the entity mentioned in the clarifying user utterance, and using the coreference machine 102 to resolve the entity with respect to the original ambiguous user utterance. For example, Figure 2K and 2M Exemplary clarifying questions and clarifying responses are shown.

[0093] In some examples, attempting to resolve an ambiguous entity includes determining that the ambiguous entity is an ambiguous reference to two or more entities, where the clarifying question is a request to select one of the two or more entities. For example, the clarifying question may include an ordered list of two or more entities, and thus identifying the disambiguating entity in the clarifying user utterance may include identifying the disambiguating entity by an ordinal number of the reference to the ordered list of two or more entities. Figure 2K As shown in , there are two different entities that can match the ambiguous entity from the user utterance 208K, namely Bill Gates and Paul Allen. Therefore, the clarifying question mentions these options to help the user disambiguate: "Who else founded what company? Bill Gates or Paul Allen?". In addition, the clarifying reply re-mentions these options by clarifying that the user means "the first" (i.e., Bill Gates). Alternatively or additionally, identifying the disambiguating entity in the clarifying user utterance may include identifying the disambiguating entity by including a partial entity name in the clarifying user utterance, for example, the clarifying user utterance may have stated "Bill" or "Gates" instead of "the first."

[0094] In some examples, attempting to resolve the ambiguous entity includes determining that the ambiguous entity cannot be resolved into any entity, and thus the clarifying question is a request for the user to explicitly indicate the disambiguated entity. Figure 2M , due to the mismatch between the gender of the ambiguous entity and the tracked entity 212M from history, there is no entity 212M from history that can match the ambiguous entity from user utterance 208M. Therefore, the clarification question does not list the options but instead simply asks the user to clarify: "Who founded what other company?". Therefore, the user's clarification response specifies a specific person "Melinda Gates", although as described above, the user's clarification response may instead specify "Melinda Gates" via coreference or any other appropriate description (e.g., "Bill Gates' wife"). In any case, the user's clarification response is resolved to determine the disambiguated entity, i.e. Figure 2K In the example of "Bill Gates" and Figure 2M "Melinda Gates" in the example.

[0095] At 612, method 600 rewrites the unresolved user utterance into a rewritten utterance that replaces the ambiguous entity with the disambiguated entity. The rewritten utterance is disambiguated to explicitly mention the disambiguated entity so that it can be processed by downstream query answering machines without any further need for disambiguation or analysis of computer-accessible conversation history 202. Thus, at 614, method 600 includes outputting the rewritten utterance to one or more query answering machines.

[0096] In some examples, method 600 includes using a predefined language model to identify entity constraints for an ambiguous entity and entity attributes for a disambiguating entity. Thus, identifying a disambiguating entity in a clarifying user utterance includes determining whether the entity constraints correspond to entity attributes with sufficient confidence. In some examples, the disambiguating entity is one of a plurality of candidate entities and the sufficient confidence is the highest confidence among the plurality of candidate entities. In some examples, the sufficient confidence is any confidence value that exceeds a predefined threshold confidence. The disambiguating entity may be identified in any suitable manner (e.g., as described above with reference to Figure 3 as described) to determine the corresponding confidence.

[0097] Outputting a clarification question at 606 can be performed in any suitable manner. In some examples, method 600 includes determining a predicate clause of an unresolved user utterance using a predefined language model. The predicate clause can be any relevant phrase from the unresolved user utterance, such as related to an entity attribute of interest or any other query intent. Thus, the predicate clause can indicate what aspect of the unresolved user utterance requires clarification. For example, if a user asks "What else did she sing?" without mentioning any specific singer, then "singer" can be used as a predicate clause describing the ambiguous entity. Thus, the clarification question can mention the predicate clause, such as "Who else sang what?" or "Which singer also sang what?". In some examples, a predefined language model can be used to identify entity attributes of an ambiguous entity, where the clarification question mentions the entity attribute. For example, in Figure 2K In , the user is interested in the attribute "company founded" about the ambiguous entity. Therefore, the clarification question is asked by asking "Who founded what other companies? Bill Gates or Paul Allen?".

[0098] In some examples, outputting a clarifying question includes generating a parse graph for an unresolved user utterance and selecting a connected subgraph of the parse graph. In some examples, the connected subgraph of the parse graph includes a root node of the parse graph. For example, the parse graph can be a dependency parse graph that identifies clause relationships between root words of an unresolved user utterance and clauses in the utterance. Thus, selecting an appropriate connected subgraph can include finding “question words” from the unresolved user utterance, such as “who,” “what,” “where,” “is,” etc., where the question words are closest to the ambiguous coreference in the unresolved user utterance. For example, in Figure 2M In , the ambiguous coreference is "she" and the related interrogative word is "what". Outputting clarification questions can also include using another interrogative word (e.g., Figure 2M The ) alternatives shown in have ambiguous coreference.

[0099] In some specific human languages, specific coreference words may appear in different parts of speech. For example, in English, "her" may appear as part of a noun and may be replaced by the interrogative word "whose" for question generation, e.g., "whose child is her?" may be rewritten as "whose child is who?" Alternatively, "her" may appear as a pronoun, as in "when did Tom Hanks marry her" (which may be rewritten as "when did Tom Hanks marry who").

[0100] In some examples, the ambiguous entity is an ambiguous coreference. Thus, the clarifying question may include a rewritten phrase from an unresolved user utterance that replaces the ambiguous coreference with a question word associated with the ambiguous coreference, so as to indicate what coreference needs to be clarified in the user's clarifying response. For example, in Figure 2M , the user utterance 208M includes the ambiguous coreference "she" and the proxy utterance 210M asks "Who founded what other companies?" The proxy utterance 210M forms a clarifying question by replacing the ambiguous coreference "she" from the user utterance 208M with the relevant interrogative word "who".

[0101] In some examples, the clarifying reply may fail to adequately disambiguate the original, unresolved user utterance. For example, an entity mentioned in the clarifying reply may not correspond with sufficient confidence to an ambiguous coreference or suggestion from the unresolved user utterance. Therefore, disambiguating using the clarification resolver 122 and / or the clarification technique 116 may include making multiple attempts to disambiguate. Therefore, in response to an entity constraint failing to correspond with at least sufficient confidence to an entity attribute, the clarification technique 116 may be configured to output additional clarifying questions and receive additional clarifying user utterances. In this way, many clarifying questions may be output before the original, unresolved user utterance is ultimately disambiguated.

[0102] The methods and processes described herein may be associated with a computing system of one or more computing devices. Specifically, such methods and processes may be implemented as an executable computer application, a network accessible computing service, an application programming interface (API), a library, or a combination of the above and / or other computing resources.

[0103] Figure 7 Schematically illustrated is a simplified representation of a computing system 700 configured to provide any to all of the computing functionality described herein. The computing system 700 may take the form of one or more personal computers, computer-accessible service computers, tablet computers, home entertainment computers, gaming devices, mobile computing devices, mobile communication devices (e.g., smart phones), virtual / augmented / mixed reality computing devices, wearable computing devices, Internet of Things (IoT) devices, embedded computing devices, personal assistant computing devices, and / or other computing devices.

[0104] The computing system 700 includes a logic subsystem 702 and a storage subsystem 704. The computing system 700 may optionally include an input / output (I / O) subsystem 708, a communication subsystem 706, and / or Figure 7 In some examples, the I / O subsystem 708 includes a microphone subsystem 714 configured to receive user speech from a user in the form of audible speech, a keyboard subsystem 716 configured to receive user speech from a user in the form of text, a display subsystem 710 configured to output agent speech in the form of text, and / or a speaker subsystem 712 configured to output agent speech in the form of audible speech.

[0105] The logic subsystem 702 includes one or more physical devices configured to run instructions. For example, the logic subsystem may be configured to run instructions that are part of one or more applications, services, or other logical structures. The logic subsystem may include one or more hardware processors configured to run software instructions. Additionally or alternatively, the logic subsystem may include one or more hardware or firmware devices configured to run hardware or firmware instructions. The processor of the logic subsystem may be single-core or multi-core, and the instructions run thereon may be configured for sequential, parallel, and / or distributed processing. The individual components of the logic subsystem may optionally be distributed among two or more separate devices, which may be remotely located and / or configured for collaborative processing. Various aspects of the logic subsystem may be virtualized and run by a remotely accessible networked computing device configured in a cloud computing configuration.

[0106] The storage subsystem 704 includes one or more physical devices configured to temporarily and / or temporarily hold computer information such as data and instructions that can be run by the logic subsystem. When the storage subsystem includes two or more devices, these devices can be located in parallel and / or remotely. The storage subsystem 704 can include volatile, non-volatile, dynamic, static, read / write, read-only, random access, sequential access, location addressable, file addressable and / or content addressable devices. The storage subsystem 704 can include removable and / or built-in devices. When the logic subsystem runs instructions, the state of the storage subsystem 704 can be transformed, for example to hold different data.

[0107] Aspects of logic subsystem 702 and storage subsystem 704 may be integrated together into one or more hardware logic components. For example, such hardware logic components may include program and application specific integrated circuits (PASIC / ASIC), program and application specific standard products (PSSP / ASSP), systems on chips (SOC), and complex programmable logic devices (CPLD).

[0108] The logic subsystem and the storage subsystem may collaborate to formulate the data flow architecture 100 of FIG. 1 and / or instantiate one or more logic machines (e.g., the coreference resolution machine 102, the entity suggestion / correction resolver machine 118, the clarification resolver machine 122, the greeting / confirmation / rejection reply machine 112, the query answering machine 114, and the clarification technique machine 116) and / or instantiate a computer-accessible conversation history 202.

[0109] As used herein, the term "machine" is used to collectively refer to a combination of hardware, firmware, software, instructions, and / or any other components that collaborate to provide computer functionality. In other words, a "machine" is by no means an abstract concept and always has a tangible shape. A machine may be instantiated by a single computing device, or a machine may include two or more subcomponents instantiated by two or more different computing devices. In some embodiments, a machine includes a local component (e.g., a software application run by a computer processor) that collaborates with a remote component (e.g., a cloud computing service provided by a network of server computers). Software and / or other instructions given for a particular machine may optionally be saved as one or more non-executing modules on one or more appropriate storage devices.

[0110] The machine may be implemented using any suitable combination of existing and / or future machine learning (ML), artificial intelligence (AL), and / or natural language processing (NLP) techniques and hard-coded or other programmer-defined computing functions. It should be understood that the above referenced functions may be programmatically executed by a corresponding machine and / or other computing entity of the computing system 700. Figure 2A-2M All of the decision making, reasoning, detection, and other logic described.

[0111] Non-limiting examples of techniques that may be incorporated into embodiments of one or more machines include support vector machines, multi-layer neural networks, convolutional neural networks (e.g., including spatial convolutional networks for processing images and / or videos, temporal convolutional networks for processing audio signals and / or natural language sentences, and / or any other suitable convolutional neural network configured to convolve and pool features across one or more temporal and / or spatial dimensions), recurrent neural networks (e.g., long short-term memory networks), associative memories (e.g., lookup tables, hash tables, Bloom filters, neural Turing machines, and / or neural random access memories), word embedding models (e.g., GloVe or Word2Vec), unsupervised spatial and / or clustering methods (e.g., nearest neighbor algorithms, topological data analysis, and / or k-means clustering), graphical models (e.g., (hidden) Markov models, Markov random fields, (hidden) conditional random fields, and / or AI knowledge bases), and / or natural language processing techniques (e.g., tokenization, stemming, component and / or dependency parsing, and / or intent identification, segmental models, and / or suprasegmental models (e.g., hidden dynamic models)).

[0112] In some examples, one or more differentiable functions can be used to implement the methods and processes described herein, where the gradient of the differentiable function can be calculated and / or estimated with reference to the input and / or output of the differentiable function (e.g., with reference to training data and / or with reference to an objective function). Such methods and processes can be determined at least in part by a trainable parameter set. Thus, a trainable parameter set for a particular method or process can be adjusted by any appropriate training process to continuously improve the functionality of the method or process.

[0113] Non-limiting examples of training processes for adjusting trainable parameters include supervised training (e.g., using gradient descent or any other appropriate optimization method), zero-shot, few-shot, unsupervised learning methods (e.g., classification based on classes derived from unsupervised clustering methods), enhanced learning (e.g., deep Q-learning based on feedback), and / or generative adversarial neural network training methods, belief propagation, RANSAC (random sample consistency), contextual bandit method, maximum likelihood method, and / or expectation maximization. In some examples, the various methods, processes, and / or components described herein can be trained simultaneously with respect to an objective function that measures the performance of the collective function of multiple components (e.g., with respect to enhanced feedback and / or with respect to labeled training data). Training multiple methods, processes, and / or components simultaneously can improve such collective functions. In some examples, one or more methods, processes, and / or components can be trained independently of other components (e.g., offline training on historical data).

[0114] A language model (e.g., any (multiple) predefined language models mentioned herein) can utilize vocabulary features to guide sampling / searching for words for identifying speech. For example, a language model can be defined at least in part by a statistical distribution of words or other vocabulary features. For example, a language model can be defined by a statistical distribution of n-grams, thereby defining transition probabilities between candidate words based on vocabulary statistics. The language model is also based on any other suitable statistical features and / or the results of processing statistical features using one or more machine learning and / or statistical algorithms (e.g., confidence values ​​resulting from such processing). In some examples, a statistical model can constrain what words can be recognized for an audio signal, such as based on an assumption that words in the audio signal are from a particular vocabulary.

[0115] Alternatively or additionally, the language model can be based on one or more neural networks previously trained to represent audio input and words in a shared latent space, such as a vector space learned by one or more audio and / or word models (e.g., wav2letter and / or word2vec). Thus, finding a candidate word can include searching the shared latent space based on a vector encoded by the audio model for the audio input to find a candidate word vector for decoding with the word model. The shared latent space can be used to evaluate, for one or more candidate words, the confidence that the candidate word appears in the speech audio.

[0116] The language model can be used in conjunction with an acoustic model, which is configured to evaluate the confidence of the candidate word in the speech audio included in the audio signal based on the acoustic features of the word (e.g., Mel-frequency cepstral coefficients, resonance peaks, etc.) for the candidate word and the audio signal. Optionally, in some examples, the language model can incorporate the acoustic model (e.g., the evaluation and / or training of the language model can be based on the acoustic model). The acoustic model defines a mapping between an acoustic signal and basic sound units (such as phonemes), for example, based on labeled speech audio. The acoustic model can be implemented based on any appropriate combination of existing and / or future machine learning (ML) and / or artificial intelligence (AL) models, such as: deep neural networks (e.g., long short-term memory, temporal convolutional neural networks, restricted Boltzmann machines, deep belief networks), hidden Markov models (HMMs), conditional random fields (CRFs) and / or Markov random fields, Gaussian mixture models, and / or other graphical models (e.g., deep Bayesian networks). The audio signal to be processed with the acoustic model can be preprocessed in any suitable manner, such as encoding at any suitable sampling rate, Fourier transform, bandpass filtering, etc. The acoustic model can be trained to identify the mapping between the acoustic signal and the sound unit based on training with labeled audio data. For example, the acoustic model can be trained based on labeled audio data including speech audio and correction text to learn the mapping between the speech audio signal and the sound unit indicated by the correction text. Therefore, the acoustic model can be continuously improved to improve its practicality for correctly identifying speech audio.

[0117] In some examples, in addition to statistical models, neural networks, and / or acoustic models, the language model can incorporate any suitable graphical model, such as a hidden Markov model (HMM) or a conditional random field (CRF). The graphical model can utilize statistical features (e.g., transition probabilities) and / or confidence values ​​to determine the probability of recognizing a word given the speech audio and / or other words recognized so far. Thus, the graphical model can utilize statistical features, previously trained machine learning models, and / or acoustic models to define transition probabilities between states represented in the graphical model.

[0118] When included, the display subsystem 710 can be used to present a visual representation of the data maintained by the storage subsystem 704. The visual representation can take the form of a graphical user interface (GUI). The display subsystem 710 can include one or more display devices utilizing nearly any type of technology. In some embodiments, the display subsystem may include one or more virtual reality, augmented reality, or mixed reality displays.

[0119] When included, the I / O subsystem 708 may include or interface one or more input devices, such as a microphone subsystem 714 and / or a keyboard subsystem 716. The input device may include a sensor device or a user input device. Examples of user input devices include a keyboard, a mouse, a touch screen, or a game controller. In some embodiments, the input subsystem may include or interface a selected natural user input (NUI) component. Such components may be integrated or peripheral, and the conversion and / or processing of input actions may be processed on-board or off-board. Exemplary NUI components may include microphones for voice and / or sound identification; infrared, color, stereoscopic vision, and / or depth cameras for machine vision and / or gesture identification; head trackers, eye trackers, accelerometers, and / or gyroscopes for motion detection and / or intent identification.

[0120] When included, the communication subsystem 706 can be configured to communicatively couple the computing system 700 with one or more other computing devices. The communication subsystem 706 can include wired and / or wireless communication devices compatible with one or more different communication protocols. The communication subsystem can be configured for communication via a personal network, a local area network, and / or a wide area network.

[0121] In an example, a method for disambiguating a user query comprising a set of user utterances in a multi-turn conversation comprises: using a predefined language model to identify an ambiguous entity in an unresolved user utterance from the multi-turn conversation; outputting a clarifying question about the ambiguous entity; receiving a clarifying user utterance; identifying the disambiguating entity in the clarifying user utterance; rewriting the unresolved user utterance into a rewritten utterance that replaces the ambiguous entity with the disambiguating entity; and outputting the rewritten utterance to one or more query answering machines. In this or any other example, attempting to resolve the ambiguous entity by analyzing a computer-accessible conversation history of the multi-turn conversation; and outputting the clarifying question about the ambiguous entity is in response to failing to resolve the ambiguous entity. In this or any other example, analyzing a computer-accessible conversation history of a multi-turn conversation using a predefined language model includes: identifying an initial unresolved entity in an unresolved user utterance using the predefined language model; identifying an entity constraint of the initial unresolved entity using the predefined language model; and searching a set of previously resolved entities for a resolved entity having an entity attribute corresponding to the entity constraint of the initial unresolved entity in the computer-accessible conversation history of the multi-turn conversation, and determining that the resolved entity is not included in the set of previously resolved entities. In this or any other example, attempting to resolve an ambiguous entity includes determining that the ambiguous entity is an ambiguous reference to two or more entities, wherein the clarifying question is a request to select one of the two or more entities. In this or any other example, the clarifying question includes an ordered list of two or more entities, and wherein identifying a disambiguated entity in the clarifying user utterance includes identifying the disambiguated entity by an ordinal reference to the ordered list of two or more entities. In this or any other example, attempting to resolve an ambiguous entity includes determining that the ambiguous entity cannot be resolved into any entity, and wherein the clarifying question is a request to the user to explicitly indicate the disambiguated entity. In this or any other example, identifying the disambiguated entity in the clarifying user utterance includes identifying the disambiguated entity by including a partial entity name in the clarifying user utterance. In this or any other example, the method also includes using a predefined language model to identify entity constraints of the ambiguous entity and entity attributes of the disambiguated entity, and wherein identifying the disambiguated entity in the clarifying user utterance includes determining whether the entity constraints correspond to the entity attributes with sufficient confidence. In this or any other example, the method also includes, in response to the entity constraints failing to correspond to the entity attributes with at least sufficient confidence, outputting additional clarifying questions and receiving additional clarifying user utterances. In this or any other example, the disambiguated entity is one of a plurality of candidate entities, and the sufficient confidence is the highest confidence among the plurality of candidate entities.In this or any other example, the method also includes determining a predicate clause of the unresolved user utterance using a predefined language model, wherein the clarifying question refers to the predicate clause.

[0122] In this or any other example, the method also includes determining entity attributes of the ambiguous entity using a predefined language model, wherein the clarifying question refers to the entity attributes. In this or any other example, the ambiguous entity is an ambiguous coreference, and wherein the clarifying question includes a rewritten phrase from an unresolved user utterance that replaces the ambiguous coreference with a question word associated with the ambiguous coreference. In this or any other example, outputting the clarifying question includes generating a parse graph for the unresolved user utterance and selecting a connected subgraph of the parse graph. In this or any other example, the connected subgraph of the parse graph includes a root node of the parse graph.

[0123] In an example, an automated assistant system includes: a microphone; a speaker; a logic subsystem; and a storage subsystem configured to retain instructions executable by the logic subsystem to: identify an ambiguous entity in an unresolved user utterance received via the microphone; output a clarifying question about the ambiguous entity via the speaker; identify the disambiguated entity in the clarifying user utterance received via the microphone; output a rewritten utterance that replaces the ambiguous entity with the disambiguated entity in the unresolved user utterance to one or more query answering machines; and output a reply returned by one of the query answering machines via the speaker. In this or any other example, the unresolved user utterance is included in a multi-turn conversation, and wherein identifying the ambiguous entity includes: using a computer-accessible conversation history of the multi-turn conversation, attempting to resolve the ambiguous entity by searching a set of previously resolved entities of the multi-turn conversation; and wherein outputting the clarifying question about the ambiguous entity is in response to failing to resolve the ambiguous entity.

[0124] In an example, a method for disambiguating a user query comprising a set of user utterances in a multi-turn dialogue comprises: using a predefined language model to identify ambiguous entities in unresolved user utterances from the multi-turn dialogue, attempting to resolve the ambiguous entity by analyzing a computer-accessible conversation history of the multi-turn dialogue; and in response to failing to resolve the ambiguous entity: outputting a clarifying question about the ambiguous entity; receiving a clarifying user utterance; identifying the disambiguating entity in the clarifying user utterance; rewriting the unresolved user utterance into a rewritten utterance that replaces the ambiguous entity with the disambiguating entity; and outputting the rewritten utterance to one or more query answering machines. In this or any other example, attempting to resolve the ambiguous entity is based on determining that the ambiguous entity is an ambiguous reference to two or more entities, wherein the clarifying question is a request to select one of the two or more entities. In this or any other example, attempting to resolve the ambiguous entity includes determining that the ambiguous entity cannot be resolved into any entity, and wherein the clarifying question is a request to the user to explicitly indicate the disambiguating entity.

[0125] The present disclosure is presented by way of example and with reference to the associated drawings. However, it will be noted that components, process steps, and other elements that may be substantially the same in one or more of the drawings are identified collaboratively and described with minimal repetitiveness. However, it will be noted that the collaboratively identified elements may also be different to some extent. It will also be noted that some of the drawings may be schematic and not drawn to scale. The various drawing scales, aspect ratios, and quantities of the components shown in the drawings may be intentionally distorted to make certain features or relationships easier to see.

[0126] It will be understood that the configuration and / or method described herein are exemplary in nature, and these specific embodiments or examples should not be considered to be restrictive, because many variations are possible. The specific routine or method described herein can represent one or more processing strategies in any number of processing strategies. Therefore, the various actions illustrated and / or described can be performed in the order illustrated and / or described, in other orders, in parallel, or omitted. Similarly, the order of the process described above can be changed.

Claims

1. A method performed by a digital assistant, the method comprising: receiving a first user utterance from a user; Based on the first user utterance, identifying an entity, wherein the entity has attributes assigned thereto, and further wherein the entity is stored in a conversation history between the user and the digital assistant; After receiving the first user utterance, receiving a second user utterance from the user; identifying, using a predefined language model, that the second user utterance refers to an ambiguous entity, wherein the ambiguous entity has the attribute assigned thereto; constructing a clarifying question based on the entity stored in the conversation history and the ambiguous entity both having the attribute assigned thereto, the clarifying question being configured to solicit a response from the user regarding the ambiguous entity; outputting the clarifying question to the user; receiving the response from the user, wherein the response includes the entity stored in the session history; rewriting the second user utterance into a rewritten utterance, wherein the rewritten user utterance includes the entity stored in the conversation history; and The rewritten utterance is output to one or more query answering machines.

2. The method of claim 1, wherein the conversation history includes a plurality of entities having the attribute assigned thereto, and further wherein the clarifying question is output based on the conversation history including a plurality of entities having the attribute assigned thereto. The method of claim 1 , wherein the ambiguous entity is a pronoun. 4 . The method of claim 1 , wherein the clarifying question is a request to select one of two or more entities identified from the conversation history.

5. The method of claim 4, wherein the clarifying question comprises an ordered list of the two or more entities, and wherein the response comprises an ordinal number that references the ordered list of the two or more entities.

6. The method of claim 2, wherein the clarifying question is a request for the user to explicitly indicate that the ambiguous entity refers to the entity stored in the conversation history.

7. The method of claim 1, wherein rewriting the second user utterance comprises replacing the ambiguous entity in the second user utterance with the entity stored in the conversation history.

8. The method of claim 1, further comprising determining a predicate clause of the second user utterance using the predefined language model, wherein the clarifying question refers to the predicate clause.

9. The method of claim 1, further comprising determining an entity attribute of the ambiguous entity using the predefined language model, wherein the clarification question refers to the entity attribute.

10. The method of claim 1, wherein the ambiguous entity is an ambiguous coreference, and wherein the clarifying question comprises a rewritten phrase from the second user utterance replacing the ambiguous coreference with a question word associated with the ambiguous coreference. 11 . The method of claim 1 , wherein outputting the clarifying question comprises generating a parse graph for the second user utterance and selecting a connected subgraph of the parse graph.

12. The method of claim 11, wherein the connected subgraph of the parse graph comprises a root node of the parse graph.

13. An automated assistant system, comprising: microphone; speaker; Logical subsystem; as well as A storage subsystem configured to retain instructions executable by the logic subsystem to: receiving a first user utterance from a user via the microphone; Based on the first user utterance, identifying an entity, wherein the entity has attributes assigned thereto, and further wherein the entity is stored in a conversation history between the user and a digital assistant; After receiving the first user utterance, receiving a second user utterance through the microphone, wherein the second user utterance includes an ambiguous entity; identifying that the second user utterance includes the ambiguous entity and that the ambiguous entity has the attribute assigned thereto; constructing a clarifying question based on the entity stored in the conversation history and the ambiguous entity both having the attribute assigned thereto, wherein the clarifying question is configured to solicit a response from the user regarding the ambiguous entity; outputting the clarifying question via the speaker; receiving the response from the user, wherein the response identifies the entity stored in the session history; outputting a rewrite of the second user utterance to one or more query answering machines, wherein the rewrite of the second user utterance includes the entity stored in the conversation history as a replacement for the ambiguous entity; and An answer returned by one of the query answering machines is output via the speaker, wherein the answer is based on the rewriting of the second user utterance. 14 . The automated assistant system of claim 13 , wherein the first user utterance and the second user utterance are included in a multi-turn dialog between the user and the automated assistant.

15. A method for semantically eliminating a user query in a multi-turn dialogue, wherein the multi-turn dialogue includes a number of user utterances formulated by a user to a digital assistant, the method comprising: receiving a first user utterance from the user, the first user utterance being included in the multi-turn conversation; identifying an entity in the first user utterance, wherein the entity has attributes assigned thereto, and further wherein the entity is stored in a conversation history for the multi-turn conversation between the user and the digital assistant; After receiving the first user utterance, receiving a second user utterance in the multi-turn dialogue, wherein the second user utterance includes an ambiguous entity; identifying, using a predefined language model, that the second user utterance includes an ambiguous entity and that the ambiguous entity has the attribute assigned thereto, searching the conversation history to identify the entity included in the first user utterance based on the attributes assigned to the ambiguous entity; constructing a clarifying question, wherein the clarifying question requests confirmation from the user that the ambiguous entity is the entity from the first user utterance; outputting the clarifying question to the user; receiving a third user utterance from the user, wherein the third user utterance confirms that the ambiguous entity is the entity from the first user utterance; rewriting the second user utterance to replace the ambiguous entity with an entity from the first user utterance; and The rewritten second user utterance is output to one or more query answering machines.

16. The method of claim 15, wherein the clarifying question requests a selection from the user between the entity from the first user utterance having the attribute and another entity elaborated by a user in the multi-turn conversation.

17. The method of claim 15, wherein the ambiguous entity is a pronoun and the attribute is gender.

18. The method according to claim 15, further comprising: receiving an answer from one or more query answering machines based on the rewritten second user utterance; as well as The answer is spoken to the user via a loudspeaker.

19. The method of claim 15, wherein the clarifying question comprises the attributes assigned to the entity and the ambiguous entity.

20. The method of claim 15, further comprising outputting a response to the first user utterance, wherein the second user utterance is received from the user immediately after the response is output to the first user.

Images