Generating model outputs using knowledge graphs

Abstract

Techniques for constraining the results of a generative language model as valid information using knowledge-based documents. The generative language model may generate invalid results, including composite entities and incorrect entity relationships. The techniques include, for a given user query, determining a set of document information corresponding to the user query from a particular knowledge base. The technique further includes determining a sub-graph from a knowledge graph representing the knowledge base, and determining a prefix tree data structure representation of the set of document information. The user query and the sub-graph are provided as input to a trained generative language model for generating a response to the user query. The technique includes using the prefix tree data structure to verify that the generated response corresponds to real information in the set of document information.

Description

[0001] Cross-referencing of relevant application data This application claims the benefit and priority of U.S. Patent Application No. 18 / 340,342, filed June 23, 2023, entitled “GENERATING MODEL OUTPUT USING A KNOWLEDGE GRAPH”, in the name of Mahdi Namazifar et al. The above patent application is incorporated herein by reference in its entirety. Background Technology

[0002] Speech recognition systems have evolved to the point where humans can use their voices to interact with computing devices. Such systems employ techniques to recognize words spoken by human users based on received audio input of varying quality. Speech recognition, combined with natural language understanding processing (NLP), enables voice-based user control of computing devices to perform tasks based on the user's verbal commands. Speech recognition and NLP techniques may be referred to collectively or separately as speech processing in this document. Speech processing may also involve converting a user's speech into text data, which can then be provided to various text-based software applications.

[0003] Voice processing can be used by computers, handheld devices, telephone computer systems, self-service terminals, and a wide variety of other devices to improve human-computer interaction. Attached Figure Description

[0004] To gain a more complete understanding of this disclosure, reference is now made to the following description taken in conjunction with the accompanying drawings.

[0005] Figure 1A It is a conceptual diagram of a knowledge graph-based generative model process with local prefix tree constraints, based on some implementation schemes.

[0006] Figure 1B This is a conceptual diagram illustrating a virtual assistant system for natural language processing and knowledge-based information generation according to an embodiment of the present disclosure.

[0007] Figure 2 A set of documents and a knowledge graph based on some embodiments of this disclosure are illustrated.

[0008] Figure 3A and Figure 3B This is a flowchart illustrating a process of using a pre-trained generative language model to perform knowledge retrieval via the generation of a knowledge subgraph, according to some embodiments of this disclosure.

[0009] Figure 4 This is a conceptual diagram of the components of a system according to an embodiment of this disclosure.

[0010] Figure 5This is a conceptual diagram illustrating how natural language processing is performed according to the implementation scheme of this disclosure.

[0011] Figure 6 This is a conceptual diagram illustrating how natural language processing is performed according to the implementation scheme of this disclosure.

[0012] Figure 7 This is a conceptual diagram of the dialog manager component.

[0013] Figure 8 This is a conceptual diagram of a text-to-speech component according to an embodiment of this disclosure.

[0014] Figure 9 It is a block diagram that conceptually illustrates example components of an apparatus according to an embodiment of the present disclosure.

[0015] Figure 10 It is a block diagram that conceptually exemplifies example components of a system according to an embodiment of the present disclosure.

[0016] Figure 11 Examples of computer networks used in conjunction with the overall system according to embodiments of this disclosure are illustrated. Detailed Implementation

[0017] Automatic speech recognition (ASR) is a field of computer science, artificial intelligence, and linguistics that involves transforming audio data associated with speech into text representing that speech. Similarly, natural language understanding (NLU) is a field of computer science, artificial intelligence, and linguistics that involves enabling computers to derive meaning from text input containing natural language. ASR and NLU are often used together as part of a speech processing system, sometimes referred to as spoken language understanding (SLU) systems. Natural language generation (NLG) involves enabling computers to generate output text or other data, such as sentences or phrases, in human-understandable words. Text-to-speech (TTS) is a field of computer science that involves transforming text and / or other data into audio data, which is then synthesized to resemble human speech. ASR, NLU, NLG, and TTS can be used together as part of a speech processing / virtual assistant system.

[0018] Users can converse with the virtual assistant system, such as by providing voice or text input and receiving synthesized voice or text responses from the system. In some cases, user input / utterances may include questions for the system to answer. That is, the user may request the system to provide some information in response to user input. The final command in this example is to provide the requested information. As part of the processing in response to user queries, the system may rely on a structured knowledge base to obtain or confirm the information requested by the query. A structured knowledge base is a repository of data that includes facts organized in a specific way and can be used to respond to user queries. Structured knowledge bases can present many types of information. For example, information may correspond to documents related to a specific topic (e.g., a repair manual for a [brand] car, a basketball article in a local newspaper, etc.), collections of many different topics (e.g., Wikipedia, research journal archives, books in a library, etc.), or other kinds of information.

[0019] In one example, a knowledge base may include a collection of tuples (related items) and may encode the data stored within the knowledge base with formal semantics (relationships between items). Knowledge bases or other organized information stores may store information in a "tuple" format, where entities are linked together in pairs, triples, etc., defining relationships between two things. For example, a tuple such as [John Doe, husband of, Jane Doe] describes John Doe as Jane Doe's husband. Many such tuples may exist in a knowledge base and may be used to answer incoming information queries. For example, the above tuples could be used to answer questions such as "Who is Jane Doe's husband?" or "Who is John Doe married to?".

[0020] In another form of structured knowledge base, information can be stored as a knowledge graph, such as a directed acyclic graph (DAG). This DAG plots nodes representing entities (e.g., people, places, things) connected by vertices or links, where each link corresponds to a relationship. To answer questions about a specific entity in the knowledge graph, the system can traverse a path from one entity to another along the knowledge graph and track information along the traversed links to obtain information about how one entity is related to another. For example, a DAG could include nodes for Jane Doe, John Doe, Jake Doe, etc. A link between Jane Doe and John Doe could be associated with various pieces of information such as "married in 2002" or "met in 2000." A link between John Doe and Jake Doe could be associated with information such as "brothers." Even if there is no direct link between Jane Doe's node and Jake Doe's node, the system may be able to determine the path between them and collect information along that path along the vertices (e.g., from Jane Doe to John Doe, and then from John Doe to Jake Doe) to obtain the information needed to answer the query "When did John Doe become Jane Doe's uncle?"

[0021] It should be understood that this document may refer to computational systems that use knowledge graphs, where the data corresponding to the knowledge graph may not actually include a visual graph, but rather data representing the graph. Such data can take many forms; for example, one set of data (e.g., for tables) may include entries representing entities (e.g., nodes), while another set of data (e.g., for tables) may include entries representing vertices or connections between nodes. A node's table entry may include a reference to a specific vertex pointing to the corresponding node (and / or other data) or derived from the corresponding node. Similarly, a vertex's table entry may include a reference to a specific node (and / or other data) connected to the vertex. In this way, the data of the knowledge base can be stored and made available to computational systems. As those skilled in the art will understand, the operations discussed herein can use such data to manipulate, traverse, and / or edit the knowledge graph.

[0022] A knowledge graph is a graph in which nodes and edges correspond to evidenced world meanings such as those from books, news articles, or research papers. Knowledge graphs present a representation of knowledge / data that differs from the representation of knowledge in text document formats. For example, a knowledge graph provided by Wikidata is a representation of document information presented in Wikipedia. As can be understood, the knowledge presented in a knowledge graph for entities (e.g., "dog," "car," "United States," etc.) can be found "nearby" (e.g., in the context of information pairings) the nodes associated with that entity. That is, data related to an entity can be directly linked to the entity or is available only a few connected nodes away. However, outside of a knowledge graph, the same information may reside in many written (e.g., electronic) documents (e.g., web pages, news articles, books, etc.). Therefore, using a knowledge graph to obtain information related to entities may, in some ways, be more advantageous than obtaining the same information from many different documents. As described herein, a document or record of information can be viewed as a set of qualified data including fact definitions, entities, and relationships related to knowledge or information.

[0023] Knowledge-grounding for system-generated responses is a crucial objective in dialogue systems because it facilitates richer conversations beyond simply enabling continuous user requests. A fundamental challenge is retrieving appropriate and / or valid information given the dialogue context corresponding to user input. Simple text-based retrieval from data sources used to identify relevant information may not leverage the advantages offered by knowledge graphs, such as information colocation. On the other hand, graph coding techniques may not incorporate the relationships between nodes in a knowledge graph.

[0024] Generative models are machine learning models configured to generate output data that may not have existed previously. Examples include text generation models, image generation models, and so on. Generative models (especially text generation models) can be trained using knowledge graphs. In doing so, as a result of this training, the generative model can incorporate information from the knowledge graph. Therefore, during inference time when the generative model is prompted to create output, it may be able to generate the requested information without involving separate operations to traverse the knowledge graph to identify and retrieve the requested information. Thus, a generative model can receive text, such as a textual representation of a user's utterance query received by a virtual assistant system, and then generate relevant data in response to the query based on the information it has incorporated from the knowledge graph during training.

[0025] Generative models can be configured to generate output data in different formats. Generative models (such as generative language models) can generate natural language responses. For example, the generated response data can be in text format for TTS translation or presentation, such as in interactions with chatbots. Generative models can alternatively or additionally generate data tuples, such as triples comprising a subject entity, a relation, and an object entity (e.g., "Sacramento, is the capitol of, California"). Such tuples can be passed to another component (such as a natural language generation component) to generate natural language response data that can be output to the user.

[0026] In some system configurations, information stored in a knowledge graph can be retrieved by traversing the knowledge graph. Depending on the size of the data representing the knowledge graph, a significant amount of computer memory may be required to store it, and considerable computational power / time may be needed to traverse it. As is understood, a "comprehensive" knowledge graph (such as Wikidata) can be extremely large, and performing operations on such a graph can involve consuming substantial computational resources. However, generative models trained using knowledge graphs may have smaller memory requirements than the knowledge graph itself, and therefore reduce the use of computational resources when attempting to obtain answers to questions that require reference to the knowledge graph.

[0027] However, because generative models are typically configured to generate or infer relevant data corresponding to the input query, situations arise where they may generate invalid data, such as non-existent entities (e.g., "West America", "green dog") or invalid triples of unrelated entities and relationships (e.g., "Chicago, is president of, dog"). Therefore, even if a knowledge base can be used to train the generative model, its inference operations cannot guarantee that the output of the generative model necessarily represents the facts presented in the knowledge base itself. Furthermore, if the knowledge base changes to reflect updated or new information, a generative model trained on a specific knowledge base may become outdated or obsolete. Therefore, updating the knowledge base may require retraining a new generative model. This training itself is a significant drain on computational resources.

[0028] The methods and techniques described in this paper propose dynamically constructing an ordered tree-based data structure (called a "trie") that is based on empirical knowledge documents but more restrictive to the topic of the user query, and can be used to ensure the accuracy of the information output by a generative language model. This finite data structure can be called a local trie. The local trie can be used as input to the generative model, and thus constrains the generative model to reason about actually valid triples and / or other data. Trie-based constrained decoding is used to guide the decoding steps of the generative model based on the local trie constructed from a knowledge base.

[0029] Figure 1A This is a diagram illustrating a knowledge graph-based generative model process with local prefix tree constraints according to an embodiment of this disclosure. As described above, and as... Figure 1B As shown, user 105 can interact or converse with the virtual assistant system, which can... Figure 1A The implementation scheme shown includes operational components. For example... Figure 1B As illustrated, user 105 may directly transmit natural language input (e.g., speech) (e.g., audio input 111) to a local device 110, which is part of the virtual assistant system 100. Device 110 may convert the input audio 111 into audio data 411 and send the audio data to system component 120. User speech may include queries or requests for information. Virtual assistant system 100 may access one or more information sources (e.g., Wikipedia, library archives, etc.) for use as an evidence-based knowledge base 125.

[0030] The virtual assistant system 100 can receive user speech, and as referenced Figures 4 to 6 As described, system 100 can perform speech processing (such as ASR processing) to determine the text data representation of a user's utterance. Input text data 150 represents the text data of the user's utterance. (Input text data 150 may also be received via text or other input mechanisms.) Input text data 150 may represent a request for information (e.g., “Tell me about different dog breeds”, “What year was the capitol of Arizona founded?”, etc.). Context determination component 155 can receive (130) input text data 150. Context determination component 155 can determine (132) the context and / or topic of the received input text data 150. For example, if input text data 150 is “Tell me about different dog breeds”, then context determination component 155 can determine the context of “dog breeds”.

[0031] The context determination component 155 can receive intent data from the NLU component 460. (See reference...) Figure 5 and Figure 6 As described, NLU component 460 can determine the intent or meaning of user utterances or input text data 150. In some embodiments, context determination component 155 can use previous interactions and conversational sessions as part of context determination. Input text data 150 can be part of a conversational session that includes earlier user utterances and system responses. Context determination component 155 can use earlier interactions in the conversational session to determine and / or refine the context of input text data 150. For example, if user 105 asks "What is the most popular dog breed?" before input text data 150 "Tell me about different dog breeds" and the system generates a response of "Labrador Retriever", then context determination component 155 can determine context such as "dog breeds similar to the Labrador Retriever", such that response generation can focus on dog breeds related to the Labrador Retriever.

[0032] In some implementations, the context determination component 155 may use previous conversations as part of the context determination. For example, if the user has previously had a conversation with the virtual assistant system about European cars, when the user makes a query such as "Tell me the new cars this year," the context determination component 155 may use the previous conversation to direct the context to "new European cars." The context determination component 155 may also use other data to determine the context, such as other data 620, 691, 704, 825 discussed below, user profile data, etc.

[0033] The context determination component 155 can provide the determined context to the subgraph generation component 195 and the prefix tree generation component 160. As described above, a prefix tree is an ordered tree data structure that can be used to store associative data structures. The subgraph generation component 195 and the prefix tree generation component 160 can use the context to retrieve data from the knowledge base 125. In some embodiments, the subgraph generation component 195 can retrieve data from the knowledge base 125 and provide the data to the prefix tree generation component 160. The subgraph generation component 195 and / or the prefix tree generation component 160 can receive the context data and determine (134) at least one document (e.g., knowledge document 165) corresponding to the context from the knowledge base 125. The information retrieved from the knowledge base 125 can be a subset of the information stored in the knowledge base 125 based on the determined context. The subgraph generation component 195 and / or the prefix tree generation component 160 can retrieve the most relevant information record (e.g., news article, Wikipedia page, etc.) corresponding to the context stored in the knowledge base 125, i.e., knowledge document 165. For example, the subgraph generation component 195 can be configured to retrieve the eight most relevant Wikipedia pages corresponding to the determined context. For example, and as... Figure 1A As illustrated, knowledge document 165 may include documents or information records of popular dog breeds based on input text data 150.

[0034] Subgraph generation component 195 can generate (136) a subgraph 190 corresponding to knowledge document 165. In some embodiments, subgraph 190 can be generated based on knowledge document 165. In some embodiments, the knowledge graph can be stored in knowledge base 125 based on documents or information records stored in knowledge base 125 (or stored in a manner associated with the knowledge base). Subgraph 190 can be constructed from graph portions within the knowledge graph, wherein the graph portions correspond to knowledge document 165.

[0035] Prefix tree generation component 160 can generate a local prefix tree 175. The local prefix tree 175 may include triples or other associated data groups determined from information in the selected knowledge document 165. The local prefix tree 175 uses a tree-based structure to efficiently look up triples represented within it. The prefix tree generated by prefix tree generation component 160 can be considered a local prefix tree because it is based on the selected knowledge document 165, in contrast to a global prefix tree that may be based on all or most of the knowledge base 125.

[0036] For each set of received input text data 150, the context determination component 155 can determine the context. Therefore, a new context can be determined for each received user input. The subgraph generation component 195 and / or the prefix tree generation component 160 can sequentially retrieve the knowledge document 165 corresponding to each determined context. The prefix tree generation component 160 can dynamically generate a local prefix tree 175 based on the knowledge document 165 corresponding to the context of each received input text data 150. In some cases, the local prefix tree 175 can be reused between user inputs, for example, when the determined context for a user input is determined to be substantially similar to a previously determined context for a previous user input.

[0037] Prefix tree generation component 160 can provide a local prefix tree 175, and subgraph generation component 195 can provide subgraph 190 to generative language model component 170. Context determination component 155 can provide context data to generative language model component 170. Generative language model component 170 can use a language generation model to generate (138) response data corresponding to input text data 150 (e.g., question, inquiry, comment, etc.), wherein the input text data 150 and subgraph 190 of at least one document in knowledge document 165 are provided as input to the language generation model.

[0038] The generative language model component 170 can use a bidirectional autoregressive Transformer (BART) model architecture as known in the art. The BART model architecture can use a sequence-to-sequence (seq2seq) / machine translation architecture with a bidirectional encoder (such as a Transformer-based bidirectional encoder representation (BERT) model) and a left-to-right decoder, as known in the art. The BART model can be pre-trained on a large amount of text data to reconstruct sentences based on incomplete versions of the sentences. The BART model architecture can be used to generate response data. Specifically, the model aims to learn, in an autoregressive manner, the conditional probability of the output y given the input x, as represented by equation (1), where θ represents the parameters of the model:

[0039] During the training of the generative language model, the objective is set to maximize the conditional log-likelihood of the target sequence using cross-entropy loss and label smoothing for regularization.

[0040] Generative language model component 170 can be trained with different subgraphs to generate sentence-level subgraphs, or in other words, to generate subgraphs corresponding to sentences from input information (e.g., knowledge document 165). In some implementations, the language generation model can traverse subgraph 190 to determine the next potential edge or node of subgraph 190 in response to input text data 150. For example, the language generation model can determine a set of potential nodes representing words from subgraph 190 and assign a score to each potential node. The score can represent a confidence value based on previously determined objects and / or relations. Local prefix trees 175 can be used to validate the potential node with the highest score.

[0041] Generative language model component 170 uses a prefix tree-based constraint decoding method to generate valid triples for subgraph 190 based on knowledge document 165. During the decoding process, generative language model component 170 is constrained by the content represented in the local prefix tree 175. This constraint applies to both the elements of the generated triples (e.g., object, relation, and subject) and multi-word entities (e.g., "North America", "US Patent and Trademark Office", etc.). In this way, generative language model component 170 can be trained in a manner such that, during inference, it does not generate output data that violates the knowledge document 165 provided at runtime.

[0042] At runtime, the generative language model component 170 can receive input text data 150, a subgraph 190, and a local prefix tree 175 as input. The model of the generative language model component 170 can be trained using the subgraph to traverse the subgraph 190 and generate relevant triples (e.g., subject, relation, and object) corresponding to the provided context data based on the relations defined in the subgraph 190. Therefore, the generative language model component 170 is trained to use the local prefix tree 175 to constrain the model's output to valid triples.

[0043] Generative language model component 170 can traverse subgraph 190 and use subgraph 190 as a data source to generate triples (e.g., output sequences of related entities) in response to input text data 150. Generative language model component 170 can determine (140) that the response data is embodied in at least one document. During decoding, local prefix trees 175 are used to validate the triples. For a given entity (e.g., the body of the triple), the model can generate a relation that is validated using local prefix trees 175 to determine whether the relation is valid for that entity. Generative language model component 170 can then validate the object of the generated triple against the definition of local prefix trees 175. If the relation or object of the generated triple does not correspond to the data or information item represented in the local prefix tree, generative language model component 170 generates another triple. The model can be further trained based on cases where triples are rejected due to invalidity.

[0044] If the generative language model component 170 determines, using the local prefix tree 175, that the generated triples are valid (e.g., the triples exist in the local prefix tree 175), then the generative language model component 170 can identify sentences corresponding to valid triples. Based on verification against the local prefix tree 175, the valid triples correspond to at least one sentence in the knowledge document 165. The correspondence can be based on a linearized subgraph of sentences from the knowledge document 165 and a comparison of the linearized subgraph with the output sequence 180. The output sequence 180 exemplifies an example sequence (e.g., triples) generated by the model and verified against the local prefix tree 175. The output sequence graph 185 exemplifies the output sequence 180 in the form of a knowledge graph. The output sequence graph 185 is a sub-subgraph of subgraph 190 (e.g., dog breed subgraph 290). In response to determining that response data is embodied in at least one document, a component such as the dialogue manager 472 or the NLG component 479 can generate (142) output text data based at least partially on the response data. Figure 4 and Figure 7 As illustrated, the context determination component 155, the prefix tree generation component 160, and the generative language model component 170 may be combined as part of the dialogue manager 472. In some implementations, the output sequence 180 may be sent to the NLG 479 to generate a natural language response for the user that incorporates information from the output sequence 180.

[0045] Generative language models with local prefix tree constraints can be considered dynamic because they do not need to be retrained if the factual knowledge (e.g., knowledge base 125) and / or the knowledge graph representing that factual knowledge changes. In other words, any changes made to the underlying knowledge sources result in updated information being provided at runtime (e.g., knowledge document 165). Therefore, a particular generative language model component 170 does not become obsolete or outdated due to changes in knowledge base 125. Furthermore, the generation of the local prefix tree is dynamic and therefore includes any changes or updates to knowledge base 125.

[0046] Figure 1B This is a conceptual diagram illustrating a virtual assistant system 100 for natural language processing and knowledge-based information generation according to an embodiment of this disclosure. (See diagram for example.) Figure 1B As shown, the virtual assistant system 100 may include a voice-enabled device 110 local to the user 105, a natural language command processing system 120 (abbreviated as "system component 120") connected across one or more networks 199, and one or more skill support systems 425 in some embodiments. Figure 4(As shown in the figures and discussion of this disclosure). Although certain steps are illustrated in a particular order in the accompanying drawings and discussion, the steps described may be performed in a different order (and some steps may be removed or added) without departing from this disclosure.

[0047] Device 110 may receive audio corresponding to spoken natural language input from user 105. In some embodiments, device 110 may process the audio after detecting a wake word. Device 110 may generate audio data corresponding to audio input 111 and may send the audio data to system component 120. Device 110 may send the audio data to system component 120 via an application installed on device 110 and associated with system component 120. An example of such an application is the Amazon Alexa application, which may be installed on a smartphone, tablet, etc. In some implementations, device 110 may receive text data 413 corresponding to natural language input from user 105 and send the text data to system component 120. Device 110 may also receive output data from system component 120 and generate synthesized speech output. Device 110 may include a camera for capturing image and / or video data for processing by system component 120. Figure 11 Examples of various devices 110 are further illustrated. System component 120 may be a remote system, such as a set of computing components geographically located far from device 110 but accessible via network 199 (e.g., a server accessible via the Internet). System component 120 may also include remote systems physically separate from device 110 but geographically located near device 110 and accessible via network 199 (e.g., a home server located in the same residence as device 110). System component 120 may also include some combination thereof, such as a combination where some components / operations are performed via a home server and other components / operations are performed via a geographically remote server.

[0048] Generative language model component 170 may include a language model as a generative model. In some implementations, the language model may be a large language model (LLM). An LLM is an advanced artificial intelligence system designed to process, understand, and generate human-like text based on large amounts of data. LLMs can be constructed using deep learning techniques such as neural networks and can be trained on extensive datasets including text (or other types of data) from a wide range of sources such as books and websites for natural language processing.

[0049] In some implementations of the language model in generative language model component 170, the language model can be a transformer-based seq2seq model involving an encoder-decoder architecture. In some such implementations, the language model can be a multilingual seq2seq model with (approximately) 20 billion parameters, pre-trained on a combination of denoising and causal language modeling (CLM) tasks in various languages ​​(e.g., English, French, German, Arabic, Hindi, Italian, Japanese, Spanish, etc.), and the language model can be pre-trained on approximately one trillion lexical units. After training on the CLM task, the language model may be able to perform context learning. An example of such an LLM is the Alexa Teacher Model (Alexa™).

[0050] In an implementation where the language model of generative language model component 170 is an LLM, the input to the LLM (e.g., input text data 150) can be in the form of prompts. The prompts can be natural language input, such as instructions, for the LLM to generate output based on the prompts. The output generated by the LLM can be natural language output in response to the prompts. Both the prompts and the output can be text in a specific language (e.g., English, Spanish, German, etc.). For example, for the example prompt “How do I cook rice?”, the LLM could output a recipe for cooking rice (e.g., a step-by-step process). As another example, for the example prompt “I am hungry. What restaurants in the area are open?”, the LLM could output a list of restaurants near the user that are currently open.

[0051] Figure 2 An example is given of a set of documents based on some implementation schemes and a knowledge graph based on that set of documents. For example... Figure 1A As shown, user 105 can provide input audio 111 to request information, such as input text data 150 (e.g., "Tell me about different dog breeds"). Context determination component 155 can determine the context of the request. Subgraph generation component 195 and / or prefix tree generation component 160 then use this context to determine a set of relevant documents as knowledge documents 165. The type of document (e.g., news articles, books, research papers, etc.) may depend on the type of documents stored in knowledge base 125. Additionally, knowledge document 165 may include multiple types of documents for a given context.

[0052] like Figure 2As shown, the example dog breed knowledge document 265 exemplifies a set of example evidence documents identified as knowledge document 165 by the prefix tree generation component 160. The subgraph generation component 195 can generate a subgraph 190, such as a dog breed subgraph 290, based on the dog breed knowledge document 265. The dog breed subgraph 290 can be provided as input to the generative language model component 170. The prefix tree generation component can generate a local prefix tree 175 based on the example dog breed knowledge document 265. As described above, a prefix tree is an ordered tree data structure that can be used to store associative data structures, such as entities and relations (e.g., information items) existing in the example dog breed knowledge document 265.

[0053] During decoding, a local prefix tree 175 can be used to determine valid reasoning. For example, the generative language model component 170 can generate the subject "Labrador Retriever" and the relation "belongs to". Then, the generative language model component 170 can generate the object "Working Group" and verify the generated object against the local prefix tree 175. The generative language model component 170 can determine, based on the local prefix tree 175 representing the example dog breed knowledge document 265, that the object "Working Group" is invalid for the previously determined subject and relation (e.g., "Labrador Retriever", "belongs to"), and attempt to generate a new entity for that object. A similar verification process can be performed in the generation of multi-word elements. For example, the generative language model component 170 can generate "Doberman" during the decoding process, and then generate "Retriever". The generative language model component 170 can verify "Doberman Retriever" against the local prefix tree 175 and determine that it is not a valid subject.

[0054] Figure 3A and Figure 3B This is a flowchart illustrating a process 300 of using a pre-trained generative language model to perform knowledge retrieval via the generation of a knowledge subgraph, according to some embodiments of this disclosure. For example... Figure 3A As illustrated, the system may receive (305) input audio 111 corresponding to a question, comment, and / or request for information. The system may perform (310) speech recognition on the input audio 111 to obtain text data representing the speech of user 105. In some embodiments, the system may be configured to receive text input, such as input typed via a graphical user interface, and thus may be received as text data and bypass operations 305 and 310.

[0055] In some implementations, the system may perform (315) NLU processing to determine intent data corresponding to the text data. The intent data may represent the input audio 111 indicating a question and / or a request for information. The orchestrator 430 may receive the results of language processing 492, as referenced below. Figure 4 Detailed description. Dialogue manager 472 can receive text data and determine one or more meanings associated with phrases or statements represented in the text data based on the words represented in the text data. Dialogue manager 472 can determine a target corresponding to an action the user expects to perform, as described in the following reference. Figure 4 and Figure 7 Described.

[0056] The dialogue manager 472 can determine whether user 105 is asking a question and / or requesting information based on text data and / or intent data. The dialogue manager 472 can direct the text data and / or intent data to the context determination component 155.

[0057] In some implementations, the context determination component 155, subgraph generation component 195, prefix tree generation component 160, and generative language model component 170 may be part of one or more application programming interfaces (APIs). System 100 can call APIs as referenced. Figure 1A The function in the API corresponding to at least one of the described context determination component 155, subgraph generation component 195, prefix tree generation component 160 and / or generative language model component 170.

[0058] The context determination component 155 may request previous conversation data from the conversation storage 730. If previous conversation data corresponding to user 105 exists, the context determination component 155 may receive (320) the previous conversation data / user profile and / or other data of user 105. The previous conversation data may include previous user input from the current conversation session and / or data corresponding to a previous conversation session. The conversation storage 730 may store the previous conversation data.

[0059] The context determination component 155 may use text data and / or intent data, as well as (if available) previous dialogue data, to determine (325) the context or topic of the input audio 111. The context determination component 155 may refer to the following. Figure 7 The described dialog manager 472 operates similarly. The context determination component 155 may utilize the entity resolver 770 to determine the context. In some implementations, the entity resolver 770 may be used in conjunction with a reference... Figure 5 and Figure 6 The NER component 562 and / or entity resolution component 670 described in relation to NLU operations operate similarly.

[0060] Subgraph generation component 195 and / or prefix tree generation component 160 may receive context from context determination component 155. Subgraph generation component 195 and / or prefix tree generation component 160 may select (330) a set of documents (e.g., knowledge document 165) that are context-dependent from knowledge base 125. The type of document may depend on the context and the type of document stored in knowledge base 125. Subgraph generation component 195 may generate (335) subgraph 190, such as the subgraph described above, based on knowledge document 165. Subgraph generation component 195 may identify entities that appear in two or more of the knowledge document 165 and construct a subgraph 190 of triples based on the collected data from knowledge document 165.

[0061] In some implementations, subgraph 190 may be extracted from a pre-existing knowledge graph. For example, Wikidata includes a knowledge graph representing information existing in Wikipedia documents (e.g., individual Wikipedia pages). Subgraph 190 may be based on a subgraph extracted from the knowledge graph corresponding to each document in knowledge document 165. Therefore, in such implementations, subgraph generation component 195 may retrieve subgraph 190 rather than generate it.

[0062] In some implementations, subgraph 190 may be limited to the information contained in the set of documents constituting knowledge document 165. In other words, the terminating node of the subgraph includes entities included in knowledge document 165. In some implementations, subgraph 190 can be expanded to include links in knowledge base 125 or documents connected to knowledge document 165. Limitations may be imposed on the expansion of subgraph 190, such as limiting the number of hops beyond nodes representing entities present in knowledge document 165.

[0063] Subgraph generation component 195 can linearize (340) the subgraph 190 generated from knowledge document 165. In some implementations, such as Figure 2 As illustrated in the dog breed subgraph 290, subgraph 190 may include one or more triples (subject, relation, object), where entities are nodes and relations are edges. Subgraph generation component 195 can linearize subgraph 190 into a sequence, making it adaptable to the sequence-to-sequence architecture of generative language model component 170. For example, subgraph 190 can be linearized using special lexical units, such as... Used to start the main entity, <rel>Used for relation types, and <obj>For the object entity in each triple, and using additional special morphemes. <et>This indicates the end of the triple sequence. During the process of generating the subgraph 190 of triples based on knowledge document 165 or the knowledge graph traversal, interdependencies between triples can be considered. An entity appearing as the tail in one triple may appear as the head in another. Therefore, it may be beneficial to ensure a consistent ordering of the target triples to capture this interdependency during subgraph linearization. Entity nodes appearing in the target subgraph can be ordered in descending order based on their in-degree value (e.g., the number of edges leading to a vertex or node). A higher in-degree value may mean that the entity is more likely to be visited by other entities, indicating the importance of the node, and more important nodes are likely to be visited first. Subsequently, starting with each unvisited and ordered entity node, a depth-first search (DFS) traversal can be performed on the unvisited entity nodes to generate distinct paths. These paths can then be connected to create a single sequence, which serves as the target sequence for the generative model of the generative language model component 170 to learn.

[0064] Additionally, for each document of knowledge document 165, prefix tree generation component 160 may generate a prefix tree (336) for the document. Local prefix tree 175 may include the prefix tree generated for the document of knowledge document 165. The prefix tree is dynamically constructed based on the evidence knowledge document and thus represents the currently valid triples for each inference of generative language model component 170. The prefix tree may include triples for a given evidence document, where traversal from the root node to the leaf node produces valid and complete triples.

[0065] Input text data 150, subgraph 190, and local prefix tree 175 can be provided (345) to generative language model component 170. Generative language model component 170 can receive (350) input text data 150, subgraph 190, and local prefix tree 175 and begin inference. The model of generative language model component 170 can be trained using the subgraph to traverse subgraph 190 and generate relevant triples (e.g., subject, relation, and object) based on the relations defined in subgraph 190 and the provided context data. The language model can identify one or more edges and / or nodes represented in subgraph 190 and determine confidence value scores corresponding to one or more edges and / or nodes. Therefore, the model may generate invalid triples (e.g., "lemonade, is made from, apples"). Generative language model component 170 uses local prefix tree 175 to constrain the model's traversal of subgraph 190 to generate valid triples (e.g., triples of local prefix tree 175).

[0066] Generative language model component 170 can use subgraph 190 as a data source to generate (355) triples (e.g., output sequences) corresponding to input text data 150. During decoding, local prefix trees 175 are used to validate (360) the triples. For a given entity (e.g., the body of a triple), the model can generate a relation that is validated using information items from the local prefix tree 175 to determine if the relation is valid for that entity. Generative language model component 170 can then validate the object of the generated triple against the definition of the local prefix tree 175. If the relation or object of the generated triple does not correspond to the information items represented in the local prefix tree, the process returns to generating (355) another triple. The model can be further trained based on cases where triples are rejected due to invalidity.

[0067] If, at the triple verification (360), the generative language model component 170 determines using the local prefix tree 175 that the generated triple is valid (e.g., the triple exists in the local prefix tree 175), then the generative language model component 170 can identify (365) the sentence corresponding to the valid triple. Based on the verification against the local prefix tree 175, the valid triple corresponds to at least one sentence in the knowledge document 165. The correspondence can be based on a linearized subgraph of sentences in the knowledge document 165. The selection of at least one sentence in the knowledge document 165 can be computed based on the similarity between each candidate sentence in the document and the valid triple. The generative language model component 170 can select the sentence with the highest similarity to the valid triple.

[0068] In some implementations, the generative language model component 170 may use a previous object as a new subject for a triple and continue the generative process for relations and objects based on the new subject. In some implementations, the generative language model component 170 may generate (370) additional paths based on valid triples, such as information that may drive the conversation forward. The generative language model component 170 may generate additional sequences (triples) based on valid triples to identify interesting or relevant triples. The paths of these additional sequences may create subgraphs from the knowledge document 165.

[0069] At least one sentence in knowledge document 165 corresponding to a valid triple, or a sequence of subgraphs representing that sentence, can be output from generative language model component 170 and provided as a result (375) to dialogue manager 472.

[0070] For reference Figure 4 and Figure 7 As described, the dialogue manager can use the results to generate (380) a response to the input text data 150. In some implementations, the results can be sent (380) to the skill component 490 for further processing or to perform an action. The dialogue manager 472 can direct the response to the language output component 493 to generate (385) output using TTS, as described in the reference. Figure 8 Described.

[0071] This system can be configured to consolidate user licenses and perform the activities disclosed herein only with user approval. Thus, the systems, apparatus, components, and techniques described herein will generally be configured to restrict processing where appropriate and to process user information only in a manner that ensures compliance with all applicable laws, regulations, standards, etc. This system and techniques can be implemented on a geographical basis to ensure compliance with the laws of the respective jurisdictions and agencies where the system's components and / or users are located.

[0072] System 100 can be used as follows Figure 4 The various components described operate. These components may reside on the same or different physical devices. Communication between the components may be direct or across network 199. Device 110 may include an audio capture component (such as a microphone or microphone array of device 110) that captures audio 111 and creates corresponding audio data. Once speech is detected in the audio data representing audio 111, device 110 can determine whether the speech is directed to device 110 / system component 120. In at least some embodiments, this determination may be performed using a wake word detection component 420. The wake word detection component 420 may be configured to detect various wake words. In at least some examples, each wake word may correspond to the name of a different digital assistant. An example wake word / digital assistant name is "Alexa". In another example, system input may be in the form of text data 413, such as the result of a user typing input in the user interface of device 110. Other forms of input may include a user pressing a physical or virtual button on device 110, a user making a gesture, or other indications. Device 110 may also use its camera 918 to capture images and may send image data 421 representing those images to system component 120. Image data 421 may include raw image data or image data processed by device 110 before being sent to system component 120. Different components of the system may use image data 421 in various ways to perform operations, such as determining whether a user is speaking to the system, interpreting user commands, responding to user commands, etc.

[0073] The wake word detector 420 of device 110 can process audio data representing audio 111 to determine whether speech is present therein. Device 110 can use various techniques to determine whether the audio data includes speech. In some examples, device 110 can apply voice activity detection (VAD) techniques. Such techniques can determine the presence of speech in audio data based on various quantitative aspects of the audio data, such as the spectral slope between one or more frames of the audio data; the energy levels of the audio data in one or more spectral bands; the signal-to-noise ratio of the audio data in one or more spectral bands; or other quantitative aspects. In other examples, device 110 can implement a classifier configured to distinguish speech from background noise. The classifier can be implemented using techniques such as linear classifiers, support vector machines, and decision trees. In still other examples, device 110 can apply Hidden Markov Model (HMM) or Gaussian Mixture Model (GMM) techniques to compare the audio data with one or more stored acoustic models, which can include models corresponding to speech, noise (e.g., ambient noise or background noise), or silence. Still other techniques can be used to determine the presence of speech in audio data.

[0074] Wake word detection is typically performed without language analysis, text analysis, or semantic analysis. Instead, the audio data representing audio 111 is analyzed to determine whether specific characteristics of the audio data match pre-configured acoustic waveforms, audio signatures, or other data corresponding to a wake word.

[0075] Therefore, the wake word detection component 420 can compare audio data with stored data to detect wake words. One method for wake word detection uses a general large vocabulary continuous speech recognition (LVCSR) system to decode the audio signal, where the wake word search is performed in the resulting lattices or confusion networks. Another method for wake word detection constructs a Hidden Markov Model (HMM) for each wake word and non-wake word speech signal. Non-wake word speech includes other spoken words, background noise, etc. One or more HMMs can be constructed to model the characteristics of non-wake word speech; this is called a filled model. Viterbi decoding is used to search for the optimal path in the decoding graph, and the decoding output is further processed to make a decision about the presence of a wake word. This method can be extended to include discriminative information by incorporating a hybrid DNN-HMM decoding framework. In another example, the wake word detection component 420 can be built directly on a deep neural network (DNN) / recurrent neural network (RNN) structure without involving an HMM. This architecture can estimate the posterior of the wake word using context data by stacking frames within a context window for use with a DNN or by using an RNN. Subsequent posterior threshold adjustment or smoothing is applied to make the decision. Other techniques for wake word detection, such as those known in the art, may also be used.

[0076] Once the wake word detector 420 detects a wake word and / or the input detector detects input, the device 110 can "wake up" and begin transmitting audio data 411, representing audio 111, to the system component 120. The audio data 411 may include data corresponding to the wake word; in other embodiments, the portion of the audio corresponding to the wake word is removed by the device 110 before the audio data 411 is sent to the system component 120. In the case of touch input detection or gesture-based input detection, the audio data may not include the wake word.

[0077] In some implementations, system 100 may include more than one system component 120. System component 120 may respond to different wake words and / or perform different categories of tasks. Each system component 120 may be associated with its own wake word, such that saying a specific wake word causes audio data to be sent to and processed by a specific system. For example, wake word detector 420 detecting the wake word "Alexa" may cause audio data to be sent to system component 120a for processing, while wake word detector detecting the wake word "Computer" may cause audio data to be sent to system component 120b for processing. The system may have separate wake words and systems for different skills / systems (e.g., "Dungeon Master" for game skill / system component 120c) and / or such skills / systems may be coordinated by one or more skills 490 of one or more system components 120.

[0078] After being received by system component 120, audio data 411 can be sent to orchestration component 430. Orchestration component 430 may include memory and logic that enable orchestration component 430 to transmit various data segments and various forms of data to various components of the system, and to perform other operations as described herein.

[0079] The orchestrator component 430 can send audio data 411 to the language processing component 492. The language processing component 492 (sometimes also referred to as the spoken language understanding (SLU) component) includes an automatic speech recognition (ASR) component 450 and a natural language understanding (NLU) component 460. The ASR component 450 can transcribe the audio data 411 into text data. The text data output by the ASR component 450 represents one or more (e.g., in the form of an N-best list) ASR assumptions that represent the speech presented in the audio data 411. The ASR component 450 interprets the speech in the audio data 411 based on the similarity between the audio data 411 and a pre-established language model. For example, the ASR component 450 can compare the audio data 411 with models of sound (such as acoustic units, such as phonemes, senons, phonemes, etc.) and sound sequences to identify words that match the sound sequences of the speech presented in the audio data 411. In some embodiments, the ASR component 450 sends the resulting text data to the NLU component 460 via the orchestrator component 430. The text data sent from the ASR component 450 to the NLU component 460 may include a single highest-scoring ASR hypothesis, or may include an N-best list comprising multiple highest-scoring ASR hypotheses. The N-best list may additionally include a corresponding score associated with each ASR hypothesis represented therein.

[0080] The speech processing system 492 may also include an NLU component 460. The NLU component 460 may receive text data from the ASR component. The NLU component 460 may attempt to semantically interpret phrases or statements represented in the text data by determining one or more meanings associated with phrases or statements represented in the text data. The NLU component 460 may determine an intent representing an action the user expects to perform and may determine information that allows a device (e.g., device 110, system component 120, skill component 490, skill system 425, etc.) to perform that intent. For example, if the text data corresponds to "play the 5th Symphony by Beethoven," the NLU component 460 may determine the system's intent to output music and may identify "Beethoven" as the artist / composer and "5th Symphony" as the piece of music to be played. For a further example, if the text data corresponds to "what is the weather," then NLU component 460 can determine the system's intent to output weather information associated with the geographic location of device 110. In another example, if the text data corresponds to "turn off the lights," then NLU component 460 can determine the system's intent to turn off the lights associated with device 110 or user 105. However, if NLU component 460 cannot resolve the entity (e.g., because the entity is mentioned by an anaphora such as "this song" or "my next appointment"), speech processing system 492 can send a decoding request to another speech processing system 492 to obtain information about the entity mention and / or other context related to the utterance. Speech processing system 492 can enhance, correct, or base the resulting data on both the audio data 411 and any data received from the other speech processing system 492.

[0081] NLU component 460 can return NLU result data 685 / 625 (which may include tagged text data, intent indicators, etc.) to orchestrator 430. Orchestrator 430 can forward the NLU result data to skill component 490. If the NLU result data includes a single NLU hypothesis, NLU component 460 and orchestrator component 430 can direct the NLU result data to the skill component 490 associated with the NLU hypothesis. If the NLU result data 685 / 625 includes a list of N-best NLU hypotheses, NLU component 460 and orchestrator component 430 can direct the highest-scoring NLU hypothesis to the skill component 490 associated with the highest-scoring NLU hypothesis. The system may also include an NLU post-sorter 665 that can combine other information to sort the potential interpretations determined by NLU component 460. (See below for more information.) Figure 5 and Figure 6 The NLU component 460, the NLU post-sorter 665, and other components are described in more detail.

[0082] A skill component can be software similar to a software application that runs on system component 120. That is, skill component 490 enables system component 120 to perform specific functionalities to provide data or generate output for some other request. As used herein, "skill component" can refer to software that can be placed on a machine or virtual machine (e.g., software that can be launched in a virtual instance when invoked). A skill component can be software customized to perform one or more actions, such as those instructed by a business entity, device manufacturer, user, etc. Things described herein as skill components may be referred to using many different terms (such as action, robot, app, etc.). System component 120 may be configured with more than one skill component 490. For example, a weather service skill component enables system component 120 to provide weather information, a car service skill component enables system component 120 to book trips relative to taxi or ride-sharing services, and a restaurant skill component enables system component 120 to order pizzas relative to a restaurant's online ordering system. Skill component 490 can cooperate between system component 120 and other devices (such as device 110) to perform certain functions. The input to skill component 490 may come from voice processing interaction or utilize other interaction or input sources. Skill component 490 may include hardware, software, firmware, etc., that may be dedicated to a specific skill component 490 or shared between different skill components 490.

[0083] Skill support system 425 can communicate with skill component 490 within system component 120 and / or directly with orchestrator component 430 or other components. Skill support system 425 can be configured to perform one or more actions. The ability to perform such actions is sometimes referred to as a "skill." That is, a skill enables skill support system 425 to perform a specific function in order to provide data or perform another action requested by the user. For example, a weather service skill enables skill support system 425 to provide weather information to system component 120, a car service skill enables skill support system 425 to book a trip relative to a taxi or ride-sharing service, and a pizza ordering skill enables skill support system 425 to order pizza relative to a restaurant's online ordering system, etc. Additional types of skills include home automation skills (e.g., skills that enable users to control home appliances such as lights, door locks, cameras, thermostats, etc.), entertainment device skills (e.g., skills that enable users to control entertainment devices such as smart TVs), video skills, newsletter skills, and custom skills not associated with any pre-configured type of skill.

[0084] System component 120 may be configured with a skill component 490 specifically for interacting with the skill support system 425. Unless otherwise explicitly stated, references to a skill, skill device, or skill component may include the skill component 490 operated by system component 120 and / or the skill operated by skill support system 425. Furthermore, the functionality or skill described herein may be referred to using many different terms such as action, robot, app, etc. Skill 490 and / or skill support system 425 may return output data to orchestrator 430.

[0085] Dialogue processing is a field of computer science that involves communication between computing systems and humans via text, audio, and / or other forms of communication. While some forms of dialogue processing simply involve generating responses based on only the user's latest input (i.e., a single-turn dialogue), more complex dialogue processing involves identifying one or more goals expressed by the user in a multi-turn dialogue and optionally taking action accordingly, such as making a restaurant reservation and / or booking a flight. These multi-turn "goal-oriented" dialogue systems typically require identifying, retaining, and using information collected during more than one input in back-and-forth or "multi-turn" interactions with the user.

[0086] System 100 may include a dialogue manager component 472 that manages and / or tracks dialogues between a user and a device. As used herein, a “dialogue” may refer to data transmissions (such as those involving multiple user inputs and system 100 outputs) between system 100 and a user (e.g., via device 110), all of which are associated with a single “dialogue” between the system and the user, which may have been initiated by a single user input that started the dialogue. Therefore, the data transmissions of a dialogue may be associated with the same dialogue identifier, which components throughout system 100 can use to track information about the entire dialogue. Subsequent user inputs within the same dialogue may or may not begin with a spoken wake word. Each natural language input in a dialogue may be associated with a different natural language input identifier, such that multiple natural language input identifiers can be associated with a single dialogue identifier. Furthermore, depending on the context of the input, other non-natural language inputs (e.g., image data, gestures, button presses, etc.) may be associated with a particular dialogue. For example, a user can initiate a conversation with system 100 to request food delivery using verbal language, and the system can respond by displaying images of available food items. The user can respond by speaking (e.g., "Item 1" or "that one"), by gesture (e.g., pointing to an item on the screen or giving a thumbs up), or by touching the desired item on the screen. Non-voice input (e.g., gestures, screen touches, etc.) can be part of the conversation, and the associated data can be linked to a conversation identifier.

[0087] After identifying that a user is engaging in a conversation with the user, the conversation manager component 472 can associate a conversation session identifier with that conversation. The conversation manager component 472 can track user input and the corresponding system-generated responses to that user input as a round. The conversation session identifier can correspond to multiple rounds of user input and corresponding system-generated responses. The conversation manager component 472 can directly transmit the data identified by the conversation session identifier to the orchestrator component 430 or other components. Depending on the system configuration, the conversation manager 472 can determine an appropriate system-generated response to provide a specific utterance or user input round. Alternatively, the creation of system-generated responses can be managed by another component of the system (e.g., the speech output component 493, NLG 479, orchestrator 430, etc.) while the conversation manager 472 selects an appropriate response. Alternatively, another component of system component 120 can use the techniques discussed herein to select responses. The text of the system-generated response can be sent to the TTS component 480 to create audio data corresponding to the response. The audio data can then be sent to the user device (e.g., device 110) for final output to the user. Alternatively (or additionally), dialogue responses may be returned in text or some other form.

[0088] The dialogue manager 472 may receive one or more ASR hypotheses (i.e., text data) and perform semantic interpretation on the phrases or statements represented therein. That is, the dialogue manager 472 determines one or more meanings associated with the phrases or statements represented in the text data based on the words represented in the text data. The dialogue manager 472 determines the target corresponding to the action the user expects to perform and the text data fragments that allow the device (e.g., device 110, system component 120, skill 490, skill system 425, etc.) to perform the intent. For example, if the text data corresponds to "How is the weather?", the dialogue manager 472 may determine that system component 120 should output weather information associated with the geographic location of device 110. In another example, if the text data corresponds to "Turn off the lights", the dialogue manager 472 may determine that system component 120 should turn off the lights associated with device 110 or user 105.

[0089] The dialogue manager 472 may include a context determination component 155, a prefix tree generation component 160, and a generative language model component 170 to generate responses to user queries using a generative language model with local prefix tree constraints as described herein.

[0090] The dialogue manager 472 can send the resulting data to one or more skills 490. If the resulting data includes a single hypothesis, the orchestrator component 430 can send the resulting data to the skill 490 associated with that hypothesis. If the resulting data includes an N-best list of hypotheses, the orchestrator component 430 can send the highest-scoring hypothesis to the skill 490 associated with that highest-scoring hypothesis.

[0091] System component 120 includes a language output component 493. Language output component 493 includes a natural language generation (NLG) component 479 and a text-to-speech (TTS) component 480. NLG component 479 can generate text for the purpose of TTS output to a user. For example, NLG component 479 can generate text corresponding to instructions that correspond to a specific action to be performed by the user. NLG component 479 can generate appropriate text for various outputs as described herein. NLG component 479 may include one or more trained models configured to output appropriate text for a particular input. The text output by NLG component 479 can become input to TTS component 480 (e.g., output text data 815 discussed below). Alternatively or additionally, TTS component 480 can receive text data from skill 490 or other system components for output.

[0092] NLG component 479 may include a training model. NLG component 479 generates text data 815 based on dialogue data received from dialogue manager 472, such that the output text data 815 has a natural feel and, in some implementations, includes words and / or phrases specifically formatted for the requesting individual. The NLG may use templates to formulate responses. And / or the NLG system may include models trained based on various templates for shaping the output text data 815. For example, the NLG system may analyze transcripts of local news programs, television programs, sporting events, or any other media programs to obtain common components of the relevant language and / or region. As an illustrative example, the NLG system may analyze transcripts of regional sports programs to identify common words or phrases used to describe scores or other sports news in a particular region. The NLG may further receive dialogue history, formality indicators and / or command history, or other user history (such as dialogue history) as input.

[0093] The NLG system can generate dialogue data based on one or more response templates. Continuing the example above, the NLG system can select a template to respond to the question "What is the weather currently like?", which is in the form of "The weather currently is $weather_information$". The NLG system can analyze the logical form of the template to generate one or more text responses (including tags and annotations) to familiarize itself with the generated responses. In some implementations, the NLG system can determine which response is the most appropriate to select. Therefore, selection can be based on past responses, past questions, formality, and / or any other characteristics, or any other combination thereof. The text-to-speech component 480 can then be used to generate response audio data representing the responses generated by the NLG system.

[0094] The TTS component 480 can use one or more different methods to generate audio data from text data (e.g., synthesized speech). The text data input to the TTS component 480 may come from the skill component 490, the orchestrator component 430, or another component of the system. In a synthesis method known as unit selection, the TTS component 480 matches the text data against a database of recorded speech. The TTS component 480 selects matching units of the recorded speech and concatenates these units to form audio data. In another synthesis method known as parametric synthesis, the TTS component 480 modifies parameters (such as frequency, volume, and noise) to create audio data that includes an artificial speech waveform. Parametric synthesis uses a computerized speech generator, sometimes called a vocoder.

[0095] Device 110 may include still images and / or video capture components, such as one or more cameras, that capture one or more images. Device 110 may include circuitry for digitizing images and / or video for transmission as image data to system component 120. Device 110 may also include circuitry for voice command-based control of the camera, allowing user 105 to request the capture of image or video data. Device 110 may process commands locally or send audio data 411 representing a command to system component 120 for processing, after which system component 120 may return output data that causes device 110 to activate its camera.

[0096] User identification component 495 may treat audio data 411 and / or text data output by ASR component 450 as input. User identification component 495 may perform user identification by comparing audio characteristics in audio data 411 with stored audio characteristics of the user. User identification component 495 may also perform user identification by comparing biometric data (e.g., fingerprint data, iris data, etc.) received by a system associated with the current user input with stored biometric data of the user, assuming user permission and prior authorization. User identification component 495 may also perform user identification by comparing image data (e.g., including a representation of at least one feature of the user) received by a system associated with the current user input with stored image data (which includes representations of features of different users). User identification component 495 may perform additional user identification processes, including those known in the art.

[0097] User identification component 495 determines a score indicating whether user input originates from a specific user. For example, a first score may indicate the probability that user input originated from a first user, a second score may indicate the probability that user input originated from a second user, and so on. User identification component 495 also determines an overall confidence level regarding the accuracy of the user identification operation.

[0098] The output of user identification component 495 may include a single user identifier corresponding to the most likely user who initiated the user input. Alternatively, the output of user identification component 495 may include an N-best list of user identifiers, where the corresponding score indicates the likelihood that the corresponding user initiated the user input. The output of user identification component 495 can be used to provide information for NLU processing and processing performed by other components of the system.

[0099] System 100 (on device 110, system component 120, or a combination thereof) may include profile storage for storing various information related to individual users, user groups, devices, etc., who interact with the system. As used herein, a "profile" refers to a set of data associated with a user, user group, device, etc. Profile data may include user, device, etc., specific preferences; device input and output capabilities; internet connectivity information; user bibliographic information; subscription information, and other information.

[0100] Profile storage 470 may include one or more user profiles, each associated with a different user identifier / user profile identifier. Each user profile may include various user identification data. Each user profile may also include data corresponding to the user's preferences. Each user profile may also include user preferences and / or one or more device identifiers representing one or more devices of the user. For example, a user account may include one or more IP addresses, MAC addresses, and / or device identifiers (such as serial numbers) of each attached electronic device associated with the identified user account. When a user logs in to an application installed on device 110, the user profile (associated with the presented login information) may be updated to include information about device 110, such as an indication that the device is currently in use. Each user profile may include identifiers of skills that the user has enabled. When a user enables a skill, the user grants permission to system component 120 to allow the skill to be executed relative to the user's natural language user input. If the user has not enabled a skill, system component 120 may not invoke the skill to execute relative to the user's natural language user input.

[0101] Profile storage 470 may include one or more group profiles. Each group profile may be associated with a different group identifier. A group profile may be specific to a group of users. That is, a group profile may be associated with two or more individual user profiles. For example, a group profile may be a family profile associated with user profiles associated with multiple users in a single household. A group profile may include preferences shared by all user profiles associated with it. Each user profile associated with a group profile may additionally include preferences specific to the user associated with it. That is, each user profile may include preferences unique to one or more other user profiles associated with the same group profile. User profiles may be independent profiles or may be associated with group profiles.

[0102] Profile storage 470 may include one or more device profiles. Each device profile may be associated with a different device identifier. Each device profile may include various device identification information. Each device profile may also include one or more user identifiers representing one or more users associated with the device. For example, a profile for a home device may include user identifiers for home users.

[0103] Figure 5 and Figure 6 This illustrates how NLU component 460 can perform NLU processing. Figure 5 This is a conceptual diagram illustrating how natural language processing is performed according to the embodiments of this disclosure. Additionally, Figure 6 This is a conceptual diagram illustrating how natural language processing is performed according to the implementation scheme of this disclosure.

[0104] Figure 5 This illustrates how NLU processing can be performed on text data. NLU component 460 can process text data that includes several ASR hypotheses from a single user input. For example, if ASR component 450 outputs text data that includes an n-best list of ASR hypotheses, then NLU component 460 can process the text data relative to all (or a subset of) the ASR hypotheses represented in the text data.

[0105] The NLU component 460 can annotate text data by parsing and / or tagging it. For example, for the text data "tell me the weather for Seattle", the NLU component 460 can tag "tell me the weather for Seattle" as... <outputweather>The intention was to identify "Seattle" as the location for weather information.

[0106] NLU component 460 may include candidate list component 550. Candidate list component 550 selects skills (e.g., applications executable relative to user input) that are performed relative to ASR output data 610 input to NLU component 460. ASR output data 610 (also referred to as ASR data 610) may include textual representations of utterances, such as words, sub-word units, etc. Therefore, candidate list component 550 limits the downstream, more resource-intensive NLU processes to be performed relative to the skills executable relative to user input.

[0107] In the absence of candidate list component 550, NLU component 460 can process ASR output data 610 input to it in parallel, serially, or using some combination thereof with respect to each skill in the system. By implementing candidate list component 550, NLU component 460 can process ASR output data 610 only with respect to skills that can be performed relative to user input. This reduces the overall computational power and latency attributable to NLU processing.

[0108] Candidate list component 550 may include one or more training models. These trainable models can identify various forms of user input that system component 120 can receive. For example, during a training period, a skill system 425 associated with a skill may provide training text data to system component 120, representing sample user input that a user might provide to invoke the skill. For example, for a carpooling skill, the skill system 425 associated with the carpooling skill may provide training text data to system component 120, including text corresponding to phrases such as "get me a cab to [location]", "get me a ride to [location]", "book me a cab to [location]", and "book me aride to [location]". The training text data representing sample user input can be used to train one or more training models that candidate list component 550 will use to determine other potentially relevant user input structures that a user might attempt to use to invoke a particular skill. During training, system component 120 may solicit opinions from skill system 425 associated with the skill regarding whether other determined user input structures are permissible for invoking the skill from the perspective of skill system 425. Alternative user input structures may be derived from one or more training models during model training and / or may be based on user input structures provided by different skills. Skill system 425 associated with a specific skill may also provide system component 120 with training text data indicating syntax and annotations. System component 120 may use training text data representing sample user input, determined relevant user input, syntax, and annotations to train a model that indicates, at least in part, when user input is likely to point to / be handled by a skill, based on the structure of the user input. Each training model of candidate list component 550 may be trained relative to a different skill. Alternatively, candidate list component 550 may use one training model per domain, such as one training model for a skill associated with the weather domain, one training model for a skill associated with the carpooling domain, etc.

[0109] System component 120 can use sample user input provided by skill system 425 and potentially relevant sample user input determined during training as binary examples to train a model associated with the skills associated with skill system 425. Candidate list component 550 can then operate the model associated with a specific skill at runtime. For example, some sample user inputs may be positive examples (e.g., user input that can be used to invoke a skill). Other sample user inputs may be negative examples (e.g., user input that may not be usable to invoke a skill).

[0110] As described above, candidate list component 550 may include different training models for each skill of the system, different training models for each domain, or some other combination of training models. For example, candidate list component 550 may alternatively include a single model. A single model may include a portion trained relative to features shared by all skills of the system (e.g., semantic features). A single model may also include skill-specific portions, where each skill-specific portion is trained relative to a specific skill of the system. Implementing a single model with skill-specific portions can result in less latency than implementing different training models for each skill, because a single model with skill-specific portions limits the number of features processed at each skill level.

[0111] The portion trained relative to features shared by more than one skill can be clustered based on domain. For example, the first portion trained relative to multiple skills can be trained relative to weather domain skills, the second portion trained relative to multiple skills can be trained relative to music domain skills, and the third portion trained relative to multiple skills can be trained relative to travel domain skills, and so on.

[0112] Clustering may not be beneficial in every situation because it allows the candidate list component 550 to output an indication of only a portion of the skills that the ASR output data 610 may involve. For example, user input may correspond to "tell me about Tom Collins." If the model clusters based on domain, the candidate list component 550 may determine that the user input corresponds to a recipe skill (e.g., a beverage recipe), even if the user input may also correspond to an information skill (e.g., including information about a person named Tom Collins).

[0113] NLU component 460 may include one or more recognizers 563. In at least some embodiments, the recognizer 563 may be associated with skill system 425 (e.g., the recognizer may be configured to interpret text data to correspond to skill system 425). In at least some other examples, the recognizer 563 may be associated with domains such as smart home, video, music, weather, customs (e.g., the recognizer may be configured to interpret text data to correspond to that domain).

[0114] If the candidate list component 550 determines that the ASR output data 610 is potentially associated with multiple domains, then the identifiers 563 associated with these domains can process the ASR output data 610, while identifiers 563 not indicated in the output of the candidate list component 550 may not process the ASR output data 610. Identifiers 563 "included in the candidate list" can process the ASR output data 610 in parallel, serial, partially parallel, or other modes. For example, if the ASR output data 610 is potentially associated with both the communications domain and the music domain, then the identifier associated with the communications domain can process the ASR output data 610 in parallel or partially parallel with the identifier associated with the music domain.

[0115] Each recognizer 563 may include a Named Entity Recognition (NER) component 562. The NER component 562 attempts to identify grammatical and lexical information that can be used to interpret the meaning relative to the text data input therein. The NER component 562 identifies portions of the text data corresponding to named entities associated with a domain, which is associated with the recognizer 563 implementing the NER component 562. The NER component 562 (or other components of the NLU component 460) may also determine whether a word refers to an entity in the text data whose identity is not explicitly mentioned, such as "he," "she," "it," or other anaphoric or extrinsic terms.

[0116] Each identifier 563, and more specifically each NER component 562, may be associated with a specific grammar database 576, a specific set of intents / actions 574, and a specific personalized dictionary 586. The grammar database 576 and intents / actions 574 may be stored in NLU storage 573. Each geodetic dictionary 584 may include domain / skill index vocabulary information associated with a specific user and / or device 110. For example, geodetic dictionary A (584a) includes skill index vocabulary information 586aa to 586an. For example, a user's music domain vocabulary information may include album names, artist names, and song titles, while a user's communications domain vocabulary information may include contact names. This is because each user's music collection and contact list are presumably different. This personalization improves entity resolution performed later.

[0117] NER component 562 uses syntactic information 576 and lexical information 586 associated with the domain (associated with the recognizer 563 implementing NER component 562) to determine references to one or more entities in the text data. In this way, NER component 562 identifies "slots" (each slot corresponding to one or more specific words in the text data) that can be useful for later processing. NER component 562 can also label each slot with a type (e.g., noun, location, city, artist name, song title, etc.).

[0118] Each grammar database 576 includes entity names (i.e., nouns) that are frequently present in speech related to a specific domain associated with the grammar database 576, while vocabulary information 586 is personalized for user input initiated by the user and / or device 110. For example, a grammar database 576 associated with the shopping domain may include a database of words commonly used when people discuss shopping.

[0119] The downstream process known as entity resolution (discussed in detail elsewhere in this document) links slots in text data to specific entities known to the system. To perform entity resolution, the NLU component 460 may utilize geodetic dictionary information (584a-584n) stored in the entity library store 582. The geodetic dictionary information 584 can be used to match text data (representing a portion of user input) with text data representing known entities (such as song titles, contact names, etc.). The geodetic dictionary 584 may be linked to a user (e.g., a specific geodetic dictionary may be associated with a specific user's music collection), linked to a domain (e.g., shopping, music, video, etc.), or organized in various other ways.

[0120] Each recognizer 563 may also include an intent classification (IC) component 564. The IC component 564 parses text data to determine intents that potentially represent user input (associated with the domain to which the recognizer 563 implementing the IC component 564 is associated). An intent represents an action the user expects to perform. The IC component 564 may communicate with a database 574 of words linked to intents. For example, a music intent database may link words and phrases such as "quiet," "volume off," and "mute" to... <mute>Intent. IC component 564 identifies potential intents by comparing words and phrases (representing at least a portion of user input) in text data with words and phrases in intent database 574 (associated with the domain of the recognizer 563 that implements IC component 564).

[0121] The intent identifiable by a specific IC component 564 is linked to a domain-specific (i.e., the domain associated with the recognizer 563 that implements IC component 564) grammatical frame 576, which has "slots" to be filled. Each slot in the grammatical frame 576 corresponds to a portion of the text data that the system considers to correspond to an entity. For example, with <playmusic>The grammatical framework 576 corresponding to the intent can correspond to textual data sentence structures such as "Play {ArtistName}", "Play {Album Name}", "Play {Song Name}", and "Play {Song Name} by {Artist Name}". However, to make entity parsing more flexible, the grammatical framework 576 may not be constructed as a sentence, but rather based on associating slots with grammatical tags.

[0122] For example, before recognizing named entities in text data, NER component 562 can parse the text data based on grammatical rules and / or models to identify words as subjects, objects, verbs, prepositions, etc. IC component 564 (implemented by the same recognizer 563 as NER component 562) can use the recognized verbs to identify intent. Then, NER component 562 can determine the grammatical model 576 associated with the identified intent. For example, for use with... <playmusic>The corresponding intentional grammar model 576 can specify a list of slots applicable to the identified "objects" and any object modifiers (e.g., prepositional phrases), such as {artist name}, {album name}, {song name}, etc. Then, the NER component 562 can search the corresponding fields in the dictionary 586 (associated with the domain associated with the recognizer 563 implementing the NER component 562) to attempt to match the words and phrases in the text data previously marked as grammatical objects or object modifiers by the NER component 562 with the words and phrases identified in the dictionary 586.

[0123] NER component 562 performs semantic tagging, which involves labeling words or word combinations based on their type / semantic meaning. NER component 562 can parse text data using heuristic grammar rules, or the model can be constructed using techniques such as Hidden Markov Models, Maximum Entropy Models, Log-Linear Models, and Conditional Random Fields (CRF). For example, NER component 562, implemented using a music domain identifier, can parse text data corresponding to "play mother's little helper by the rolling stones" and tag it as {verb}: "Play", {object}: "mother's little helper", {object preposition}: "by", and {object modifier}: "the rolling stones". NER component 562 identifies "play" as a verb based on a word database associated with the music domain (also implemented by the music domain identifier). IC component 564 can determine that the verb is related to... <playmusic>The intent corresponds. At this stage, the meaning of "Mom's Little Helper" or "Rolling Stones" has not yet been determined, but based on grammatical rules and models, NER component 562 has determined that the text of these phrases is related to the grammatical objects (i.e., entities) of the user input represented in the text data.

[0124] NER component 562 can tag text data to give it meaning. For example, NER component 562 can tag "Mom's Little Helper Playing the Rolling Stones" as: {domain}Music, {intent} <playmusic>{artist name} Rolling Stones, {media type} Song, and {song title} Mother's Little Helper. For a further example, NER component 562 could tag "play songs by the Rolling Stones" as: {domain} Music, {intent} <playmusic>{Artist Name} The Rolling Stones and {Media Type} songs.

[0125] Candidate list component 550 can receive ASR output data 610 (e.g., from ASR component 450 or from device 110b) Figure 6 (As illustrated). ASR component 450 can embed ASR output data 610 into a form that can be processed by a trained model using sentence embedding techniques known in the art. Sentence embedding results in ASR output data 610 comprising text in a structure that enables the trained model of candidate list component 550 to operate on ASR output data 610. For example, the embedding of ASR output data 610 can be a vector representation of ASR output data 610.

[0126] Candidate list component 550 can make a binary determination (e.g., yes or no) of which domains are relevant to ASR output data 610. Candidate list component 550 can make such a determination using one or more trained models described above. If candidate list component 550 implements a single trained model for each domain, it can simply run the model associated with the enabled domain, as indicated in the user profile associated with the device 110 initiating user input and / or the user.

[0127] Candidate list component 550 can generate an n-best list data 615 representing the domains executable relative to the user input represented in ASR output data 610. The size of the n-best list represented in n-best list data 615 is configurable. In one example, n-best list data 615 may indicate each domain of the system, along with an indication for each domain regarding whether the user input represented in ASR output data 610 is likely to be executable in that domain. In another example, instead of indicating each domain of the system, n-best list data 615 may only indicate domains that are likely to be executable based on the user input represented in ASR output data 610. In yet another example, candidate list component 550 may implement thresholding such that n-best list data 615 may indicate a maximum number of domains that are executable based on the user input represented in ASR output data 610. In one example, the threshold number of domains that can be represented in n-best list data 615 is ten. In another example, the fields included in the n-best list data 615 may be limited by a score threshold, wherein only fields that indicate a higher probability of processing user input than a certain score (as determined by the candidate list component 550 processing ASR output data 610 relative to such fields) are included in the n-best list data 615.

[0128] ASR output data 610 may correspond to more than one ASR hypothesis. When this occurs, candidate list component 550 may output a different n-best list (represented in n-best list data 615) for each ASR hypothesis. Alternatively, candidate list component 550 may output a single n-best list that represents the domain associated with the multiple ASR hypotheses represented in ASR output data 610.

[0129] As indicated above, the candidate list component 550 can implement thresholding, such that the n-best list output by it may include no more than a threshold number of entries. If the ASR output data 610 includes more than one ASR hypothesis, the n-best list output by the candidate list component 550 may include no more than a threshold number of entries, regardless of the number of ASR hypotheses output by the ASR component 450. Alternatively or additionally, the n-best list output by the candidate list component 550 may include no more than a threshold number of entries for each ASR hypothesis (e.g., no more than five entries for the first ASR hypothesis, no more than five entries for the second ASR hypothesis, etc.).

[0130] In addition to making a binary determination of whether a domain is potentially related to ASR output data 610, candidate list component 550 may also generate a confidence score representing the probability that a domain is related to ASR output data 610. If candidate list component 550 implements a different training model for each domain, it may generate different confidence scores for each individual domain training model being run. If candidate list component 550 runs a model for each domain upon receiving ASR output data 610, it may generate different confidence scores for each domain of the system. If candidate list component 550 runs only the models for domains associated with skills indicated as enabled in the user profile of the device 110 and / or the user associated with the user input, it may generate different confidence scores only for each domain associated with at least one enabled skill. If candidate list component 550 implements a single training model with a domain-specific training portion, it may generate different confidence scores for each domain running the domain-specific training portion. The candidate list component 550 can perform matrix-vector modifications to obtain confidence scores for all domains of the system in a single instance of processing ASR output data 610.

[0131] The N-best list data 615, including confidence scores, which can be output by the candidate list component 550, can be represented as follows: Search domain, 0.67 In the realm of recipes, 0.62 Information field, 0.57 In the shopping sector, 0.42 As indicated, the confidence score output by the candidate list component 550 can be a numerical value. Alternatively, the confidence score output by the candidate list component 550 can be a gradation value (e.g., high, medium, low).

[0132] The optimal list may include only entries for domains that have confidence scores that satisfy (e.g., equal to or greater than) the minimum threshold confidence score. Alternatively, candidate list component 550 may include entries for all domains associated with the user-enabled skill, even if one or more of the domains are associated with confidence scores that do not satisfy the minimum threshold confidence score.

[0133] In determining which domains might be associated with the user input represented in the ASR output data 610 and their corresponding confidence scores, the candidate list component 550 may consider additional data 620. Additional data 620 may include usage history data associated with the device 110 and / or the user initiating the user input. For example, if user input initiated by device 110 and / or the user frequently invokes a particular domain, the confidence score for that domain may be higher. Conversely, if user input initiated by device 110 and / or the user rarely invokes a particular domain, the confidence score for that domain may be lower. Therefore, additional data 620 may include indicators of the user associated with the ASR output data 610, such as those determined by the user identification component 495.

[0134] Other data 620 may be embedded with characters before being input into the candidate list component 550. Alternatively, other techniques known in the art may be used to embed other data 620 before it is input into the candidate list component 550.

[0135] Other data 620 may also include data indicating the domains associated with the device 110 that initiated the user input and / or the skills enabled by the user. The candidate list component 550 may use such data to determine which domain-specific training models to run. That is, the candidate list component 550 may determine to run only training models associated with the domains associated with the user-enabled skills. Alternatively, the candidate list component 550 may use such data to modify the domain confidence scores.

[0136] As an example, consider two domains: a first domain associated with at least one enabled skill, and a second domain not associated with any user-enabled skills of the user who initiated the user input. Candidate list component 550 may run a first model specific to the first domain and a second model specific to the second domain. Alternatively, candidate list component 550 may run a model configured to determine a score for each of the first and second domains. Candidate list component 550 may initially determine the same confidence score for each of the first and second domains. Then, candidate list component 550 may modify those confidence scores based on which domains are associated with at least one currently enabled skill of the user. For example, candidate list component 550 may increase the confidence score associated with the domain associated with at least one enabled skill while keeping the confidence score associated with the other domain unchanged. Alternatively, candidate list component 550 may keep the confidence score associated with the domain associated with at least one enabled skill unchanged while decreasing the confidence score associated with the other domain. In addition, the candidate list component 550 can increase the confidence score associated with a domain that is associated with at least one enabled skill, and decrease the confidence score associated with another domain.

[0137] As indicated, the user profile may indicate which skills the corresponding user has enabled (e.g., authorized to use data associated with the user to perform actions). Such indications may be stored in profile storage 470. When the candidate list component 550 receives ASR output data 610, the candidate list component 550 may determine whether the profile data associated with the user and / or device 110 that initiated the command includes indications of enabled skills.

[0138] Other data 620 may also include data indicating the type of device 110. Device type can indicate the output capabilities of the device. For example, the device type may correspond to a device with a visual display, a headless (e.g., displayless) device, whether the device is mobile or stationary, whether the device includes audio playback capabilities, whether the device includes a camera, other device hardware configurations, etc. Candidate list component 550 may use such data to determine which domain-specific training models to run. For example, if device 110 corresponds to a displayless type device, candidate list component 550 may determine that domain-specific training models specific to the output video data should not be run. Candidate list component 550 may alternatively use such data to modify the domain confidence score.

[0139] As an example, consider two domains, one outputting audio data and the other outputting video data. Candidate list component 550 can run a first model specific to the domain generating the audio data and a second model specific to the domain generating the video data. Alternatively, candidate list component 550 can run a model configured to determine a score for each domain. Candidate list component 550 can initially determine the same confidence score for each domain. Then, candidate list component 550 can modify the original confidence scores based on the type of device 110 that initiated the user input corresponding to the ASR output data 610. For example, if device 110 is a displayless device, candidate list component 550 can increase the confidence score associated with the domain generating the audio data while maintaining the confidence score associated with the domain generating the video data. Alternatively, if device 110 is a displayless device, candidate list component 550 can maintain the confidence score associated with the domain generating the audio data while decreasing the confidence score associated with the domain generating the video data. Furthermore, if device 110 is a displayless device, candidate list component 550 may increase the confidence score associated with the domain of generated audio data and decrease the confidence score associated with the domain of generated video data.

[0140] The device type information represented in Other Data 620 may indicate the output capability of a device used to output content to a user, which may not necessarily be the device initiating the user input. For example, a user may input verbal user input corresponding to "play Game of Thrones" into a device that does not include a display. The system may determine that a smart TV or other display device (associated with the same user profile) is used to output Game of Thrones. Therefore, Other Data 620 may represent a smart TV, rather than a displayless device that captures verbal user input.

[0141] Other data 620 may also include data indicating the speed, location, or other mobility information of the device initiating the user input. For example, the device may correspond to a vehicle including a display. If the vehicle is moving, the candidate list component 550 may lower the confidence score associated with the area where the video data was generated, because outputting video content to the user while the user is driving may be undesirable. The device may output data to system component 120 indicating when the device is moving.

[0142] Other data 620 may also include data indicating the domain currently invoked. For example, a user may state a first (e.g., previous) user input that causes the system to invoke the music domain skill to output music to the user. While the system is outputting music to the user, it may receive a second (e.g., current) user input. The candidate list component 550 may use such data to modify the confidence score for a domain. For example, the candidate list component 550 may run a first model specific to a first domain and a second model specific to a second domain. Alternatively, the candidate list component 550 may run a model configured to determine a score for each domain. The candidate list component 550 may also first determine the same confidence score for each domain. Then, the candidate list component 550 may modify the original confidence score based on the first domain invoked to cause the system to output content when the current user input is received. Based on the invoked first domain, the candidate list component 550 may (i) increase the confidence score associated with the first domain while keeping the confidence score associated with the second domain unchanged, (ii) keep the confidence score associated with the first domain unchanged while decreasing the confidence score associated with the second domain, or (iii) increase the confidence score associated with the first domain and decrease the confidence score associated with the second domain.

[0143] The thresholding process implemented relative to the n-best list data 615 generated by the candidate list component 550, and the different types of additional data 620 considered by the candidate list component 550, are configurable. For example, the candidate list component 550 can update the confidence score as more additional data 620 is considered. As a further example, if thresholding is implemented, the n-best list data 615 can exclude relevant domains. Thus, for example, the candidate list component 550 can include indications of domains in the n-best list 615 unless the candidate list component 550 is 100% certain that the domain may not be able to perform the user input represented in the ASR output data 610 (e.g., the candidate list component 550 determines that the confidence score for that domain is zero).

[0144] Candidate list component 550 may send ASR output data 610 to an identifier 563 associated with the domains represented in the n-best list data 615. Alternatively, candidate list component 550 may send some other indicator of the n-best list data 615 or a selected subset of the domains to another component (such as orchestrator component 430), which in turn may send ASR output data 610 to an identifier 563 corresponding to the domains included in the n-best list data 615 or otherwise indicated in the indicator. If candidate list component 550 generates an n-best list representing domains without any associated confidence scores, then candidate list component 550 / orchestrator component 430 may send ASR output data 610 to an identifier 563 associated with the domains for which candidate list component 550 determines user input is executable. If the candidate list component 550 generates an n-best list representing the domains with associated confidence scores, then the candidate list component 550 / arranger component 430 can send ASR output data 610 to the identifier 563 associated with the domains that are associated with confidence scores that meet (e.g., reach or exceed) a threshold minimum confidence score.

[0145] The identifier 563 can output labeled text data generated by the NER component 562 and the IC component 564, as described above herein. The NLU component 460 can compile the labeled text data output by the identifier 563 into a single cross-domain n-best list 640, and can send the cross-domain n-best list 640 to the pruning component 650. Each entry of the labeled text represented in the cross-domain n-best list data 640 (e.g., each NLU hypothesis) can be associated with a corresponding score indicating the probability that the NLU hypothesis corresponds to the domain associated with the identifier 563 that outputs the NLU hypothesis. For example, the cross-domain n-best list data 640 can be represented as (where each row corresponds to a different NLU hypothesis): [0.95] Intent: <playmusic>Artist: Beethoven; Song: Waldstein Sonata [0.70] Intent: <playvideo>Artist Name: Beethoven Video Title: Waldstein Sonata [0.01] Intent: <playmusic>Artist: Beethoven Album: Waldstein Sonata [0.01] Intent: <playmusic>Song Title: Waldstein Sonata The pruning component 650 can sort NLU hypotheses according to their corresponding scores represented in the cross-domain best-of-n list data 640. The pruning component 650 can perform score thresholding relative to the cross-domain NLU hypotheses. For example, the pruning component 650 can select NLU hypotheses associated with scores that meet (e.g., reach and / or exceed) a threshold score. The pruning component 650 can also, or alternatively, perform NLU hypothesis number thresholding. For example, the pruning component 650 can select the highest-scoring NLU hypothesis. The pruning component 650 can output a portion of the NLU hypotheses input therein. The purpose of the pruning component 650 is to create a streamlined list of NLU hypotheses so that downstream, more resource-intensive processes can operate only on the NLU hypotheses most likely representing the user's intent.

[0146] NLU component 460 may include lightweight slot filler component 652. Lightweight slot filler component 652 may take text from slots represented in the NLU hypotheses output by trimming component 650 and modify it to make the text easier for downstream components to process. Lightweight slot filler component 652 may perform low-latency operations that do not involve heavyweight operations such as referencing a knowledge base (e.g., 572). The purpose of lightweight slot filler component 652 is to replace words with other words or values ​​that might be more easily understood by downstream components. For example, if the NLU hypothesis includes the word "tomorrow," lightweight slot filler component 652 may replace the word "tomorrow" with an actual date for downstream processing purposes. Similarly, lightweight slot filler component 652 may replace the word "CD" with "album" or the word "compact disc." The replaced words are then included in cross-domain n-optimal list data 660.

[0147] Cross-domain best-in-class list data 660 can be input into entity parsing component 670. Entity parsing component 670 can apply rules or other instructions to normalize tags or lexical units from previous stages into intent / slot representations. The exact transformation can depend on the domain. For example, for the travel domain, entity parsing component 670 can transform the text corresponding to "Boston airport" into the standard BOS three-letter code referring to that airport. Entity parsing component 670 can reference a knowledge base (e.g., 572) that specifically identifies the exact entity mentioned in each slot of each NLU assumption represented in the cross-domain best-in-class list data 660. Specific intent / slot combinations may also be bound to a specific source, which can then be used to parse the text. In the example "play songs by the stones," entity parsing component 670 could reference a personal music catalog, an Amazon Music account, a user profile, etc. Entity resolution component 670 may output a modified list of n-bests, which is based on the cross-domain list of n-bests 660 but includes more detailed information about the specific entities mentioned in the slots (e.g., entity IDs) and / or more detailed slot data that can ultimately be used by the skill. NLU component 460 may include multiple entity resolution components 670, and each entity resolution component 670 may be specific to one or more domains.

[0148] NLU component 460 may include reorderer 690. Reorderer 690 may assign a specific confidence score to each NLU hypothesis input therein. The confidence score of a particular NLU hypothesis may be affected by whether the NLU hypothesis has unfilled slots. For example, if an NLU hypothesis includes all filled / resolved slots, that NLU hypothesis may be assigned a higher confidence score than another NLU hypothesis that includes at least some unfilled / unresolved slots of entity parsing component 670.

[0149] The reorderer 690 may apply score recalculation, biasing, or other techniques. The reorderer 690 may consider not only the data output by the entity parsing component 670, but also other data 691. Other data 691 may include various information. For example, other data 691 may include skill ratings or popularity data. For instance, if a skill has a high rating, the reorderer 690 may increase the score of the NLU hypothesis that the skill can handle. Other data 691 may also include information about skills that the user initiating the user input has enabled. For instance, the reorderer 690 may assign higher scores to NLU hypotheses that are enabled than those that are not. Other data 691 may also include data indicating the user's usage history, such as whether the user initiating the user input uses a particular skill regularly or at a specific time of day. Other data 691 may additionally include data indicating date, time, location, weather, device 110 type, user identifier, context, and other information. For example, the reorderer 690 may consider whether any particular skill is currently active (e.g., music being played, a game being played, etc.).

[0150] As illustrated and described, entity resolution component 670 is implemented prior to reorderer 690. Alternatively, entity resolution component 670 may be implemented after reorderer 690. Implementing entity resolution component 670 after reorderer 690 restricts the NLU hypotheses processed by entity resolution component 670 to only those hypotheses that successfully passed reorderer 690.

[0151] Reorderer 690 can be a global reorderer (e.g., a reorderer not specific to any particular domain). Alternatively, NLU component 460 can implement one or more domain-specific reorderers. Each domain-specific reorderer can reorder the NLU hypotheses associated with the domain. Each domain-specific reorderer can output an n-best list of the reordered hypotheses (e.g., 5-10 hypotheses).

[0152] NLU component 460 can perform the skills described above relative to those fully implemented as part of system component 120 (e.g., in...). Figure 4 The NLU processing described above is associated with the domain description of a skill (indicated by 490). NLU component 460 can independently perform the NLU processing described above with respect to the domain description associated with a skill that is at least partially implemented as part of skill system 425. In the example, candidate list component 550 can process only with respect to these latter domains. The results of these two NLU processing paths can be combined into NLU output data 685, which can be sent to NLU post-sorter 665, which can be implemented by system component 120.

[0153] NLU post-sorter 665 may include a statistical component that generates a ranked list of intent / skill pairs with associated confidence scores. Each confidence score may indicate the adequacy of the execution of the intent pair relative to the NLU outcome data associated with the skill. NLU post-sorter 665 may operate one or more training models configured to process NLU outcome data 685, skill outcome data 630, and other data 620 to output ranked output data 625. The ranked output data 625 may include an n-best list, wherein the NLU assumptions in the NLU outcome data 685 are reordered such that the n-best list in the ranked output data 625 represents a list of priorities for skills responding to user input as determined by NLU post-sorter 665. The ranked output data 625 may also (as part of the n-best list or otherwise) include individual corresponding scores for each skill, wherein each score indicates the probability that the skill (and / or its corresponding outcome data) corresponds to the user input.

[0154] The system can be configured with thousands or tens of thousands of skills. The NLU post-sorter 665 enables the system to better determine the best skill to execute the user input. For example, the first and second NLU hypotheses in the NLU result data 685 may correspond substantially to each other (e.g., their scores may be very similar), even if the first NLU hypothesis can be handled by the first skill and the second NLU hypothesis can be handled by the second skill. The first NLU hypothesis may be associated with a first confidence score, which indicates the confidence of the system relative to the NLU processing performed to generate the first NLU hypothesis. Furthermore, the second NLU hypothesis may be associated with a second confidence score, which indicates the confidence of the system relative to the NLU processing performed to generate the second NLU hypothesis. The first confidence score may be similar to or the same as the second confidence score. The first and / or second confidence scores can be numerical values ​​(e.g., from 0.0 to 1.0). Alternatively, the first and / or second confidence scores can be gradations (e.g., low, medium, high).

[0155] NLU post-sequencer 665 (or other scheduling components, such as orchestrator component 430) may solicit first and second skills based on a first NLU hypothesis and a second NLU hypothesis, respectively, to provide potential outcome data 630. For example, NLU post-sequencer 665 may send the first NLU hypothesis along with a request requiring the first skill 490a to be executed at least partially with respect to the first NLU hypothesis to the first skill 490a. NLU post-sequencer 665 may also send the second NLU hypothesis along with a request requiring the second skill 490b to be executed at least partially with respect to the second NLU hypothesis to the second skill 490b. NLU post-sequencer 665 receives from the first skill 490a the first outcome data 630a generated according to the execution of the first skill 490a with respect to the first NLU hypothesis. NLU post-sequencer 665 also receives from the second skill 490b the second outcome data 630b generated according to the execution of the second skill 490b with respect to the second NLU hypothesis.

[0156] The result data 630 may include various components. For example, the result data 630 may include content to be output to the user (e.g., audio data, text data, and / or video data). The result data 630 may also include unique identifiers used by system component 120 and / or skill system 425 to locate the data to be output to the user. The result data 630 may also include instructions. For example, if the user input corresponds to "turn on the light," the result data 630 may include instructions to cause the system to turn on the light associated with the device (110a / 110b) and / or the user's profile.

[0157] The NLU post-sorter 665 may consider first result data 630a and second result data 630b to modify the first and second confidence scores of the first and second NLU hypotheses, respectively. That is, the NLU post-sorter 665 may generate a third confidence score based on the first result data 630a and the first confidence score. The third confidence score may correspond to the probability that the NLU post-sorter 665 determines the first skill will correctly respond to the user input. The NLU post-sorter 665 may also generate a fourth confidence score based on the second result data 630b and the second confidence score. Those skilled in the art will understand that a first difference between the third and fourth confidence scores may be greater than a second difference between the first and second confidence scores. The NLU post-sorter 665 may also consider other data 620 to generate the third and fourth confidence scores. While it has been described that the NLU post-ranker 665 can modify the confidence scores associated with the first and second NLU hypotheses, those skilled in the art will understand that the NLU post-ranker 665 can modify the confidence scores of more than two NLU hypotheses. The NLU post-ranker 665 can select, in response to the current user input, the resulting data 630 associated with the skill 490 having the highest modified confidence score as the data output. The NLU post-ranker 665 can also consider ASR output data 610 to modify the NLU hypothesis confidence scores.

[0158] The orchestrator component 430 can associate the intent in the NLU hypothesis with the skill 490 before sending the NLU result data 685 to the NLU post-sorter 665. For example, if the NLU hypothesis includes <playmusic>If the intention is clear, then the orchestrator component 430 can combine NLU assumptions with executable... <playmusic>One or more skills 490 are associated with the intent. Therefore, the orchestrator component 430 can send NLU result data 685, including NLU hypotheses paired with skills 490, to the NLU post-sequencer 665. In response to ASR output data 610 corresponding to "what should I do for dinner today," the orchestrator component 430 can generate pairs of skills 490 and associated NLU hypotheses corresponding to the following: Skill 1 / NLU assumptions include <help>intention Skill 2 / NLU assumptions include <order>intention Skill 3 / NLU assumptions include <dishtype>intention The NLU post-sorter 665 queries each skill 490 that is paired with an NLU hypothesis in the NLU output data 685 to provide result data 630 based on the NLU hypothesis associated with that skill. That is, with respect to each skill, the NLU post-sorter 665 verbally asks each skill, "If given this NLU hypothesis, what would you do with it?" Based on the example above, the NLU post-sorter 665 can send the following data to skill 490: Skill 1: The first NLU assumption includes <help>Intent indicator Skill 2: The second NLU assumption includes <order>Intent indicator Skill 3: The third NLU assumption includes <dishtype>Intent indicator The NLU post-sorter 665 can query each of the skills 490 in parallel or substantially in parallel.

[0159] In response to NLU post-sorter 665 requesting skill 490 for result data 630, skill 490 may provide NLU post-sorter 665 with various data and instructions. Skill 490 may simply provide NLU post-sorter 665 with an indication of whether the skill is executable relative to its received NLU hypothesis. Skill 490 may also, or alternatively, provide NLU post-sorter 665 with output data generated based on its received NLU hypothesis. In some cases, skill 490 may require further information beyond what is represented in the received NLU hypothesis to provide output data in response to user input. In these cases, skill 490 may provide NLU post-sorter 665 with result data 630 indicating which slots in the frame skill 490 further needs to be filled or which entities skill 490 further needs to be resolved, after which skill 490 can provide result data 630 in response to user input. Skill 490 may also provide the NLU post-sequencer 665 with instructions and / or computer-generated speech on how the system should solicit further information required by skill 490. Skill 490 may also provide the NLU post-sequencer 665 with instructions on whether skill 490 has all the required information after the user provides additional information once, or whether skill 490 requires the user to provide various additional information before skill 490 has all the required information. Based on the above example, skill 490 may provide the following to the NLU post-sequencer 665: Skill 1: The instruction indicates that this skill can be relative to, including <help>NLU assumptions for intent indicators Skill 2: The indicator suggests that the skill requires the system to obtain further information. Skill 3: The instruction indicates that the skill can respond to, including <dishtype>The third NLU assumption of the intent indicator provides many results. The result data 630 includes instructions provided by skill 490 regarding whether skill 490 is executable relative to the NLU hypothesis; data generated by skill 490 based on the NLU hypothesis; and instructions provided by skill 490 indicating that skill 490 requires further information beyond what is represented in the received NLU hypothesis.

[0160] The NLU post-sorter 665 uses the result data 630 provided by skill 490 to modify the NLU processing confidence scores generated by the re-sorter 690. That is, the NLU post-sorter 665 uses the result data 630 provided by the queried skill 490 to produce a larger difference between the NLU processing confidence scores generated by the re-sorter 690. Without the NLU post-sorter 665, the system may not have sufficient confidence to determine the output in response to user input, for example, when the NLU assumptions associated with multiple skills are too close for the system to confidently determine to invoke a single skill 490 in response to user input. For example, if the system does not implement the NLU post-sorter 665, the system may not be able to determine whether to obtain output data from a general reference information skill or a medical information skill in response to user input corresponding to "what is acne".

[0161] The NLU post-ranker 665 may prefer skill 490 that provides result data 630 in response to an NLU hypothesis, rather than skill 490 that provides result data 630 corresponding to an indication that further information is needed, and skill 490 that provides result data 630 indicating that these skills can provide multiple responses to the received NLU hypothesis. For example, based on a first skill 490a providing result data 630a including a response to an NLU hypothesis, the NLU post-ranker 665 may generate a first score for the first skill 490a that is higher than the NLU confidence score of the first skill. As a further example, based on a second skill 490b providing result data 630b indicating that the second skill 490b needs further information to provide a response to an NLU hypothesis, the NLU post-ranker 665 may generate a second score for the second skill 490b that is lower than the NLU confidence score of the second skill. Furthermore, based on the third skill 490c providing result data 630c indicating that the third skill 490c can provide multiple responses to the NLU hypothesis, the NLU post-sorter 665 can generate a third score for the third skill 490c, which is lower than the NLU confidence score of the third skill.

[0162] The NLU post-ranker 665 may consider additional data 620 when determining scores. This additional data 620 may include the ranking associated with the queried skill 490. The ranking may be a system ranking or a user-specific ranking. The ranking may indicate the veracity of a skill from the perspective of one or more users of the system. For example, based on the association of a first skill 490a with a high ranking, the NLU post-ranker 665 may generate a first score for the first skill 490a that is higher than the NLU processing confidence score for the first skill. As a further example, based on the association of a second skill 490b with a low ranking, the NLU post-ranker 665 may generate a second score for the second skill 490b that is lower than the NLU processing confidence score for the second skill.

[0163] Other data 620 may include information indicating whether the user who initiated the user input has enabled one or more of the skills 490 in the query. For example, based on the fact that a first skill 490a is enabled by the user who initiated the user input, the NLU post-sorter 665 may generate a first score for the first skill 490a, which is higher than the NLU processing confidence score for the first skill. As a further example, based on the fact that a second skill 490b is not enabled by the user who initiated the user input, the NLU post-sorter 665 may generate a second score for the second skill 490b, which is lower than the NLU processing confidence score for the second skill. When the NLU post-sorter 665 receives the NLU result data 685, the NLU post-sorter 665 may determine whether the profile data associated with the user and / or device who initiated the user input includes an indication of skill activation.

[0164] Other data 620 may include information indicating the output capabilities of the device, which will be used to output content to the user in response to user input. The system may include devices that include speakers but not displays, devices that include displays but not speakers, and devices that include both speakers and displays. If the device that outputs content in response to user input includes one or more speakers but not a display, the NLU post-sequencer 665 may increase the NLU processing confidence score associated with a first skill configured to output audio data and / or decrease the NLU processing confidence score associated with a second skill configured to output visual data (e.g., image data and / or video data). If the device that outputs content in response to user input includes a display but not one or more speakers, the NLU post-sequencer 665 may increase the NLU processing confidence score associated with a first skill configured to output visual data and / or decrease the NLU processing confidence score associated with a second skill configured to output audio data.

[0165] Other data 620 may include information indicating the confidence level of the result data 630 provided by skill 490. For example, if a user says "tell me a recipe for pasta sauce," then first skill 490a may provide NLU post-ranker 665 with first result data 630a corresponding to a first recipe associated with a five-star rating, and second skill 490b may provide NLU post-ranker 665 with second result data 630b corresponding to a second recipe associated with a one-star rating. In this case, NLU post-ranker 665 may increase the NLU processing confidence score associated with first skill 490a based on the first result data 630a associated with a five-star rating provided by first skill 490a and / or decrease the NLU processing confidence score associated with second skill 490b based on the second result data 630b associated with a one-star rating provided by second skill 490b.

[0166] Other data 620 may include information indicating the type of device initiating the user input. For example, if the device is located in a hotel room, the device may correspond to the "hotel room" type. If the user inputs a command corresponding to "order me food" into a device located in a hotel room, the NLU post-sorter 665 may increase the NLU processing confidence score associated with a first skill 490a, which corresponds to a room service skill associated with the hotel, and / or decrease the NLU processing confidence score associated with a second skill 490b, which corresponds to a food skill not associated with the hotel.

[0167] Other data 620 may include information indicating the location of the device and / or user initiating the user input. The system may be configured with skills 490 that operate only relative to certain geographic locations. For example, a user may provide user input corresponding to "when is the next train to Portland." First skill 490a may operate relative to trains arriving in, departing from, and passing through Portland, Oregon. Second skill 490b may operate relative to trains arriving in, departing from, and passing through Portland, Maine. If the device and / or user initiating the user input is located in Seattle, Washington, the NLU post-sequencing unit 665 may increase the NLU processing confidence score associated with first skill 490a and / or decrease the NLU processing confidence score associated with second skill 490b. Similarly, if the device initiating the user input and / or the user is located in Boston, Massachusetts, the NLU post-sequencing unit 665 may increase the NLU processing confidence score associated with the second skill 490b and / or decrease the NLU processing confidence score associated with the first skill 490a.

[0168] Other data 620 may include information indicating the time of day. The system may be configured with skills 490 that operate relative to certain times of day. For example, a user may provide user input corresponding to "order food for me." A first skill 490a may generate first result data 630a corresponding to breakfast. A second skill 490b may generate second result data 630b corresponding to dinner. If system component 120 receives user input in the morning, NLU post-sorter 665 may increase the NLU processing confidence score associated with the first skill 490a and / or decrease the NLU processing confidence score associated with the second skill 490b. If system component 120 receives user input in the afternoon or evening, NLU post-sorter 665 may increase the NLU processing confidence score associated with the second skill 490b and / or decrease the NLU processing confidence score associated with the first skill 490a.

[0169] Other data 620 may include information indicating user preferences. The system may include multiple skills 490 configured to perform in substantially the same manner. For example, first skill 490a and second skill 490b may both be configured to order food from a corresponding restaurant. The system may (e.g., in profile storage 470) store user preferences associated with users providing user input to system component 120, indicating that the user prefers first skill 490a more than second skill 490b. Therefore, when a user provides user input that can be performed by both first skill 490a and second skill 490b, NLU post-sorter 665 may increase the NLU processing confidence score associated with first skill 490a and / or decrease the NLU processing confidence score associated with second skill 490b.

[0170] Other data 620 may include information indicating the system usage history associated with the user who initiated the user input. For example, the system usage history may indicate that the user input that invokes the first skill 490a is more frequent than the user input that invokes the second skill 490b. Based on this, if the current user input can be performed by both the first skill 490a and the second skill 490b, the NLU post-sequencing unit 665 may increase the NLU processing confidence score associated with the first skill 490a and / or decrease the NLU processing confidence score associated with the second skill 490b.

[0171] Other data 620 may include information indicating the speed at which the device 110 initiating the user input is moving. For example, the device 110 may be located in a moving vehicle, or may be a moving vehicle. When the device 110 is in motion, the system may prefer audio output over visual output to reduce the likelihood of distracting the user (e.g., a vehicle driver). Therefore, for example, if the moving speed of the device 110 initiating the user input is equal to or higher than a threshold speed (e.g., higher than the average user walking speed), the NLU post-sequencer 665 may increase the NLU processing confidence score associated with the first skill 490a that generates the audio data. The NLU post-sequencer 665 may also, or alternatively, decrease the NLU processing confidence score associated with the second skill 490b that generates image or video data.

[0172] Other data 620 may include information instructing skill 490 on how long it takes to provide result data 630 to NLU post-sorter 665. When NLU post-sorter 665 requests result data 630 from multiple skills 490, skills 490 may respond to queries at different rates. NLU post-sorter 665 may implement a delay budget. For example, if NLU post-sorter 665 determines that skill 490 responds to NLU post-sorter 665 within a threshold amount of time after receiving a query from NLU post-sorter 665, then NLU post-sorter 665 may increase the NLU processing confidence score associated with skill 490. Conversely, if NLU post-sorter 665 determines that skill 490 does not respond to NLU post-sorter 665 within a threshold amount of time after receiving a query from NLU post-sorter 665, then NLU post-sorter 665 may decrease the NLU processing confidence score associated with skill 490.

[0173] The NLU post-sorter 665 has been described as using additional data 620 to increase and decrease the NLU processing confidence scores associated with various skills 490 to which the NLU post-sorter 665 has requested result data. Alternatively, the NLU post-sorter 665 may use additional data 620 to determine which skills 490 to request result data from. For example, the NLU post-sorter 665 may use additional data 620 to increase and / or decrease the NLU processing confidence scores associated with skills 490 associated with the NLU result data 685 output by the NLU component 460. The NLU post-sorter 665 may select n number of changes to the highest-scoring NLU processing confidence scores. The NLU post-sorter 665 may then request result data 630 only from the skills 490 associated with the selected n number of NLU processing confidence scores.

[0174] As described, the NLU post-sequencer 665 may request result data 630 from all skills 490 associated with the NLU result data 685 output by the NLU component 460. Alternatively, the system component 120 may prefer result data 630 from skills fully implemented by the system component 120, rather than result data from skills at least partially implemented by the skill system 425. Thus, in the first case, the NLU post-sequencer 665 may request result data 630 only from skills associated with the NLU result data 685 and fully implemented by the system component 120. If no skill (fully implemented by the system component 120) provides result data 630 (indicating a data response to the NLU result data 685) to the NLU post-sequencer 665, the skill can perform user-input instructions, or instructions requiring further information, then the NLU post-sequencer 665 may request result data 630 only from skills associated with the NLU result data 685 and at least partially implemented by the skill system 425.

[0175] As indicated above, the NLU post-sequencer 665 may request result data 630 from multiple skills 490. If one of the skills 490 provides result data 630 indicating a response to the NLU hypothesis and other skills provide result data 630 indicating they cannot perform or they require further information, the NLU post-sequencer 665 may select the result data 630 indicating the response to the NLU hypothesis as the data to be output to the user. If more than one of the skills 490 provides result data 630 indicating a response to the NLU hypothesis, the NLU post-sequencer 665 may consider other data 620 to generate a modified NLU processing confidence score and select the result data 630 of the skill associated with the highest score as the data to be output to the user.

[0176] A system that does not implement an NLU post-sorter 665 may select the highest-scoring NLU hypothesis from the NLU result data 685. The system may send the NLU hypothesis along with a request for output data to its associated skill 490. In some cases, skill 490 may be unable to provide output data to the system. This causes the system to indicate to the user that it cannot process the user input, even if another skill associated with a lower-ranked NLU hypothesis can provide output data in response to the user input.

[0177] NLU post-sorter 665 reduces instances of the aforementioned situation. As described, NLU post-sorter 665 queries multiple skills associated with NLU result data 685 to provide result data 630 to NLU post-sorter 665 before NLU post-sorter 665 finally determines which skill 490 to invoke in response to user input. Some of the skills 490 may provide result data 630 indicating a response to the NLU assumption, while other skills 490 may provide result data 630 indicating that these skills cannot provide response data. Systems that do not implement NLU post-sorter 665 may select one of the skills 490 that cannot provide a response, while NLU post-sorter 665 selects only the skill 490 that provides result data corresponding to a response to NLU post-sorter 665, indicating that further information is needed or indicating that multiple responses can be generated.

[0178] The NLU post-sorter 665 may select the result data 630 associated with the skill 490 with the highest score for output to the user. Alternatively, the NLU post-sorter 665 may output sorted output data 625 indicating the sorting of skill 490 and its corresponding NLU post-sorter. Because the NLU post-sorter 665 receives the result data 630 potentially corresponding to a response to user input from skill 490 before selecting one of the skills or outputting the sorted output data 625, there is little or no delay from the time the skill provides the result data 630 to the time the system outputs a response to the user.

[0179] If the NLU post-sequencer 665 selects the resulting audio data to be output to the user and the system determines that the content should be output in an audible manner, then the NLU post-sequencer 665 (or another component of system component 120) can cause device 110a and / or device 110b to output audio corresponding to the resulting audio data. If the NLU post-sequencer 665 selects the resulting text data to be output to the user and the system determines that the content should be output in a visual manner, then the NLU post-sequencer 665 (or another component of system component 120) can cause device 110b to display text corresponding to the resulting text data. If the NLU post-sequencer 665 selects the resulting audio data to be output to the user and the system determines that the content should be output in a visual manner, then the NLU post-sequencer 665 (or another component of system component 120) can send the resulting audio data to ASR component 450. ASR component 450 can generate output text data corresponding to the resulting audio data. Then, system component 120 can cause device 110b to display text corresponding to the output text data. If the NLU post-sequencer 665 selects the resulting text data to output to the user and the system determines that the content should be output in an audible manner, the NLU post-sequencer 665 (or another component of system component 120) may send the resulting text data to the TTS component 480. The TTS component 480 may generate output audio data (corresponding to computer-generated speech) based on the resulting text data. Then, system component 120 may cause device 110a and / or device 110b to output audio corresponding to the output audio data.

[0180] As described, skill 490 can provide result data 630 indicating a response to user input, indicating that skill 490 needs more information to provide a response to user input, or indicating that skill 490 is unable to provide a response to user input. If skill 490 associated with the highest NLU post-sequencer score provides result data 630 indicating a response to user input to NLU post-sequencer 665, then NLU post-sequencer 665 (or another component of system component 120, such as orchestrator component 430) can simply output content corresponding to result data 630 to the user. For example, NLU post-sequencer 665 can send result data 630 to orchestrator component 430. Orchestrator component 430 can send result data 630 to a device (110a / 110b) that can output audio and / or display text corresponding to result data 630. Depending on the circumstances, the arranger component 430 may send the resulting data 630 to the ASR component 450 to generate output text data and / or send the resulting data 630 to the TTS component 480 to generate output audio data.

[0181] Skill 490, associated with the highest NLU post-sequencer score, can provide NLU post-sequencer 665 with result data 630 indicating the need for more information, as well as instruction data. The instruction data can instruct skill 490 to suggest how the system can obtain the required information. For example, the instruction data can correspond to text data or audio data (i.e., computer-generated speech) that corresponds to "Please indicate ________________". The instruction data can be in a format that the device (110a / 110b) can output (e.g., text data or audio data). When this occurs, NLU post-sequencer 665 can simply have the received instruction data output by the device (110a / 110b). Alternatively, the instruction data can be in a format that the device (110a / 110b) cannot output. When this occurs, NLU post-sequencer 665 can, depending on the situation, have ASR component 450 or TTS component 480 process the instruction data to generate instruction data that can be output by the device (110a / 110b). Once the user provides the system with all the further information required by skill 490, skill 490 can provide the system with result data 630 indicating the response to the user input, which can be output by the system as detailed above.

[0182] The system may include an "informative" skill 490 that simply provides information to the system, which then outputs that information to the user. The system may also include a "transactional" skill 490 that requires system instructions to execute user input. Transactional skills 490 include carpooling skills, flight booking skills, etc. Transactional skill 490 may simply provide the NLU post-sequencer 665 with result data 630 instructing the transactional skill 490 to execute the user input. The NLU post-sequencer 665 then allows the system to solicit instructions from the user, allowing the system to have the transactional skill 490 execute the user input. The user-provided instructions may be audible or tactile (e.g., activation of a virtual button or text input via a virtual keyboard). In response to receiving the user-provided instructions, the system may provide transactional skill 490 with data corresponding to the instructions. In response, transactional skill 490 may execute commands (e.g., booking a flight, booking a train ticket, etc.). Therefore, although the system may not further enable the informational skill 490 after the informational skill 490 provides the result data 630 to the NLU post-sequencer 665, the system may further enable the transactional skill 490 after the transactional skill 490 provides the result data 630 indicating that the transactional skill 490 can execute user input to the NLU post-sequencer 665.

[0183] Figure 7 The operation of handling dialogue management according to the implementation plan of this disclosure is illustrated. Figure 7 One or more components shown may be part of dialogue manager 472. For example, dialogue manager 472 may include entity parser 770, focus data component 716, and action selector 718. Dialogue manager 472 may work in conjunction with other language processing components (e.g., NLU 460), or may operate in place of such components in some embodiments. In some embodiments and as described herein, dialogue manager 472 may include context determination component 155, prefix tree generation component 160, and generative language model component 170.

[0184] The system receives input text data 150, which may be received, for example, by a device (e.g., 413) or from another component of the system (e.g., as ASR output data 610). Input text data 150 may include text corresponding to user input and metadata indicating further information about the text (such as, for example, the user's identity determined by user identification component 495), and the user's emotional state. Input text data 150 may be text representing words, instructions, markup language, speech, or gestures, or may be vectors or other representations thereof. Input text data 150 may be generated by the user via a keyboard, touchscreen, microphone, camera, or other such input device. In other embodiments, input text data 150 is created using ASR as described above from audio data received from the user. The system may further receive additional input data 704, which may correspond to button presses, gestures, or other inputs, such as image data that can be interpreted by device 110, system component 120, or other components. As described in more detail below, using input text data 150 and / or other input data 704, the system can determine and output text data 815 and / or other output data 708. The system can alternatively or additionally perform actions based on input text data 150 and / or other input data 704, such as calling one or more APIs 710.

[0185] Entity parser 770 can be used to determine that input text data 150 includes representations of one or more entities. This can include named entity recognition (NER) – i.e., determining that input text data 150 includes such representations – and entity parsing (ER) – i.e., identifying the meaning or context of an entity, such as associating a nickname with a person's identity based on the identified entity. Entities can be people, places, things, ideas, and / or targets; example entities include proper nouns, nicknames, business names, place names, and / or application names. Entity parser 770 can operate similarly to the NER component 562 and / or entity parsing component 670 discussed herein with respect to NLU operations.

[0186] In some implementations, a single entity parser 770 is used for more than one domain (i.e., a "cross-domain" entity parser 770). Each domain may correspond to one or more dialogue models. One or more candidate domains corresponding to the input text data 770 may be determined; the input text data 770 may be processed using the entity parser 770 corresponding to the candidate domain. Dialogue focus data 716 may store output entities from each candidate domain and may remove unselected entities when selecting API 710 or determining the action to be performed.

[0187] Dialogue focus data 716 may (e.g., in dialogue storage 730) store state data corresponding to dialogue history data (including intents, entities, etc. corresponding to previous turns in the dialogue), action history data, and / or other data. In some embodiments, other components (e.g., action selector 718) do not store state data but instead query dialogue focus data 716 against state data. The system may send some or all of the dialogue focus data 716 to other systems and / or receive additional dialogue focus data from other systems. In some embodiments, other components (e.g., action selector 718) include a feature extractor component to extract features from the dialogue focus data 716.

[0188] Dialogue focus data 716 can be graph-based data comprising multiple graph nodes; each graph node may correspond to a piece of state data, such as entity type, entity value, previous API call, and / or user data. Other components (such as action selector 718) may access all graph nodes of dialogue focus data 716, or may access only a subset of the graph nodes of dialogue focus data 716. Dialogue focus data 716 can be any type of storage mechanism and can be used as long-term and / or short-term memory of the system, thereby enabling the tracking of entities, ASR outputs, TTS outputs, and other features at each turn of the dialogue. In some embodiments, dialogue focus data 716 is updated with updated dialogue focus data after each turn of the dialogue; in other embodiments, dialogue focus data 716 is updated after the end of the dialogue is determined.

[0189] Entity parser 770 can utilize place name dictionary information stored in the entity library store. The place name dictionary information can be used to match text data (representing a portion of user input) with text data representing known entities (such as song titles, contact names, etc.). The place name dictionary can be linked to a user (e.g., a specific place name dictionary can be associated with a specific user's music collection), linked to certain skills 490 (e.g., shopping skills, music skills, video skills, etc.), or can be organized in various other ways.

[0190] Entity parser 770 may include a sorter component. The sorter component may assign a specific confidence score to each hypothesis input therein. The confidence score of a hypothesis may represent the confidence of the system in the processing performed relative to that hypothesis. The confidence score of a particular hypothesis may be affected by whether the hypothesis has unfilled slots. For example, if a hypothesis associated with the first skill component includes all filled / resolved slots, that hypothesis may be assigned a higher confidence score than another NLU hypothesis that includes at least some unfilled / unresolved slots of entity parser 770.

[0191] Focus data 716 can store dialogue-related data. In various embodiments, dialogue focus data 716 stores input text data 150, other input data 704, entity data from entity parser 770, and / or action data and dialogue data from action selector 718. Dialogue focus data 716 may also include additional information such as location data, user preference data, and environmental data. In various embodiments, dialogue focus data 716 uses an encoder to encode some or all of the received data into one or more feature vectors, and uses a decoder to determine intent data corresponding to the user's intent based on the feature vectors. Dialogue focus data 716 may also include state data representing earlier dialogue, actions, or other earlier user information or data.

[0192] Action selector 718 determines the action to be performed in response to a user request, where the action may include calling an API to perform an action and / or presenting output to the user. Action selector 718 may include a trained model and may process dialogue focus data 716. If action selector 718 determines to call an API, one or more APIs 710 may be activated and the corresponding action performed. If action selector 718 determines to present a prompt or other output data to the user, NLG component 479 may be used to generate output text data 815 and / or other output data 708. In either case, action selector 718 may update dialogue focus data 716 based on the API call or the output presented to the user.

[0193] In some implementations, the action selector 718 may process data from the dialogue store 730 to select one or more skills 490 / skill systems 425 that can respond to a user request, and present the selected skills to the user using output text data 815.

[0194] In some implementations, system component 120 may determine past interaction data (such as satisfaction ratings corresponding to skill 490) and store it in storage 730. Satisfaction ratings may be based on past interactions between the user and the skill of system component 120. In some implementations, system component 120 may determine a skill-specific satisfaction rating based on user-specific data, interaction-specific data, and / or focus data 716 (such as user profile data 470 associated with a specific user, location data, past user interactions with system component 120, past user interactions with skill 490, user preferences, device type of device 110 receiving user requests, device type of device 110 capable of outputting / responding to user requests, device capabilities, the state of the first few rounds of dialogue, and other data).

[0195] In some implementations, a user may provide feedback to system component 120 or skill system 425, indicating the user's satisfaction with the service skills performed in response to a user request / action performed in response to a user request. This feedback may be solicited by system component 120 or skill system 425. In some cases, the feedback may be explicit and may be received as voice input, text input, gesture, or other types of input. In other cases, the feedback may be implicit and may be determined by processing data related to the user interaction, such as image data representing the user's facial expressions, audio data indicating the user's emotions / feelings (representing speech from the user), text data indicating the user's emotions (representing user input), other sounds / voices indicating the user's emotions / feelings / feelings (e.g., "ugh" indicating frustration), and other types of data.

[0196] The action selector 718 can select skills based on a variety of other factors, including but not limited to the user's age, user group (user preferences or other social organizations), location information (e.g., the location where the user initiates input, the place the user intends to go, etc.), skill location data, intent, intent type, time of day when the user input is received, time of year when the user input is received, etc.

[0197] Action selector 718 can select skills based on skill availability; for example, restaurant reservations can indicate availability or unavailability for a specific day. System component 120 / action selector 718 can determine the number of diners at the restaurant or the number of takeout orders the restaurant is processing based on the number of user requests routed to a skill by system component 120. In another example, system component 120 / action selector 718 can determine that multiple user requests are routed to a first skill for a first location / geographic region (e.g., a city), and can determine to route subsequent / future user requests to another skill for that geographic region.

[0198] For restaurant reservations, the action selector 718 can select a skill based on the number of people in the user's group, as some restaurants may not accept reservations for small groups of fewer than a certain number. The action selector 718 can also select a skill based on people other than the user who may be involved in the user's request. For example, if the user says, "Alexa, what should we do tomorrow?", system component 120 can determine information relevant to other people in the user's household to select a skill.

[0199] Action selector 718 can select skills based on dialogue state, where dialogue state indicates multiple rounds / attempts of suggested skills / service providers, and action selector 718 can modify search criteria (e.g., expand postal code, expand time range, etc.) to select skills.

[0200] Figure 8 The diagram shows components of a system that can be used for execution unit selection, parametric TTS processing, and / or model-based audio synthesis. Figure 8 This is a conceptual diagram illustrating operation for generating synthesized speech using a TTS component 480 according to an embodiment of the present disclosure. The TTS component 480 may receive text data 815 and process it using one or more TTS models 860 to generate synthesized speech in the form of spectrogram data 845. A vocoder 890 may convert the spectrogram data 845 into output speech audio data 895, which may represent a time-domain waveform suitable for amplification and output as audio (e.g., from a speaker).

[0201] The TTS component 480 may additionally receive other input data 825. The other input data 825 may include, for example, identifiers and / or tags corresponding to the expected speaker identity, voice characteristics, emotion, and voice style of the synthesized speech. In some implementations, the other input data 825 may include text tags or text metadata that may indicate, for example, how a particular word should be pronounced, by indicating the desired output speech quality in tags formatted according to Speech Synthesis Markup Language (SSML) or some other form. For example, the first text tag may include a marker indicating when the text should be whispered at the beginning (e.g.,<begin whisper> The text, and the second marker may be included at the point where the text should be whispered (e.g.,<end whisper> The text may include tags in the text data 815 and / or other input data 825, such as metadata accompanying the TTS request and indicating what text should be whispered (or audio characteristics with some other indication).

[0202] TTS component 480 may include a preprocessing component 820 that can convert text data 815 and / or other input data 825 into a form suitable for processing by TTS model 860. Text data 815 may come from, for example, an application, a skill component (described further below), an NLG component, another device or source, or may be input by a user. The text data 815 received by TTS component 480 is not necessarily text, but may include other data (such as symbols, codes, other data, etc.) that can reference the text to be synthesized (such as indicators of words and / or phonemes). Preprocessing component 820 may transform text data 815 into, for example, a symbolic language representation for processing by TTS component 480. This symbolic language representation may include language contextual features such as phoneme data, punctuation data, syllable-level features, word-level features, and / or emotion, speaker, accent, or other features. Syllable-level features may include syllable emphasis, syllable rate, syllable intonation, or other such syllable-level features; word-level features may include word emphasis, word rate, word intonation, or other such word-level features. Emotional characteristics may include data corresponding to emotions associated with the text data 815, such as surprise, anger, or fear. Speaker characteristics may include data corresponding to speaker type, such as gender, age, or occupation. Accent characteristics may include data corresponding to accents associated with the speaker, such as Southern accent, Boston accent, British accent, French accent, or other such accents. Style characteristics may include reading style, poetry recitation style, news anchor style, sports commentator style, various singing styles, etc.

[0203] The preprocessing component 820 may include functionalities and / or components for performing text normalization, language analysis, prosodic generation, or other such operations. During text normalization, the preprocessing component 820 may first process the text data 815 and generate standard text, converting content such as numbers, abbreviations (such as Apt., St., etc.), symbols ($, %), etc., into equivalents of the written words.

[0204] During language analysis, preprocessing component 820 analyzes the language in normalized text to generate a sequence of speech units corresponding to the input text. This process can be called word-to-phoneme conversion. A speech unit consists of a symbolic representation of a sound unit that will ultimately be combined by the system and output as speech. Various speech units can be used to segment the text for speech synthesis. In some implementations, TTS model 860 can process speech based on phonemes (individual sounds), semiphones, diphones (the latter half of a phoneme combined with the first half of an adjacent phoneme), disyllabics (two consecutive phonemes), syllables, words, phrases, sentences, or other units. Each word can be mapped to one or more speech units. This mapping can be performed using a language dictionary stored by the system (e.g., stored in a storage component). The language analysis performed by preprocessing component 820 can also identify different grammatical components, such as prefixes, suffixes, phrases, punctuation, syntactic boundaries, etc. These grammatical components can be used by TTS component 480 to produce a natural-sounding audio waveform output. The language dictionary may also include letter-to-sound rules and other tools that can be used to pronounce previously unrecognized words or letter combinations that the TTS component 480 may encounter. Generally, the more information included in the language dictionary, the higher the quality of the speech output.

[0205] The output of the preprocessing component 820 may be a symbolic language representation that may include a sequence of speech units. In some implementations, the sequence of speech units may be annotated with prosodic features. In some implementations, prosody may be applied partially or entirely by the TTS model 860. This symbolic language representation may be sent to the TTS model 860 to be converted into audio data (e.g., in the form of a Mel spectrogram or other frequency content data format).

[0206] TTS component 480 can retrieve one or more previously trained and / or configured TTS models 860 from voice profile storage 885. TTS model 860 can be, for example, a neural network architecture, which can be described as interconnected artificial neurons or "units" linked by layers and / or blocks. Generally, neural network model architectures can be broadly described using hyperparameters that describe the number of layers and / or blocks, how many units are contained in each layer and / or block, what activation functions they implement, how they are interconnected, etc. Neural network models include trainable parameters (e.g., "weights") that indicate how much weight a unit should give to a particular input when generating output (e.g., in the form of an arithmetic multiplier). In some implementations, neural network models may include other features such as self-attention mechanisms, which can determine certain parameters at runtime based on the input rather than, for example, based on loss calculations during training. Various data describing a particular TTS model 860 can be stored in voice profile storage 885. TTS model 860 can represent a specific speaker identity and can be modulated based on speaking style, emotion, etc. In some implementations, a specific speaker identity may be associated with more than one TTS model 860; for example, with different models representing different speaking styles, languages, emotions, etc. In some implementations, a specific TTS model 860 may be associated with more than one speaker identity; that is, it can generate synthesized speech that reproduces the speech characteristics of more than one role. Therefore, a first TTS model 860a can be used to create synthesized speech for a first speech processing system component 120a, while different second TTS models 860b can be used to create synthesized speech for a second speech processing system component 120b. In some cases, the TTS model 860 may generate desired speech characteristics based on conditional data received or determined from text data 815 and / or other input data 825. For example, the synthesized speech of the first speech processing system component 120a may differ from the synthesized speech of the second speech processing system component 120b.

[0207] The TTS component 480 may retrieve the TTS model 860 from the voice profile storage 885 based on instructions received along with text data 815 and / or other input data 825, and use it to process the input to generate synthesized speech. The TTS component 480 may provide any relevant conditional labels to the TTS model 860 to generate synthesized speech with the desired speech characteristics. The TTS model 860 may generate spectrogram data 845 representing the synthesized speech (e.g., frequency content data), and send the spectrogram data to the vocoder 890 to convert it into an audio signal.

[0208] The TTS component 480 may generate additional output data 855. This additional output data 855 may include, for example, indications or instructions for processing and / or outputting synthesized speech. For example, text data 815 and / or other input data 825 may be received along with metadata (such as SSML markers) indicating that selected portions of the text data 815 should be louder or quieter. Therefore, the additional output data 855 may include volume markers instructing the vocoder 890 to increase or decrease the amplitude of the output speech audio data 895 at times corresponding to the selected portions of the text data 815. Additionally or alternatively, the volume markers may instruct the playback device to increase or decrease the volume of the synthesized speech from the device's current volume level, or to decrease the volume of other media output by the device (e.g., to deliver an emergency message).

[0209] The vocoder 890 can convert the spectrogram data 845 generated by the TTS model 860 into an audio signal (e.g., an analog or digital time-domain waveform) suitable for amplification and as an audio output. The vocoder 890 can be, for example, a general-purpose neural vocoder based on Parallel WaveNet or a related model. The vocoder 890 can take audio data in Mel spectrogram form, for example, with 80 coefficients and a frequency range of 50 Hz to 12 kHz, as input. The synthesized speech audio data 895 can be a time-domain audio format (e.g., Pulse Code Modulation (PCM), Waveform Audio Format (WAV), µ-law, etc.), which can be easily converted into an analog signal for amplification and output by a speaker. The synthesized speech audio data 895 can consist of 8-bit, 16-bit, or 24-bit audio with sampling rates such as 16 kHz, 24 kHz, and 44.1 kHz. In some implementations, other bit and / or sampling rates may be used.

[0210] Various machine learning techniques can be used to train and operate models to perform the various steps described in this paper, such as user identification, sentiment detection, image processing, dialogue management, etc. Models can be trained and operated using a variety of machine learning techniques. Such techniques can include, for example, neural networks (such as deep neural networks and / or recurrent neural networks), inference engines, trained classifiers, etc. Examples of trained classifiers include Support Vector Machines (SVMs), neural networks, decision trees, AdaBoost (short for "adaptive boosting") combined with decision trees, and random forests. Taking SVMs as an example, an SVM is a supervised learning model with associated learning algorithms that analyze data and identify patterns in the data, and is often used for classification and regression analysis. Considering a set of training examples, each labeled as belonging to one of two categories, the SVM training algorithm builds a model that assigns new examples to one category or the other, thus making it a non-probabilistic binary linear classifier. More complex SVM models can be built using training sets that identify more than two categories, where the SVM determines which category is most similar to the input data. Mappable SVM models separate examples of individual classes by clear margins. New examples are then mapped into the same space, and their class is predicted based on which side of the margin they fall on. The classifier provides a "score" indicating which class the data best matches. The score provides an indication of how well the data matches the class.

[0211] In order to apply machine learning techniques, the machine learning process itself needs to be trained. Training machine learning components (such as one of the first or second models in this case) requires establishing "ground truth" for the training examples. In machine learning, the term "ground truth" refers to the accuracy of classification on the training set using supervised learning techniques. Various techniques can be used to train models, including backpropagation, statistical learning, supervised learning, semi-supervised learning, randomized learning, or other known techniques.

[0212] Figure 9 This is a block diagram conceptually illustrating a device 110 that can be used with the system. Figure 10 This is a block diagram conceptually illustrating example components of a remote device, such as a natural language command processing system component 120 that can assist with ASR processing, NLU processing, etc., and a skill system 425. The system (120 / 425) may include one or more servers. As used herein, "server" may refer to a conventional server as understood in a server / client computing architecture, but may also refer to a variety of different computing components that can assist the operations discussed herein. For example, a server may include one or more physical computing components (such as rack servers) that are physically and / or connected to other devices / components via a network and are capable of performing computing operations. A server may also include one or more virtual machines that simulate a computer system and run on one device or across multiple devices. A server may also include other combinations of hardware, software, firmware, etc., to perform the operations discussed herein. A server may be configured to operate using one or more of the following computing technologies: client-server model, computer office model, grid computing, fog computing, mainframe technology, utility computing, peer-to-peer model, sandboxing, or others.

[0213] While device 110 can operate locally to the user (e.g., within the same environment, so the device can receive input and play output for the user), server / system component 120 can be located remotely from device 110, as its operation may not require proximity to the user. Server / system component 120 can be located in a completely different location from device 110 (e.g., as part of a cloud computing system, etc.), or it can be located in the same environment as device 110 but physically separate from it (e.g., a home server or similar device residing in the user's home or office but possibly in a closet, basement, attic, etc.). Support device 120 can also be a version of user device 110 that includes processing power different (e.g., larger) than other user devices 110 in the home / office. One benefit of having server / system component 120 located in the user's home / office is that data used to process commands / return responses can be stored at the user's home, thus reducing potential privacy concerns.

[0214] The overall system 100 disclosed herein may include multiple systems (120 / 425), such as one or more natural language processing system components 120 for performing ASR processing, one or more natural language processing system components 120 for performing NLU processing, one or more skill systems 425, etc. In operation, each of these systems may include computer-readable and computer-executable instructions residing on the corresponding device (120 / 425), as will be further discussed below.

[0215] Each of these devices (110 / 120 / 425) may include one or more controllers / processors (904 / 1004), each of which may include a central processing unit (CPU) for processing data and computer-readable instructions, and a memory (906 / 1006) for storing the data and instructions of the respective device. The memory (906 / 1006) may individually include volatile random access memory (RAM), non-volatile read-only memory (ROM), non-volatile magnetoresistive memory (MRAM), and / or other types of memory. Each device (110 / 120 / 425) may also include a data storage component (908 / 1008) for storing data and controller / processor executable instructions. Each data storage component (908 / 1008) may individually include one or more non-volatile memory types, such as magnetic storage, optical storage, solid-state storage, etc. Each device (110 / 120 / 425) can also be connected to removable or external non-volatile memory and / or storage (such as removable memory cards, memory key drives, network storage, etc.) via the corresponding input / output device interface (902 / 1002).

[0216] The computer instructions for operating each device (110 / 120 / 425) and its various components can be executed by the controller / processor (904 / 1004) of the corresponding device during operation, using memory (906 / 1006) as temporary "working" storage. The computer instructions for the device can be stored in a non-transitory manner in non-volatile memory (906 / 1006), storage (908 / 1008), or an external device. Alternatively, as a supplement or replacement to software, some or all of the executable instructions can be embedded in the hardware or firmware of the corresponding device.

[0217] Each device (110 / 120 / 425) includes an input / output device interface (902 / 1002). Various components can be connected via the input / output device interface (902 / 1002), as will be discussed further below. Additionally, each device (110 / 120 / 425) may include an address / data bus (924 / 1024) for transferring data between components of the respective device. As a supplement (or alternative) to connecting to other components across the bus (924 / 1024), each component within a device (110 / 120 / 425) may also be directly connected to other components.

[0218] refer to Figure 9 The device 110 may include an input / output device interface 902 that connects to various components, such as audio output components, such as speaker 912, wired or wireless headphones (not illustrated), or other components capable of outputting audio. The device 110 may also include an audio capture component. The audio capture component may be, for example, a microphone 920 or a microphone array, wired or wireless headphones (not illustrated), etc. If a microphone array is included, the approximate distance to the origin of the sound can be determined by acoustic localization based on the time and amplitude differences between the sounds captured by the different microphones in the array. The device 110 may additionally include a display 916 for displaying content. The device 110 may also include a camera 918.

[0219] Via antenna 922, input / output device interface 902 can connect to one or more networks 199 via wireless local area network (WLAN) radio (such as Wi-Fi), Bluetooth, and / or wireless network radio, such as radios capable of communicating with wireless communication networks such as LTE networks, WiMAX networks, 3G networks, 4G networks, and 5G networks. Wired connections, such as Ethernet, are also supported. Through network 199, the system can be distributed in a networked environment. The I / O device interfaces (902 / 1002) may also include communication components that allow data exchange between devices (such as different physical servers or other components in a server set).

[0220] Components of device 110, natural language command processing system component 120, or skill system 425 may include their own dedicated processors, memory, and / or storage. Alternatively, one or more components of device 110, natural language command processing system component 120, or skill system 425 may utilize the I / O interfaces (902 / 1002), processors (904 / 1004), memory (906 / 1006), and / or storage (908 / 1008) of device 110, natural language command processing system component 120, or skill system 425, respectively. Thus, ASR component 450 may have its own I / O interface, processor, memory, and / or storage; NLU component 460 may have its own I / O interface, processor, memory, and / or storage; and so on for the various components discussed herein.

[0221] As described above, multiple devices can be employed in a single system. In such a multi-device system, each of the devices may include different components for performing different aspects of system processing. Multiple devices may include overlapping components. As described herein, the components of device 110, natural language command processing system component 120, and skill system 425 are exemplary and may be positioned as independent devices or may be included wholly or partially as components of a larger device or system. As will be understood, numerous components may be present on system component 120 and / or device 110. For example, language processing 492 (which may include ASR 450), language output 493 (which may include NLG 479 and TTS 480), etc., such as Figure 4 As illustrated. Unless otherwise expressly stated, the system version of such a component may operate similarly to the device version of such a component, and therefore the description of one version (e.g., the system version or the local version) applies to the description of another version (e.g., the local version or the system version), and vice versa.

[0222] like Figure 11 As illustrated, multiple devices (110a-110n, 120, 425) may contain components of the system, and the devices may be connected to network 199. Network 199 may include a local or private network, or may include a wide area network (such as the Internet). Devices may connect to network 199 via wired or wireless connections. For example, voice detection device 110a, smartphone 110b, smartwatch 110c, tablet computer 110d, vehicle 110e, voice detection device with display 110f, monitor / smart TV 110g, washer / dryer 110h, refrigerator 110i, microwave oven 110j, autonomous mobile device 110k (e.g., robot), etc., may connect to network 199 via wireless service providers, via Wi-Fi or cellular network connections, etc. Other devices are included as supporting devices for network connectivity, such as natural language command processing system component 120, skill system 425, and / or other devices. Supporting devices may connect to network 199 via wired or wireless connections. The networked device may use one or more built-in or connected microphones or other audio capture devices to capture audio, wherein processing is performed by the ASR component, NLU component or other components (such as the ASR component 450, NLU component 460, etc. of the Natural Language Command Processing System Component 120) of the same device or another device connected via network 199.

[0223] The concepts disclosed herein can be applied to many different devices and computer systems, including, for example, general-purpose computing systems, speech processing systems, and distributed computing environments.

[0224] The following terms may also be used to understand this material.

[0225] 1. A computer-implemented method, the method comprising: receiving first input audio data representing a first spoken natural language question by a speech processing system; determining first ASR output data corresponding to a first text transcription of the first input audio data using automatic speech recognition (ASR) processing; determining first data representing a first context of the first spoken natural language question using the first ASR output data; determining at least one information record of a data source based on the first data; identifying a first knowledge graph representing the data source; generating a first subgraph representing the at least one information record based on the first knowledge graph; generating a first data structure representing at least one relation of a first entity and a second entity using the at least one information record, wherein the at least one relation is described in the at least one information record; traversing the first subgraph using a language generation model to determine second data representing a first tuple in response to the first spoken natural language question; determining that the second data corresponds to at least one relation represented by the first data structure; determining a first sentence from the at least one information record based on the second data in response to determining that the second data corresponds to at least one relation; determining output data representing a response to the first spoken natural language question using the first sentence; and causing a device to present the output data.

[0226] 2. The computer-implemented method as described in Clause 1, the method further comprising: receiving second input audio data representing earlier natural language user input before receiving the first input audio data; and using ASR processing to determine second ASR output data corresponding to a second text transcription of the second input audio data, wherein determining the first data representing the first context further comprises: determining second data representing a second context based on both the first ASR output data and the second ASR output data; and using the second data representing the second context as the first data.

[0227] 3. A computer-implemented method as described in Clause 1 or 2, the method further comprising: determining a second sentence from the at least one information record using the first data structure, wherein the second sentence is determined by traversing a path derived from the first data structure of the at least one relation; generating a user prompt using the second sentence; and causing the device to display the user prompt.

[0228] 4. A computer-implemented method as described in clauses 1, 2, or 3, the method further comprising: generating a second tuple in response to the first data, using the first subgraph and the first data as input to the language generation model before generating the second data, wherein the second tuple represents a second relation between the first entity and the second entity; determining that the second relation is not reflected in the first data structure; and determining that the second tuple is invalid in response to the second relation not being reflected in the first data structure.

[0229] 5. A computer-implemented method, the method comprising: receiving input data representing user input; determining first context data of the input data; using the first context data to determine first data representing a graph of a first information record; using a language model and the first data to determine second data corresponding to a first response to the user input, wherein the second data represents an information item; determining that the information item is embodied in the first information record; and in response to determining that the information item is embodied in the first information record, generating output data at least in part based on the second data.

[0230] 6. The computer-implemented method as described in Clause 5, the method further comprising: generating a first data structure representing at least one entity relationship embodied in the first information record, wherein determining that the information item is embodied in the first information record further comprises determining that the second data corresponds to the at least one entity relationship.

[0231] 7. The computer-implemented method as described in Clause 6, the method further comprising: using the first data structure to determine third data based on the information item and corresponding to at least a second entity relationship with the first data structure; and using the third data to generate fourth data representing a user prompt, wherein the generation of the output data is based at least in part on the second data and the fourth data.

[0232] 8. A computer-implemented method as described in clauses 5, 6, or 7, the method further comprising: before generating the second data, using the language model to traverse the graph of the first data to determine third data corresponding to a second response to the user input, wherein the third data represents a second information item; and determining that the second information item is not reflected in the first information record.

[0233] 9. A computer-implemented method as described in clauses 5, 6, 7, or 8, the method further comprising: receiving third data representing at least one prior user input, wherein determining the first context further comprises using the input data and the third data.

[0234] 10. A computer-implemented method as described in clauses 5, 6, 7, 8, or 9, the method further comprising: receiving second input data; determining second context data of the second input data; determining that the second context data corresponds to first context data; traversing the graph using the language model to determine third data corresponding to a second response to the second user input, wherein the third data represents a second information item; determining that the second information item is embodied in the first information record; and generating second output data at least in part based on the third data in response to determining that the second information item is embodied in the first information record.

[0235] 11. A computer-implemented method as described in clauses 5, 6, 7, 8, 9, or 10, the method further comprising: determining a second information record corresponding to the first context data, wherein a first information item of the first information record corresponds to a second information item of the second information record, wherein the graph includes a representation of the first information record and the second information record.

[0236] 12. A computer-implemented method as described in clauses 5, 6, 7, 8, 9, 10, or 11, the method further comprising: receiving, before receiving the input data, audio data representing spoken natural language user input from a device; determining the input data from the audio data using automatic speech recognition (ASR) processing; after determining that the information item is embodied in the first information record, performing natural language generation processing using the second data to determine the output data; and transmitting the output data to the device.

[0237] 13. A system comprising: at least one processor; and at least one memory including instructions that, when executed by the at least one processor, cause the system to: receive input data representing user input; determine first context data of the input data; use the first context data to determine first data representing a graph of a first information record; use a language model and the first data to determine second data corresponding to a first response to the user input, wherein the second data represents an information item; determine that the information item is embodied in the first information record; and in response to determining that the information item is embodied in the first information record, generate output data at least in part based on the second data.

[0238] 14. The system as described in Clause 13, wherein the at least one memory further includes instructions that, when executed by the at least one processor, further cause the system to: generate a first data structure representing at least one entity relationship embodied in the first information record, wherein determining that the information item is embodied in the first information record further includes determining that the second data corresponds to the at least one entity relationship.

[0239] 15. The system as described in Clause 14, wherein the at least one memory further includes instructions that, when executed by the at least one processor, further cause the system to: determine, using the first data structure, third data based on the information item and corresponding to at least a second entity relationship with the first data structure; and generate, using the third data, fourth data representing a user prompt, wherein the generation of the output data is based at least in part on the second data and the fourth data.

[0240] 16. The system as described in Clauses 13, 14, or 15, wherein the at least one memory further includes instructions that, when executed by the at least one processor, further cause the system to: traverse the graph of the first data using the language model before generating the second data, determine third data corresponding to a second response to the user input, wherein the third data represents a second information item; and determine that the second information item is not reflected in the first information record.

[0241] 17. The system as described in Clauses 13, 14, 15, or 16, wherein the at least one memory further includes instructions that, when executed by the at least one processor, further cause the system to: receive third data representing at least one prior user input, wherein determining the first context further includes using the input data and the third data.

[0242] 18. A system as described in Clauses 13, 14, 15, 16, or 17, wherein the at least one memory further comprises instructions that, when executed by the at least one processor, further cause the system to: receive second input data; determine second context data of the second input data; determine that the second context data corresponds to the first context data; traverse the graph using the language model to determine third data corresponding to a second response to the second user input, wherein the third data represents a second information item; determine that the second information item is embodied in the first information record; and in response to determining that the second information item is embodied in the first information record, generate second output data at least in part based on the third data.

[0243] 19. The system as described in Clauses 13, 14, 15, 16, 17 or 18, wherein the at least one memory further includes instructions that, when executed by the at least one processor, further cause the system to: determine a second information record corresponding to the first context data, wherein a first information item of the first information record corresponds to a second information item of the second information record, wherein the graph includes a representation of the first information record and the second information record.

[0244] 20. A system as described in clauses 13, 14, 15, 16, 17, 18, or 19, wherein the at least one memory further includes instructions that, when executed by the at least one processor, further cause the system to: receive audio data representing spoken natural language user input from the device before receiving the input data; determine the input data from the audio data using automatic speech recognition (ASR) processing; perform natural language generation processing using the second data to determine the output data after determining that the information item is embodied in the first information record; and transmit the output data to the device.

[0245] The foregoing aspects of this disclosure are intended to be illustrative. They were chosen to explain the principles and applications of this disclosure and are not intended to be exhaustive or limiting. Many modifications and variations of the disclosed aspects will be apparent to those skilled in the art. Those skilled in the art of computer and speech processing will recognize that the components and process steps described herein are interchangeable with other components or steps, or combinations of components or steps, and still achieve the benefits and advantages of this disclosure. However, it will be apparent to those skilled in the art that this disclosure can be practiced without some or all of the specific details and steps disclosed herein. Furthermore, unless expressly stated to the contrary, features / operations / components, etc., from one embodiment discussed herein may be combined with features / operations / components, etc., from another embodiment discussed herein.

[0246] Aspects of the disclosed system may be implemented as computer methods or articles of art, such as memory devices or non-transitory computer-readable storage media. The computer-readable storage media may be readable by a computer and may include instructions for causing a computer or other device to perform the processes described in this disclosure. The computer-readable storage media may be implemented as volatile computer memory, non-volatile computer memory, hard disk drive, solid-state memory, flash memory drive, removable disk drive, and / or other media. Furthermore, components of the system may be implemented as firmware or hardware.

[0247] Unless otherwise specifically indicated, or otherwise understood as used in the context herein, the conditional language used herein (such as, among other things, "can," "may," "may," "can," "for example," etc.) is generally intended to convey that certain embodiments include certain features, elements, and / or steps that are not included in other embodiments. Therefore, such conditional language is not generally intended to imply that one or more embodiments require features, elements, and / or steps in any way, or that one or more embodiments necessarily include logic for determining, with or without further input or prompting, whether such features, elements, and / or steps are included or to be performed in any particular embodiment. The terms "comprise," "include," "have," etc., are synonymous and used inclusively in an open manner without excluding additional elements, features, actions, operations, etc. Additionally, the term "or" is used in its inclusive sense (but not its exclusive sense) such that when used, for example, to connect elements in a list, the term "or" means one, some, or all of the elements in the list.

[0248] Unless otherwise specifically stated, disjunctive languages ​​such as the phrase "at least one of X, Y, or Z" are understood in context to generally represent that items, terms, etc., can be X, Y, or Z or any combination thereof (e.g., X, Y, and / or Z). Therefore, such disjunctive languages ​​are generally not intended and should not imply that certain embodiments require the presence of at least one of X, at least one of Y, or at least one of Z.

[0249] As used in this disclosure, unless otherwise specifically indicated, the term "a" or "an" may include one or more items. Furthermore, unless otherwise explicitly indicated, the phrase "based on" is intended to mean "at least in part based on".< / dishtype> < / help> < / dishtype> < / order> < / help> < / dishtype> < / order> < / help> < / playmusic> < / playmusic> < / playmusic> < / playmusic> < / playvideo> < / playmusic> < / playmusic> < / playmusic> < / playmusic> < / playmusic> < / playmusic> < / mute> < / outputweather> < / et> < / obj> < / rel>

Claims

1. A computer-implemented method, the method comprising: receiving input data representing a user input; determining first context data for the input data; determining, using the first context data, first data representing a graph of a first information record; determining, using a language model and the first data, second data corresponding to a first response to the user input, wherein the second data represents an information item; determining that the information item is embodied in the first information record; and in response to determining that the information item is embodied in the first information record, generating output data based at least in part on the second data.

2. The computer-implemented method of claim 1, the method further comprising: generating a first data structure representing at least one entity relationship embodied in the first information record, wherein determining that the information item is embodied in the first information record further comprises determining that the second data corresponds to the at least one entity relationship.

3. The computer-implemented method of claim 2, the method further comprising: determining, using the first data structure, third data based on the information item and corresponding to at least a second entity relationship of the first data structure; and generating, using the third data, fourth data representing a user prompt, wherein generating the output data is based at least in part on the second data and the fourth data.

4. The computer-implemented method of claim 1, 2, or 3, the method further comprising: prior to generating the second data, traversing the graph of the first data using the language model to determine third data corresponding to a second response to the user input, wherein the third data represents a second information item; and determining that the second information item is not embodied in the first information record.

5. The computer-implemented method of claim 1, 2, 3, or 4, the method further comprising: receiving third data representing at least one prior user input, wherein determining first context further comprises using the input data and the third data.

6. The computer-implemented method of claim 1, 2, 3, 4, or 5, the method further comprising: receiving second input data; determining second context data for the second input data; determining that the second context data corresponds to the first context data; traversing the graph using the language model to determine third data corresponding to a second response to the second user input, wherein the third data represents a second information item; determining that the second information item is embodied in the first information record; and in response to determining that the second information item is embodied in the first information record, generating second output data based at least in part on the third data.

7. The computer-implemented method of claim 1, 2, 3, 4, 5, or 6, the method further comprising: determining a second information record corresponding to the first context data, wherein a first information item of the first information record corresponds to a second information item of the second information record, wherein the graph comprises representations of the first information record and the second information record. ​ ​ ​ 8. The computer-implemented method of claim 1, 2, 3, 4, 5, 6, or 7, further comprising: prior to receiving the input data, receiving, from a device, audio data representing spoken natural language user input; determining the input data from the audio data using automatic speech recognition (ASR) processing; after determining that the information item is embodied in the first information record, performing natural language generation processing using the second data to determine the output data; and sending the output data to the device.

9. A system comprising: at least one processor; and at least one memory including instructions that, when executed by the at least one processor, cause the system to: receive input data representing user input; determine first context data for the input data; determine first data representing a graph of a first information record using the first context data; determine second data corresponding to a first response to the user input using a language model and the first data, wherein the second data represents an information item; determine that the information item is embodied in the first information record; and in response to determining that the information item is embodied in the first information record, generate output data based at least in part on the second data.

10. The system of claim 9, wherein the at least one memory further includes instructions that, when executed by the at least one processor, further cause the system to: generate a first data structure representing at least one entity relationship embodied in the first information record, wherein determining that the information item is embodied in the first information record further comprises determining that the second data corresponds to the at least one entity relationship.

11. The system of claim 10, wherein the at least one memory further includes instructions that, when executed by the at least one processor, further cause the system to: determine third data based on the information item and corresponding to at least a second entity relationship of the first data structure using the first data structure; and generate fourth data representing a user prompt using the third data, wherein generating the output data is based at least in part on the second data and the fourth data.

12. The system of claim 9, 10, or 11, wherein the at least one memory further includes instructions that, when executed by the at least one processor, further cause the system to: prior to generating the second data, traverse the graph of the first data using the language model to determine third data corresponding to a second response to the user input, wherein the third data represents a second information item; and determine that the second information item is not embodied in the first information record.

13. The system of claim 9, 10, 11, or 12, wherein the at least one memory further includes instructions that, when executed by the at least one processor, further cause the system to: receive third data representing at least one earlier user input, ​ ​ wherein determining the first context further comprises using the input data and the third data.

14. The system of claim 9, 10, 11, 12, or 13, wherein the at least one memory further includes instructions that, when executed by the at least one processor, further cause the system to: receive second input data; determine second context data for the second input data; determine that the second context data corresponds to the first context data; use the language model to traverse the graph to determine third data corresponding to a second response to the second user input, wherein the third data represents a second information item; determine that the second information item is embodied in the first information record; and in response to determining that the second information item is embodied in the first information record, generate second output data based at least in part on the third data.

15. The system of claim 9, 10, 11, 12, 13, or 14, wherein the at least one memory further includes instructions that, when executed by the at least one processor, further cause the system to: determine a second information record corresponding to the first context data, wherein a first information item of the first information record corresponds to a second information item of the second information record, wherein the graph includes representations of the first information record and the second information record.

Images