Detecting irrelevant utterances in chatbot systems

Abstract

Techniques are described for determining whether an input utterance is irrelevant to a set of skillbots associated with a masterbot. In some embodiments, the systems described herein include a training system and a masterbot. The training system trains a classifier for the masterbot. The training includes accessing training utterances associated with the skillbot and generating a training feature vector from the training utterance. The training further includes generating a plurality of set representations of the training feature vector, each set representation corresponding to a subset of the training feature vector, and the training further includes configuring a classifier with the plurality of set representations. The masterbot accesses the input utterance and generates an input feature vector. The masterbot uses the classifier to compare the input feature vector with the plurality of set representations to determine whether the input features are out of range and therefore cannot be addressed by the skillbot.

Description

[Background technology]

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS This application claims the benefit of and priority under 35 U.S.C. §119(e) to U.S. Provisional Application No. 62 / 899,700, filed September 12, 2019, and entitled "Detecting Unrelated Utterances in a Chatbot System," the contents of which are incorporated herein by reference for all purposes.

[0002] background A chatbot is an artificial intelligence-based software application or device that provides an interface for conversation with a human user. Chatbots can be programmed to perform various tasks in response to user input provided during a conversation. User input can be supplied in a variety of forms, including, for example, voice input and text input. Thus, natural language understanding, speech-to-text, and other language processing techniques can be employed as part of the processing performed by the chatbot. In some computing environments, multiple chatbots are available to interact with a user, with each chatbot addressing a different set of tasks. Summary of the Invention

[0003] overview The techniques described herein are for determining that an input utterance from a user is not relevant to any skillbot, also referred to as a chatbot, in a set of one or more skillbots available to the masterbot. In some embodiments, the masterbot may evaluate the input utterance and may determine that the input utterance is not relevant to a skillbot or may route the input utterance to an appropriate skillbot. [Means for solving the problem]

[0004] In some embodiments, the system described herein includes a training system and a masterbot. The training system is configured to train a classifier model. Training the classifier model includes accessing training utterances associated with the skillbots, the training utterances including respective training utterances associated with each of the skillbots. Each skillbot is configured to provide a dialogue with a user. The training further includes generating a training feature vector from the training utterance. The training feature vector includes a respective training feature vector associated with each of the skillbots. The training further includes generating a plurality of set representations of the training feature vector. Each set representation of the plurality of set representations corresponds to a subset of the training feature vector. The training further includes configuring the classifier model to compare the input feature vector to the plurality of set representations. The masterbot is configured to access an input utterance as a user input and generate an input feature vector from the input utterance. The masterbot is further configured to compare the input feature vector with the plurality of set representations of the training feature vector using the classifier model and output an indication that the user input cannot be addressed by the skillbot based on the input feature vector being outside the range of the plurality of set representations.

[0005] In an additional or alternative embodiment, the method described herein includes accessing, by a computer system, training utterances associated with a skillbot, the training utterances including a respective subset of training utterances for each of the skillbots. Each skillbot is configured to provide a dialogue with a user. The method further includes generating training feature vectors from the training utterances, the training feature vectors including a respective training feature vector for each of the training utterances. The method further includes determining centroid positions for clusters in a feature space, and assigning each training feature vector to a respective cluster from among the clusters having a respective centroid position to which the training feature vector is closest. The method further includes iteratively modifying the clusters until a stopping condition is met. Modifying the clusters includes increasing the count of the cluster to an updated count, determining a new centroid position for the cluster by an amount equal to the updated count, and reassigning the training feature vector to the cluster based on its proximity to the new centroid position. The method further includes determining boundaries of the clusters, the boundaries including a respective boundary for each of the clusters. The method further includes accessing an input utterance, converting the input utterance into an input feature vector, and determining that the input feature vector is outside the boundaries of the cluster by comparing the input feature vector with the boundaries of the cluster. Additionally, the method includes outputting an indication that the input utterance cannot be addressed by the skill bot based on the input feature vector being outside the boundaries of the cluster in the feature space.

[0006] In further additional or alternative embodiments, the method described herein includes accessing, by a computer system, training utterances associated with a skillbot, the training utterances including a respective subset of training utterances for each skillbot of the skillbot. Each skillbot is configured to provide a dialogue with a user. The method further includes generating training feature vectors from the training utterances, the training feature vectors including a respective training feature vector for each training utterance of the training utterances, and the method further includes dividing the training utterances into conversation categories. The method further includes generating composite feature vectors corresponding to the conversation categories. The generating composite feature vectors includes generating, for each conversation category of the conversation categories, a respective composite feature vector as a collection of the respective training feature vectors of the training utterances in the conversation category. The method further includes accessing an input utterance, converting the input utterance to an input feature vector, and determining that the input feature vector is not sufficiently similar to the composite feature vector by comparing the input feature vector to the composite feature vector. Additionally, the method includes outputting an indication that the input utterance cannot be addressed by the skill bot based on the input feature vector being insufficiently similar to the synthetic feature vector.

[0007] In further additional or alternative embodiments, the system described herein includes a masterbot configured to perform operations including: accessing an input utterance as a user input, generating an input feature vector from the input utterance, comparing the input feature vector with a plurality of set representations of training feature vectors using a classifier model, and outputting an indication that the user input cannot be acted upon by the skillbot based on the input feature vector being outside the plurality of set representations. Each of the skillbots is configured to provide an interaction with a user.

[0008] In further additional or alternative embodiments, the method described herein includes accessing an input utterance. The method further includes converting the input utterance into an input feature vector. The method further includes determining that the input feature vector is outside the boundary of the cluster by comparing the input feature vector to a boundary of the cluster in a feature space. The method further includes outputting an indication that the input utterance cannot be addressed by the skill bot based on the input feature vector being outside the boundary of the cluster in the feature space. Each of the skill bots is configured to provide an interaction with a user.

[0009] In further additional or alternative embodiments, the method described herein includes accessing an input utterance. The method further includes converting the input utterance into an input feature vector. The method further includes determining that the input feature vector is not sufficiently similar to the composite feature vector by comparing the input feature vector to a composite feature vector. The composite feature vector corresponds to a conversation category. The method further includes outputting an indication that the input utterance cannot be addressed by a skill bot based on the input feature vector being not sufficiently similar to the composite feature vector. Each of the skill bots is configured to provide a dialogue with a user.

[0010] In further additional or alternative embodiments, the method described herein includes accessing an input utterance as a user input. The method further includes generating an input feature vector from the input utterance. The method further includes comparing the input feature vector to a plurality of set representations of the training feature vector using a classifier model. The method further includes outputting an indication that the user input cannot be addressed by a skill bot based on the input feature vector being outside of the plurality of set representations. Each of the skill bots is configured to provide an interaction with a user.

[0011] In yet additional or alternative embodiments, the method described herein is used to train a classifier model. The method includes accessing training utterances associated with a skillbot, the training utterances including a respective training utterance associated with each of the skillbots. Each of the skillbots is configured to provide a dialogue with a user. The method further includes generating training feature vectors from the training utterances, the training feature vectors including a respective training feature vector associated with each of the skillbots. The method further includes generating a plurality of set representations of the training feature vectors. Each set representation of the plurality of set representations corresponds to a subset of the training feature vector. The method further includes configuring the classifier model to compare an input feature vector to the plurality of set representations of the training feature vector.

[0012] In a further additional or alternative embodiment, the method described herein is used to generate clusters that can be used to determine whether an input utterance can be addressed by a skillbot. The method includes accessing, by a computer system, training utterances associated with a skillbot, the training utterances including a respective subset of training utterances for each of the skillbots. Each of the skillbots is configured to provide a dialogue with a user. The method further includes generating training feature vectors from the training utterances, the training feature vectors including a respective training feature vector for each of the training utterances. The method further includes determining centroid positions for clusters in a feature space. The method further includes assigning each training feature vector to a respective cluster of the clusters to which the training feature vector is closest, and iteratively refining the clusters until a stopping condition is met. Revising the cluster includes increasing a count of the cluster to an updated count, determining a new centroid location for the cluster by an amount equal to the updated count, and reassigning the training feature vectors to the cluster based on their proximity to the new centroid location. The method further includes determining boundaries of the cluster, the boundaries including a respective boundary for each cluster of the cluster.

[0013] In further additional or alternative embodiments, the method described herein is used to generate a composite feature vector that can be used to determine whether an input utterance can be addressed by a skillbot. The method further includes accessing, by a computer system, training utterances associated with a skillbot, the training utterances including a respective subset of training utterances for each skillbot of the skillbot. Each skillbot of the skillbot is configured to provide a dialogue with a user. The method further includes generating training feature vectors from the training utterances, the training feature vectors including a respective training feature vector for each training utterance of the training utterances. The method further includes dividing the training utterances into conversation categories. The method further includes generating a composite feature vector corresponding to the conversation categories. The generating the composite feature vector includes generating a respective composite feature vector for each conversation category of the conversation categories as a collection of the respective training feature vectors of the training utterances in the conversation category.

[0014] In further additional or alternative embodiments, the systems described herein include means for carrying out any of the methods described above.

[0015] The above, together with other features and embodiments, will become more apparent with reference to the following specification, the appended claims, and the accompanying drawings. [Brief description of the drawings]

[0016] [Figure 1] FIG. 1 is a block diagram illustrating an environment including a master bot that communicates with various skill bots, also referred to as chat bots, according to some embodiments described herein. [Diagram 2]A flow diagram showing a method for configuring and using a masterbot to direct input utterances to skillbots and, if applicable, determine that a particular input utterance is irrelevant to available skillbots, according to some embodiments described herein. [Diagram 3] FIG. 1 is a block diagram illustrating a master bot configured to determine whether an input utterance is irrelevant to available skill bots, according to certain embodiments described herein. [Figure 4] FIG. 1 illustrates a SkillBot, in accordance with certain embodiments described herein. [Diagram 5] FIG. 1 is a flow diagram illustrating a method for initializing a classifier model of a master bot to determine whether an input utterance is irrelevant to available skill bots, according to some embodiments described herein. [Figure 6] FIG. 2 illustrates the generation of training feature vectors from training utterances used to train a classifier model, according to some embodiments described herein. [Figure 7] FIG. 1 is a flow diagram illustrating an example of a method for using a classifier model of a masterbot to determine whether an input utterance is unrelated to any available skillbot associated with the masterbot, according to some embodiments described herein. [Figure 8] FIG. 11 is a flow diagram illustrating another example of a method for using a classifier model of a masterbot to determine whether an input utterance is unrelated to any available skillbot associated with the masterbot, according to some embodiments described herein. [Figure 9] FIG. 2 illustrates an example of a feature space that includes points representing feature vectors of an example utterance, according to some embodiments described herein. [Figure 10] 10 illustrates an example of the feature space of FIG. 9 with class boundaries between intent classes of feature vectors of an example utterance, according to certain embodiments described herein. [Figure 11]FIG. 10 illustrates an example of the feature space of FIG. 9 with class boundaries separating feature vectors associated with a common intent into respective clusters, according to certain embodiments described herein. [Figure 12] FIG. 10 illustrates an example of the feature space of FIG. 9 with overlapping class boundaries that separate feature vectors associated with a common intent into respective clusters, according to certain embodiments described herein. [Figure 13] FIG. 10 illustrates another example of the feature space of FIG. 9 with class boundaries separating the feature vectors into clusters, according to certain embodiments described herein. [Figure 14] A flow diagram showing a method for initializing a classifier model of a master bot to determine whether an input utterance is irrelevant to available skill bots using clusters, according to some implementations described herein. [Figure 15] 15 illustrates an example implementation of certain aspects of the method of FIG. 14 in accordance with certain embodiments described herein. [Figure 16] FIG. 1 is a flow diagram illustrating a method for using a classifier model of a masterbot to determine whether an input utterance is unrelated to any available skillbot associated with the masterbot, according to some embodiments described herein. [Figure 17] FIG. 2 illustrates an example of performing the method when the input feature vector is outside all cluster boundaries, according to some embodiments described herein. [Figure 18] FIG. 1 is a flow diagram illustrating a method for initializing a classifier model of a masterbot to determine whether an input utterance is irrelevant to available skillbots using a synthetic feature vector, according to some embodiments described herein. [Figure 19] FIG. 1 illustrates the generation of a synthetic feature vector using intent-based speech categories, according to some embodiments described herein. [Figure 20]FIG. 1 illustrates the generation of a synthetic feature vector using bot-based conversation categories, according to some embodiments described herein. [Figure 21] FIG. 1 is a flow diagram illustrating a method for using a classifier model of a masterbot to determine whether an input utterance is unrelated to any available skillbot associated with the masterbot, according to some embodiments described herein. [Figure 22] A flow diagram illustrating an example of a method for selecting a skill bot to address an input utterance, according to some embodiments described herein. [Diagram 23] FIG. 13 illustrates another example of a method for selecting a skill bot to address an input utterance, according to some embodiments described herein. [Figure 24] FIG. 1 illustrates a distributed system for implementing some embodiments described herein. [Diagram 25] FIG. 1 illustrates a cloud-based system environment in which various chatbot-related services may be provided as cloud services, according to some embodiments described herein. [Figure 26] FIG. 1 illustrates an example of a computer system that may be used to implement some embodiments described herein. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0017] Detailed Description In the following description, for purposes of explanation, specific details are set forth in order to provide a thorough understanding of particular embodiments. It will be apparent, however, that various embodiments may be practiced without these specific details. The figures and description are not intended to be limiting. The word "exemplary" is used herein to mean "serving as an example, instance, or illustration." Any embodiment or design described herein as "exemplary" is not necessarily to be construed as preferred or advantageous over other embodiments or designs.

[0018] Various embodiments are described herein, including methods, systems, non-transitory computer-readable storage media that store programs, code or instructions executable by one or more processors, and the like.

[0019] As mentioned above, certain environments include multiple chatbots, also referred to herein as skillbots, such that each chatbot is specialized to address a respective set of tasks or skills. It would then be advantageous to automatically direct user input from a user to the chatbot best suited to address the user input, and also to quickly identify when the user input is irrelevant to the chatbot and therefore cannot be addressed by any of the available chatbots. Some embodiments described herein enable a computing system, such as a masterbot, to perform preliminary processing on the user input to determine whether the user input is irrelevant to the chatbot (i.e., cannot be addressed by the chatbot). As a result, computing resources are not wasted by the chatbot attempting to process irrelevant user input. Thus, certain embodiments described herein conserve computing resources of the chatbot, and by detecting irrelevant user input early, they prevent chatbots in an environment from processing such user input. Furthermore, certain embodiments conserve network resources by preventing the masterbot from sending such user input to one or more chatbots that cannot address such user input.

[0020] In an environment that includes multiple chatbots, the masterbot may include a classifier that determines which chatbot should process the input utterance. For example, the classifier may be implemented as a neural network or some other ML model that outputs a set of probabilities. Each probability is associated with a chatbot and indicates a confidence level that the chatbot can handle the input utterance. This type of masterbot selects the chatbot with the highest confidence level and forwards the input utterance to that chatbot. However, the masterbot may also provide a falsely high confidence score for an irrelevant input utterance, since the confidence is effectively divided among the available chatbots without any consideration as to whether the chatbot is configured to handle the input utterance. This may result in the chatbot processing an input utterance that it is not defined to handle. The chatbot may eventually provide an indication that it cannot handle the input utterance or a request for clarification to the user. However, this may occur after the chatbot has consumed resources to process the input utterance.

[0021] Alternatively, developers may train the masterbot to help it recognize utterances that cannot be addressed by any available chatbot. For example, the masterbot may be trained with labeled training data that includes relevant utterances (i.e., that can be addressed by the chatbot) and irrelevant utterances (i.e., that cannot be addressed by the chatbot) to teach the masterbot to recognize irrelevant utterances. However, it is unlikely that the range of training data will be sufficient to enable the masterbot to recognize all irrelevant utterances. Input utterances that are not similar to any of the irrelevant utterances used during training may end up being forwarded to the chatbot for processing. As a result, computational resources for training the masterbot will be increased, and the masterbot will still fail to recognize a wide range of irrelevant utterances.

[0022] Certain embodiments described herein address the shortcomings of the above-mentioned techniques and may be used in place of or in conjunction with such techniques. In certain embodiments, a trained machine-learning (ML) model, referred to herein as a classifier model, is used to make a determination as to whether an input utterance (i.e., a natural language phrase, possibly in text form) provided as a user input is relevant to any of the skillbots in the set of available skillbots. This determination may be performed, for example, by a masterbot utilizing a classifier model. The masterbot generates an input feature vector from the input utterance. In one example, the masterbot's classifier model compares the feature vector to a set of clusters of training feature vectors, which are feature vectors of training data, to determine whether the input feature vector falls within any of the clusters. If the input feature vector is outside the range of all the clusters, the masterbot determines that the input utterance is irrelevant to the skillbot. In another example, the masterbot's classifier model compares the input feature vector to a set of composite feature vectors. Each composite feature vector represents one or more training feature vectors belonging to a respective category. If the input feature vector is not sufficiently similar to any of the composite feature vectors, the masterbot determines that the input utterance is not relevant to the skillbot. If the input utterance is determined to be not relevant to any bot, it may be considered "none class" and not routed to any bot for action. Instead, processing of the input utterance may be terminated or the masterbot may prompt the user to clarify what the user intended.

[0023] Such improved routing may prevent computational resources from being directed to addressing queries that may ultimately not be addressed effectively by the system, which may improve the overall responsiveness of the system. In addition, the user experience may be improved, as users may be guided to rephrase or clarify their queries so that they are more easily addressed by a properly trained specialized chatbot or set of chatbots. Thus, in addition to saving processing and network resources, the average time to address user queries may be reduced by such improved routing through the bot network.

[0024] If the input utterance is determined to be relevant to at least one bot, the input utterance may be subjected to intent classification. Intent classification, the intent that most closely matches the input utterance is determined to initiate the conversation flow associated with that intent. For example, each intent of a skill bot may be associated with a state machine that defines various conversational states for a conversation with a user. Intent classification may be performed at an individual bot level. For example, each skill bot registered with a master bot may have its own classifier (e.g., an ML-based classifier) ​​that is trained on the given utterances associated with that particular bot. The input utterance may be input to the intent classifier of the bot that most closely matches the input utterance to determine which of the bot's intents best matches the utterance.

[0025] This may result in an improved user experience because a user's query may be quickly routed to a selected skill bot or a selected set of skill bots that may be better equipped to handle the user's query than a non-specialized general bot or may handle the user's query more quickly than a non-specialized general bot. The improved routing may additionally or alternatively result in fewer computational resources being used because the selected skill bot or the selected set of skill bots may consume fewer processing resources in handling a user query than a non-specialized general bot.

[0026] Overview of an Exemplary Chatbot System FIG. 1 is a block diagram of an environment including a masterbot 114 that communicates with various skillbots 116, also referred to as chatbots, according to some embodiments described herein. The environment includes a digital assistant builder platform (DABP) 102 that allows developers to program and deploy a digital assistant (DA) 106 or chatbot system. The DA 106 includes or accesses a masterbot 114 that includes or accesses one or more skillbots 116. Each skillbot 116 is configured to provide one or more skills or tasks to a user. In some embodiments, the masterbot 114 and the skillbots 116 run on the DA 106 itself. However, alternatively, only the masterbot 114 runs on the DA 106 and communicates with skillbots 116 running elsewhere (e.g., on another computing device).

[0027] In some embodiments, the DABP 102 can be used to program one or more DAs 106. For example, as shown in FIG. 1, a developer can use the DABP 102 to create and deploy a digital assistant 106 for a user to access. For example, the DABP 102 can be used by a bank to create one or more digital assistants for use by the bank's customers. The same DABP 102 platform can be used by multiple businesses to create digital assistants 106. As another example, a restaurant (e.g., a pizza place) owner might use the DABP 102 to create and deploy a digital assistant that allows the restaurant's customers to order food (e.g., order pizza). Additionally or alternatively, for example, the DABP 102 can be used by a developer to deploy one or more skillbots 116 such that the one or more skillbots 116 are accessible to the masterbot 114 of an existing digital assistant 106.

[0028] Additionally or alternatively, in some embodiments, as described further below, DABP 102 is configured to train digital assistant Masterbot 114 to enable Masterbot 114 to recognize when an input utterance is unrelated to any of the available Skillbots 116.

[0029] For purposes of this disclosure, a "digital assistant" is an entity that helps a user of the digital assistant accomplish various tasks through natural language conversation. A digital assistant may be implemented using software alone (e.g., a digital assistant is a digital entity implemented using programs, codes, or instructions executable by one or more processors), using hardware, or using a combination of hardware and software. A digital assistant may be embodied or implemented in a variety of physical systems or devices, which may include general-purpose or dedicated hardware, such as computers, mobile phones, watches, appliances, vehicles, etc. A digital assistant may also be referred to as a chatbot system. Thus, for purposes of this disclosure, the terms digital assistant and chatbot system may be synonymous.

[0030] In some embodiments, the digital assistant 106 can be used to perform various tasks through a natural language-based conversation between the digital assistant and its user 108. As part of the conversation, the user can provide responses 112 to user inputs 110 and, optionally, provide user inputs 110 to the digital assistant 106, which can perform one or more tasks related to the user inputs 110 (e.g., if the user input includes instructions to perform the tasks). A conversation or dialogue can include one or more user inputs 110 and responses 112. Through the conversation, the user can request one or more tasks to be performed by the digital assistant 106, which is accordingly configured to perform the tasks requested by the user and reply to the user with appropriate responses.

[0031] The user input 110 may be in the form of natural language, referred to as speech. The speech may be in the form of text, such as when the user types a phrase, such as a sentence, a question, a text fragment, or even a single word, and provides the text as input to the digital assistant 106. In some embodiments, the user utterance may be in the form of voice input or speech, such as when the user speaks something that is provided as input to the digital assistant 106. The digital assistant 106 may include or have access to a microphone for capturing such speech. The speech is typically in the language spoken by the user. If the speech is in the form of voice input, the voice input may be converted into text speech in the same language, and the digital assistant 106 may process the text speech as user input. Various speech-to-text processing techniques may be used to convert the voice input into text speech. In some embodiments, the voice-to-text conversion is performed by the digital assistant 106 itself, although various implementations are within the scope of this disclosure. For purposes of this disclosure, it is assumed that the input utterances (i.e., utterances provided as user input) are text utterances provided directly by a user 108 of the digital assistant 106 or are the result of converting an input voice utterance into text form. However, this is not intended to be limiting or restrictive in any manner.

[0032] An utterance can be a fragment, a sentence, multiple sentences, one or more words, one or more questions, or a combination of the above types, etc. In some embodiments, the digital assistant 106, including its corresponding masterbot 114 and skillbot 116, is configured to apply natural language understanding (NLU) techniques to the utterance to understand the meaning of the user input. As part of the NLU processing for the utterance, the digital assistant 106 may perform processing to understand the meaning of the utterance, which involves identifying one or more intents and one or more entities that correspond to the utterance. Upon understanding the meaning of the utterance, the digital assistant 106, including its masterbot 114 and skillbot 116, may perform one or more actions or operations depending on the understood meaning or intent.

[0033] For example, a user input may request an order for a pizza by providing an input utterance such as "I want to order a pizza." Upon receiving such an utterance, the digital assistant 106 determines the meaning of the utterance and takes appropriate action. The appropriate action may include, for example, responding to the user with a question that requests user input regarding the type of pizza the user wants to order, the size of the pizza, or any toppings on the pizza. The responses 112 provided by the digital assistant 106 may also be in natural language form, typically in the same language as the input utterance. As part of generating these responses 112, the digital assistant 106 may perform natural language generation (NLG). Throughout the conversation between the user and the digital assistant 106, in order for the user to order a pizza, the digital assistant may guide the user to provide all the information necessary to order the pizza, and may further have the user order the pizza at the end of the conversation. The digital assistant 106 may end the conversation by outputting information to the user indicating that the pizza has been ordered.

[0034] At a conceptual level, the digital assistant 106, together with its masterbot 114 and associated skillbot 116, performs various processing in response to utterances received from a user. In some embodiments, this processing includes a series or sequence of processing steps including, for example, understanding the meaning of the input utterance (using NLU), determining an action to be performed in response to the utterance, causing the action to be performed at the appropriate time, generating a response to be output to the user in response to the utterance, and outputting the response to the user. NLU processing may include parsing the received input utterance to understand the structure and meaning of the utterance, and refining and refining the utterance to develop a form (e.g., a logical form) that is more easily parsed and understood. Generating a response may include using NLG techniques. Thus, the natural language processing performed by the digital assistant 106 may include a combination of NLU processing and NLG processing.

[0035] The NLU processing performed by the digital assistant 106 may include various NLU operations such as sentence analysis (e.g., tokenization, lemma classification, identifying part-of-speech tags, identifying named entities, generating dependency trees to represent sentence structure, splitting the utterance into sentence clauses, analyzing individual clauses, breaking down anaphora, performing chunking, etc.). In certain embodiments, the NLU processing, or parts thereof, are performed by the digital assistant 106 itself. Additionally or alternatively, the digital assistant 106 may use other resources to perform some parts of the NLU processing. For example, the syntax and structure of the input utterance may be identified by processing the input utterance with a parser, a part-of-speech tagger, or a named entity recognizer separate from the digital assistant 106.

[0036] Although various examples provided in this disclosure show speech in English, this is intended as an example only. In certain embodiments, the digital assistant 106 can additionally or alternatively address speech in languages ​​other than English. The digital assistant 106 can provide subsystems (e.g., components that implement NLU functionality) configured to perform processing for various languages. These subsystems may be implemented as pluggable units that can be invoked using service calls from an NLU core server that may run on the digital assistant 106. This makes NLU processing flexible and extensible for each language, including allowing processing in various orders. Language packs may be provided for individual languages. A language pack can register a list of subsystems that can be provided from the NLU core server.

[0037] In some embodiments, the digital assistant 106 may be available or accessible to its user 108 through a variety of different channels, for example, through a specific application, through a social media platform, through various messaging services and applications (e.g., instant messaging applications), or through other applications or channels. A single digital assistant may have several channels configured such that it can be executed and accessed simultaneously by different services. Additionally or alternatively, the digital assistant 106 may be implemented on a device local to the user and thus may be a personal digital assistant used by the user or other nearby users.

[0038] The digital assistant 106 may be associated with one or more skills. In certain embodiments, these skills are realized through individual chatbots, referred to as skillbots 116, each of which is configured to interact with a user to accomplish a particular type of task, such as tracking inventory, submitting a timecard, creating an expense report, ordering food, checking a bank account, making a reservation, purchasing a widget, or other task. For example, for the embodiment shown in FIG. 1, the digital assistant 106 includes or has access to three skillbots 116, each of which realizes a particular skill or set of skills. However, a varying amount of skillbots 116 may be supported by the digital assistant 106. Each skillbot 116 may be realized as hardware, software, or a combination of both.

[0039] Each skill associated with the digital assistant 106, and thereby implemented as a skillbot 116, is configured to assist a user of the digital assistant 106 in completing a task through a conversation with the user. The conversation may include a combination of text or voice user input 110 provided by the user and responses 112 provided by the skillbot 116 via the digital assistant 106. These responses 112 may be in the form of text or voice messages to the user, or may be provided using simple user interface elements (e.g., selection lists) that are presented to the user for the user to make a selection.

[0040] There are various ways in which a skillbot 116 can be associated with or added to a digital assistant. In some cases, the skillbot 116 can be developed by a company and can be added to the digital assistant 106 using the DABP 102, for example, via a user interface provided by the DABP 102 to register the skillbot 116 with the digital assistant 106. In other cases, the skillbot 116 can be developed and created using the DABP 102 and can be added to the digital assistant 106 using the DABP 102. In still other cases, the DABP 102 provides an online digital store (referred to as a "skill store") that offers multiple skills directed to a wide range of tasks. Skills offered through the skill store can also expose various cloud services. To add a skill to the digital assistant 106 using the DABP 102, a developer can access the skill store via the DABP 102, select the desired skill, and indicate that the selected skill should be added to the digital assistant 106. Skills from the skill store can be added to the digital assistant as is or in modified form. A DABP 102 that is added to a digital assistant 106 in the form of a skillbot 116 to add a skill can configure a masterbot 114 of the digital assistant 106 to communicate with the skillbot 116. In addition, the DABP 102 can configure the masterbot 114 with skillbot data that enables the masterbot 114 to recognize utterances that can address and thereby update data used by the masterbot 114 to determine whether an input utterance is unrelated to any of the skillbots 116 of the digital assistant 106. Operations for configuring the masterbot 114 in this manner are described in more detail below.

[0041] The skill bots 116 and therefore the skills available to the digital assistant 106 can vary widely. For example, in the case of a digital assistant 106 developed for an enterprise, the digital assistant's master bot 114 may interface with skill bots 116 having specific functions, e.g., a CRM bot to perform functions related to customer relationship management (CRM), an ERP bot to perform functions related to enterprise resource planning (ERP), an HCM bot to perform functions related to human capital management (HCM), etc. A variety of other skills may also be available to the digital assistant and may depend on the intended use of the digital assistant 106.

[0042] Various architectures may be used to realize the digital assistant 106. In certain embodiments, the digital assistant 106 may be realized using a master-child paradigm or architecture. According to this paradigm, the digital assistant 106 acts as a masterbot 114 by including or accessing a masterbot 114 and interacts with one or more child bots, which are skillbots 116. The skillbots 116 may or may not run directly on the digital assistant 106. In the example shown in FIG. 1, the digital assistant 106 includes (i.e., accesses and uses) the masterbot 114 and three skillbots 116. However, an excessive amount of skillbots 116 may also be used, and the amount may change over time as skills are added to or removed from the digital assistant 106.

[0043] A digital assistant 106 implemented according to the master-child architecture allows a user of the digital assistant 106 to interact with multiple skillbots 116, thereby making it possible to utilize multiple skills that may be implemented separately through a unified user interface, i.e., through the masterbot 114. In some embodiments, when a user engages with the digital assistant 106, user input is received by the masterbot 114. The masterbot 114 then performs preliminary processing to determine the meaning of the input utterance that acts as the user input. In this case, the input utterance may be, for example, the user input itself or a text version of the user input. The masterbot 114 determines whether the input utterance is irrelevant to the skillbot 116 available for the input utterance. This may be the case, for example, when the input utterance requires a skill other than the skill of the skillbot 116. If the input utterance is irrelevant to the skillbot 116, the masterbot 114 may return an indication to the user that the input utterance is irrelevant to the skillbot 116. For example, the digital assistant 106 may ask the user for clarification or may report that the input utterance was not understood. However, if the masterbot 114 identifies an appropriate skillbot 116, the masterbot 114 may route the input utterance, and thus the ongoing conversation, to that skillbot 116. This allows a user to interact with a digital assistant 106 with multiple skillbots 116 through a common interface.

[0044] 1 illustrates a digital assistant 106 including a masterbot 114 and a skillbot 116, but this is not intended to be limiting. In some embodiments, the digital assistant 106 may include various other components, such as other systems or subsystems that provide functionality for the digital assistant 106. These systems and subsystems may be realized solely in software (e.g., as code stored on a computer-readable medium and executable by one or more processors), in hardware, or in an implementation using a combination of software and hardware.

[0045] In certain embodiments, the masterbot 114 is configured to recognize the available skillbots 116. For example, the masterbot 114 may have access to metadata identifying the various available skillbots 116 and the capabilities of each skillbot 116, including the tasks that may be performed by that skillbot 116. Upon receiving a user request in the form of an input utterance, the masterbot 114 is configured to identify or predict a particular skillbot 116 from among the multiple available skillbots 116 that will best function or best be able to address the user request, or in the alternative, to determine that the input utterance is unrelated to any of the skillbots 116. If it is determined that a skillbot 116 can address the input utterance, the masterbot 114 may route the input utterance or at least a portion of the input utterance to that skillbot 116 for further addressing. Thus, control continues from the masterbot 114 to the skillbot 116.

[0046] In some embodiments, the DABP 102 provides an infrastructure and various services and features that enable a developer user of the DABP 102 to create a digital assistant 106 that includes one or more skillbots 116. In some cases, a skillbot 116 can be created by cloning an existing skillbot 116, for example, by cloning a skillbot 116 provided in a skill store. As described above, the DABP 102 can provide a skill store that provides multiple skillbots 116 for performing various tasks. A user of the DABP 102 can clone a skillbot 116 from the skill store, and may modify or customize the cloned skillbot 116 as needed. In some other cases, a developer user of the DABP 102 creates a skillbot 116 from scratch, such as by using tools and services provided by the DABP 102.

[0047] In particular embodiments, at a high level, creating or customizing a skillbot 116 involves the following operations.

[0048] (1) Configure the settings for the new skill bot. (2) Configure one or more intents for the skill bot.

[0049] (3) Construct one or more entities for one or more intents. (4) Train the skill bot.

[0050] (5) Create a dialogue flow for the skill bot. (6) Add custom components to your skill bot as needed.

[0051] (7) Test and deploy the skill bot. Each of the above operations will now be briefly described.

[0052] (1) Configure settings for a new skillbot 116. Various settings may be configured for a skillbot 116. For example, a skillbot developer may specify one or more invocation names for the skillbot 116 being created. These invocation names, which serve as identifiers for the skillbot 116, may further be used by a user of the digital assistant 106 to specifically invoke the skillbot 116. For example, a user may include an invocation name in the user's input utterance to specifically invoke the corresponding skillbot 116.

[0053] (2) Construct one or more intents and associated example utterances for the skillbot 116. The designer of the skillbot 116 specifies one or more intents, also referred to as chatbot intents, for the skillbot 116 being created. The skillbot 116 is then trained based on these specified intents. These intents represent categories or classes for which the skillbot 116 is trained to infer about input utterances. Upon receiving an utterance, the trained skillbot 116 infers (i.e., determines) the intent for the utterance. In this case, the inferred intent is selected from a set of predefined intents used to train the skillbot 116. The skillbot 116 then takes an appropriate action in response to the utterance based on the intent inferred for the utterance. In some cases, the intents for the skillbot 116 represent tasks that the skillbot 116 can perform for a user of the digital assistant. Each intent is given an intent identifier or intent name. For example, for a skill bot 116 trained for banking, the intents specified for the skill bot 116 may include "CheckBalance," "TransferMoney," "DepositCheck," etc.

[0054] For each intent defined for the skillbot 116, the designer of the skillbot 116 may also provide one or more example utterances that represent and illustrate the intent. These example utterances are intended to represent utterances that a user may input to the skillbot 116 for that intent. For example, for the CheckBalance intent, example utterances may include "What's my savings account balance?", "How much is in my checking account?", "How much money do I have in my account?", etc. Thus, various permutations of typical user utterances may be specified as example utterances for the intent.

[0055] The intents and their associated example utterances, also referred to as training utterances, are used as training data to train the skillbot 116. A variety of different training techniques may be used. This training results in a predictive model configured to receive an utterance as input and output an inferred intent for the utterance. In some cases, an input utterance provided as a user input is input to an intent analysis engine (e.g., a rule-based or ML-based classifier executed by the skillbot 116) configured to predict or infer an intent for the input utterance using the trained model. The skillbot 116 may then take one or more actions based on the inferred intent.

[0056] (3) Configuring entities for one or more intents of the skillbot 116. In some cases, additional context may be required to enable the skillbot 116 to respond appropriately to an input utterance. For example, there may be situations where multiple input utterances result in the same intent in the skillbot 116. For example, the utterances "What is the balance of my savings account?" and "How much is in my checking account?" both result in the same CheckBalance intent, but these utterances are different requests for different things. To disambiguate such requests, one or more entities may be added to the intent. Using the example of a banking skillbot 116, an entity called Account_Type that defines values ​​called "checking" and "savings" may enable the skillbot 116 to parse the user request and respond appropriately. In the above example, multiple utterances result in the same intent, but for these two utterances, the values ​​associated with the Account_Type entity are different. This allows the skillbot 116 to potentially perform different actions for these two utterances, even though the two utterances result in the same intent. One or more entities may be specified for a particular intent configured for the skillbot 116. Thus, entities are used to add context to the intent itself. Entities help to more fully describe the intent and enable the skillbot 116 to fulfill the user request.

[0057] In a particular embodiment, there are two types of entities: (a) embedded entities that may be provided by DABP 102, and (2) custom entities that may be specified by a developer. Embedded entities are generic entities that can be used with a wide variety of skill bots 116. Examples of embedded entities include entities related to time, date, address, number, email address, duration, circulation period, currency, phone number, uniform resource locator (URL), etc. Custom entities are used for more customized uses. For example, for a banking skill, the Account_Type entity may be defined by a developer to support various banking transactions by checking user input for keywords such as checking, savings, and credit card.

[0058] (4) Training the skillbot 116. The skillbot 116 is configured to receive user input in the form of utterances, parse or process the received user input, and identify or select an intent associated with the received user input. In some embodiments, the skillbot 116 must be trained for this, as described above. In certain embodiments, the skillbot 116 is trained based on intents configured for the skillbot 116 and example utterances associated with the intents (i.e., training utterances), which enable the skillbot 116 to resolve input utterances into one of its configured intents. In certain embodiments, the skillbot uses a predictive model trained with the training data to enable the skillbot to identify what the user is saying (or, in some cases, what the user is trying to say). DABP 102 may provide a variety of different training techniques that a developer can use to train the skillbot 116, including various ML-based training techniques, rule-based training techniques, or a combination thereof. In certain embodiments, a portion of the training data (e.g., 80%) is used to train the skillbot model, and another portion (e.g., the remaining 20%) is used to test or validate the model. Once trained, the trained model, also referred to as the trained skillbot 116, can be used to address and respond to input utterances. In certain cases, the input utterance may be a question that requires only a single answer but does not require further conversation. To address such situations, a question-and-answer (Q&A) intent may be defined for the skillbot 116. In some embodiments, the Q&A intent is created in a similar manner as other intents, although the dialogue flow for the Q&A intent may differ from that for normal intents.For example, the dialogue flow for a Q&A intent may not require prompting to solicit additional information from the user (e.g., values ​​for a particular entity) as may be the case for other intents.

[0059] (5) Create a dialogue flow for the skillbot 116. The dialogue flow specified for the skillbot 116 describes how the skillbot 116 will react when various intents for the skillbot 116 are resolved in response to the received user input 110. The dialogue flow defines the behavior or actions that the skillbot 116 will take, such as how the skillbot 116 will respond to user utterances, how the skillbot 116 will prompt the user for input, and how the skillbot 116 will return data. The dialogue flow is similar to a flowchart that the skillbot 116 follows. The designer of the skillbot 116 specifies the dialogue flow using a language such as Markdown language. In certain embodiments, a version of YAML called OBotML may be used to specify the dialogue flow for the skillbot 116. The dialogue flow definition for the skillbot 116 serves as a model for the conversation itself, allowing the designer of the skillbot 116 to choreograph the dialogue between the skillbot 116 and the user that the skillbot 116 serves.

[0060] In particular embodiments, the dialogue flow definition for a skillbot 116 includes three sections, which are described below.

[0061] (a) Context Section (b) Default transition section (c) Status Section Context Section: In the context section, the skill bot 116 developer can define variables that are used in the conversation flow. Other variables that can be named in the context section include, for example, variables for error handling, variables for built-in or custom entities, user variables that allow the skill bot 116 to recognize and maintain user preferences, etc.

[0062] Default transition section: Transitions for the skillbot 116 can be defined in the dialogue flow state section or the default transition section. Transitions defined in the default transition section act as fallbacks and are triggered when there is no applicable transition defined in a state or when the conditions required to trigger a state transition cannot be met. The default transition section can be used to define routing that allows the skillbot 116 to gracefully handle unexpected user actions.

[0063] State Section: A dialogue flow and its associated behaviors are defined as a set of temporary states that govern the logic within the dialogue flow. Each state node in the dialogue flow definition names a component that provides the functionality required at that point in the dialogue. In this way, we build states around the components. A state contains component-specific characteristics and defines transitions to other states that are triggered after the component is executed.

[0064] Special case scenarios can be addressed with the state section. For example, it may be desirable to give the user the option to temporarily leave a first skill in which the user is engaged and do something with a second skill within the digital assistant 106. For example, if the user is engaged in a conversation with a shopping skill (e.g., the user has made some selections for a purchase), the user may want to jump to a banking skill (e.g., the user wants to make sure the user has enough money for the purchase) and then return to the shopping skill to complete the user's order. To address this, a state section in the interaction flow definition of the first skill can be configured to initiate an interaction with a second, separate skill in the same digital assistant and then return to the original interaction flow.

[0065] (6) Adding Custom Components to the Skillbot 116: As described above, a state specified in the dialogue flow for the skillbot 116 nominates a component that provides the functionality required for that state. The components enable the skillbot 116 to perform the functionality. In certain embodiments, the DABP 102 provides a set of pre-configured components to perform a wide range of functions. A developer can select one or more of these pre-configured components and associate them with a state in the dialogue flow for the skillbot 116. A developer can also create custom or new components using tools provided by the DABP 102 and associate the custom components with one or more states in the dialogue flow for the skillbot 116.

[0066] (7) Testing and Deploying Skillbots 116: DABP 102 may provide several features that allow developers to test skillbots 116 as they are being developed. Additionally, skillbots 116 can be deployed and included in the digital assistant 106.

[0067] Although the above describes how to create a skillbot 116, similar techniques may be used to create a digital assistant 106 or a masterbot 114. At the masterbot or digital assistant level, built-in system intents may be configured for the digital assistant 106. In some embodiments, these built-in system intents are used to identify common tasks that the masterbot 114 can handle without invoking the skillbot 116. Examples of system intents defined for the masterbot 114 include: (1) Exit, which is applicable when a user signals a desire to end the current conversation or context in the digital assistant; (2) Help, which is applicable when a user seeks assistance or direction; (3) UnresolvedIntent, which is applicable to user input that does not match well with the Exit and Help intents. The masterbot 114 may store information about one or more skillbots 116 associated with the digital assistant 106. This information allows the masterbot 114 to select a particular skillbot 116 to address the utterance, or alternatively, to determine that the utterance is unrelated to any skillbot 116 of the digital assistant 106.

[0068] When a user inputs a phrase or utterance into the digital assistant 106, the masterbot 114 is configured to run a process that determines how to route the utterance and associated conversation. The masterbot 114 makes this decision using a routing model, which may be rule-based, ML-based, or a combination thereof. The masterbot 114 uses the routing model to determine whether the conversation corresponding to the utterance should be routed to a specific skillbot 116 for action, whether it should be actioned by the digital assistant 106 or the masterbot 114 itself per built-in system intent, whether it should be actioned as a different state in the current conversation flow, or whether it should be independent of any of the skillbots 116 associated with the digital assistant 106.

[0069] In certain embodiments, as part of this process, the masterbot 114 determines whether the input utterance clearly identifies the skillbot 116 with its invocation name. If the invocation name is present in the input utterance, the invocation name is treated as a clear invocation of the skillbot 116 corresponding to the invocation name. In such a scenario, the masterbot 114 may route the input utterance to the specifically invoked skillbot 116 for further handling. In the absence of a specific invocation or clear invocation, in certain embodiments, the masterbot 114 evaluates the input utterance and calculates (e.g., using a logistic regression model) a confidence score for the system intent and skillbot 116 associated with the digital assistant 106. The calculated score for the skillbot 116 or system intent represents how likely the input utterance is to represent a task that the skillbot 116 is configured to perform or to represent a system intent. Any system intent or skillbot 116 with an associated calculated confidence score above a threshold may be selected as a candidate for further evaluation. The masterbot 114 then selects a particular system intent or skillbot 116 from the identified candidates to further address the input utterance. In a particular embodiment, after one or more skillbots 116 are identified as candidates, the intents associated with those candidate skillbots 116 are evaluated for each skillbot 116, such as by using a trained model, and a confidence score is determined for each intent. In general, any intent with a confidence score above a threshold (e.g., 70%) is treated as a candidate intent. If a particular skillbot 116 is selected, the input utterance is routed to that skillbot 116 for further processing. Once a system intent is selected, one or more actions are performed by the masterbot 114 itself according to the selected system intent.

[0070] Some embodiments of the masterbot 114 described herein are configured to not only direct input utterances to an appropriate skillbot 116, if applicable, but also to determine whether to prompt some indication that the irrelevant input utterance is irrelevant when the input utterance becomes irrelevant to an available skillbot 116. Figure 2 is a flow diagram illustrating a method 200 of configuring and using a masterbot to direct input utterances to a skillbot 116 and, if applicable, determine that a particular input utterance is irrelevant to an available skillbot 116, according to some embodiments described herein. The method 200 of Figure 2 is a general overview of various specific cases that are described in detail below.

[0071] The method 200 shown in FIG. 2 and other methods described herein may be implemented in software (e.g., as code, instructions, or programs) executed by one or more processing units (e.g., processors or processor cores), in hardware, or a combination thereof. The software may be stored in a non-transitory storage medium, such as a memory device. The method 200 is intended to be exemplary and non-limiting. FIG. 2 depicts various operations occurring in a particular sequence or order, but this is not intended to be limiting. In certain embodiments, for example, these operations may be performed in a different order, or one or more operations of the method 200 may be performed in parallel. In certain embodiments, the method 200 may be performed by the training system and the masterbot 114.

[0072] 2, in block 205 of the method 200, a classifier model of the masterbot 114 is initialized. In some embodiments, the classifier model is configured to determine whether the input utterance is unrelated to any available skillbot 116. Additionally, in some embodiments, the classifier model may also be configured to route the input utterance to a suitable skillbot 116 for processing if the input utterance is deemed relevant to the skillbot 116.

[0073] In some embodiments, the initialization of the classifier model may include teaching the classifier model how to recognize unrelated input utterances. To this end, a training system, which may be part of the DABP 102, may have access to training utterances (i.e., example utterances) for each skillbot 116. The training system may generate respective training feature vectors that describe and represent each training utterance of the various skillbots 116 that are or will be available to the masterbot 114. Each training feature vector is a feature vector of the corresponding training utterance. The training system may then generate various set representations from the training feature vectors. Each set representation may represent a collection of training feature vectors. For example, such a collection may be a cluster or a composite feature vector, as described in more detail below. To initialize the classifier model, the training system may configure the classifier model to compare the input vector to various set representations that represent a collection of training feature vectors.

[0074] In block 210, the masterbot 114 receives user input 110 in the form of an input utterance. For example, a user may have typed the input utterance as the user input 110, or the user may have spoken the user input 110 that the digital assistant 106 converted into the input utterance. The input utterance may include a user request to be addressed by the digital assistant 106, and thus by the masterbot 114.

[0075] In block 215, the masterbot 114 utilizes a classifier model to determine whether the input utterance can be addressed by any of the skillbots 116 associated with the masterbot 114. In some embodiments, the classifier model of the masterbot 114 generates an input feature vector that describes and represents the input utterance. The classifier model may compare the input feature vector to a set representation of training feature vectors, and based on this comparison, the classifier model determines whether the input utterance is unrelated to any of the skillbots 116 available to the masterbot 114 (i.e., the skillbots 116 for which the training utterance is represented in the set representation).

[0076] Examples of classification at the digital assistant or skill bot level FIG. 3 is a block diagram illustrating a masterbot 114, also referred to as a masterbot (MB) system, according to certain embodiments described herein. The masterbot 114 can be implemented solely in software, solely in hardware, or in a combination of hardware and software. In some embodiments, the masterbot 114 includes a pre-processing subsystem 310 and a routing subsystem 320. The masterbot 114 illustrated in FIG. 3 is merely one example of an arrangement of components within the masterbot 114. Those skilled in the art will recognize many possible variations, alternatives, and modifications. For example, in some implementations, the masterbot 114 may have more or fewer systems or components than those illustrated in FIG. 3, may combine two or more subsystems, or may have a different configuration or arrangement of subsystems.

[0077] In some embodiments, the language processing subsystem 310 is configured to process user input 110 provided by a user. Such processing may include, for example, converting the user input 110 into text input utterances 303 using automatic speech recognition or some other tool if the user input is in voice form or some other form that is not text.

[0078] In some embodiments, the routing subsystem 320 is configured to determine (a) whether the input utterance 303 is unrelated to any available skillbot 116, and (b) if the input utterance 303 is related to at least one skillbot 116, which skillbot 116 is best suited to address the input utterance 303. In particular, the classifier model 324 of the routing subsystem 320 may be rule-based or ML-based, or a combination thereof, and may be configured to determine whether the input utterance 303 is unrelated to any available skillbot 116, represents a particular skillbot 116, or represents a particular intent configured for a particular skillbot 116. For example, as described above, a skillbot 116 may be configured with one or more chatbot intents. Each chatbot intent may have its own dialogue flow and may be associated with one or more tasks that the skillbot 116 can perform. If it is determined that the input utterance 303 represents a particular skillbot 116 or represents an intent for which the particular skillbot 116 has been configured, the routing subsystem 320 can invoke the particular skillbot 116 to convey the input utterance 303 as an input 335 for the particular skillbot 116. However, if the input utterance 303 is deemed unrelated to any available skillbot 116, the input utterance 303 is deemed to belong to a non-class 316, which is a class of utterances that cannot be addressed by the available skillbots. In that case, the digital assistant 106 may indicate to the user that the input utterance 303 cannot be addressed.

[0079] In some embodiments, the classifier model 324 may be implemented using a rule-based model or an ML-based model, or both. For example, in some embodiments, the classifier model 324 may include a rule-based model trained on training data 354 including example utterances. Thus, the rule-based model is used to determine whether an input utterance is unrelated to any available skill bot 116 (i.e., any skill bot 116 associated with the digital assistant 106). Additionally or alternatively, in some embodiments, the classifier model 324 may include a neural network trained on the training data 354. The training data 354 may include, for each skill bot 116, a corresponding set of example utterances (e.g., two or more example utterances for each intent for which the skill bot 116 is configured). 3, the available skillbots 116 include a first skillbot 116a, a second skillbot 116b, and a third skillbot 116c, and the training data 354 includes respective skillbot data for each such skillbot 116, including a first skillbot data 358a corresponding to the first skillbot 116a, a second skillbot data 358b corresponding to the first skillbot 116b, and a third skillbot data 358c corresponding to the first skillbot 116c. The skillbot data for a skillbot 116 may include training utterances (i.e., example utterances) that represent utterances that may be addressed by that skillbot 116 in particular. Thus, the training data 354 includes training utterances for each of the various skillbots 116. A training system 350, which may be, but need not be, integrated into the DABP 102, may use training data 354 to train the classifier model 324 of the Masterbot 114 to perform these tasks.

[0080] In some embodiments, the training system 350 uses the training data 354 to train the classifier model 324 to determine whether the input utterance is unrelated to any available skill bot 116. In general, the training system 350 may generate training feature vectors to describe and represent the training utterances in the training data 354. As described in more detail below, the training system 350 may generate set representations. Each set representation represents a collection of the training feature vectors. After training and during operation, the classifier model 324 may compare the input utterance 303 to the set representations to determine whether the input utterance 303 is unrelated to any available skill bot 116.

[0081] In some embodiments, the classifier model 324 includes one or more sub-models, each configured to perform a different task related to classifying the input utterance 303. In addition to or instead of the above, for example, the classifier model 324 may include an ML model or other type of model configured to determine which skill bot 116 is most suitable for the input utterance if the input utterance 303 is deemed relevant (i.e., not irrelevant) to at least one available skill bot 116. In some embodiments, the classifier model 324 may determine the most appropriate skill bot 116 while determining that the input utterance is relevant to at least one skill bot 116. In this case, no further determination needs to be made to identify the most suitable skill bot 116. However, alternatively, the classifier model 324 may determine (e.g., using a logistic regression model) for each skill bot 116 an associated confidence score indicating the likelihood that the skill bot 116 is most suitable to process the input utterance 303 (i.e., is most suitable for the input utterance). To this end, a neural network included in the classifier model 324 may be trained by a training system to determine the likelihood that the input utterance 303 represents each skill bot 116 or represents one of the intents configured for the skill bot 116. For example, the neural network may determine and output a respective confidence score associated with each skill bot 116. In this case, the confidence score for a skill bot indicates the likelihood that the skill bot 116 can address the input utterance 303 or is the best available skill bot 116 to address the input utterance. Taking the confidence scores into account, the routing subsystem 320 may select the skill bot 116 associated with the highest confidence score and route the input utterance 303 to that skill bot 116 for processing.

[0082] Figure 4 illustrates a diagram of a skillbot 116, also referred to as a skillbot system, according to certain embodiments described herein. The instance of skillbot 116 illustrated in Figure 4 can be used as the skillbot 116 of Figure 1 and can be implemented solely in software, solely in hardware, or in a combination of hardware and software. As illustrated in Figure 4, the skillbot 116 can include a bot classifier model 424 that determines the intent of an input utterance 303 and a conversation manager 430 that generates a response 435 based on the intent.

[0083] The bot classifier model 424 may be implemented using a rule-based model or an ML-based model, or both, and may receive as input the input utterance 303 routed by the master bot 114 to the skill bot 116. The bot classifier model 424 may access the rules 452 and the intent data 454 in a data store 450 on the skill bot 116 or may be accessible to the skill bot 116. For example, the intent data 454 may include example utterances or other data for each intent, and the rules 452 may describe how to use the intent data to determine an intent for the input utterance 303. The bot classifier model 424 may apply these rules to the input utterance 303 and the intent data 454 to determine the intent of the input utterance 303.

[0084] More specifically, in some embodiments, the bot classifier model 424 may operate in a manner similar to how the classifier model 324 of FIG. 3 determines the skill bot 116 to address the input utterance 303. For example, the bot classifier model 424 may determine (e.g., using a logistic regression model) a respective confidence level and assign the respective confidence level to each chatbot intent for which the skill bot 116 is configured in a manner similar to how the classifier model 324 may assign a confidence level to the skill bot 116. Thus, the confidence level indicates the highest likelihood that the respective chatbot intent is applicable to the input utterance 303. The bot classifier model 424 may then select the intent that is assigned the highest confidence level. In additional or alternative embodiments, the bot classifier model 424 determines the intent for the input utterance 303 by comparing the input feature vector describing the input utterance 303 to either or both of (a) a composite feature vector that includes one composite feature vector to represent a training feature vector for each intent, or (b) a cluster of training feature vectors, where each cluster represents an intent or multiple intents. The use of such feature vectors is described in more detail below.

[0085] Once the intent that the utterance 202 best represents has been identified, the bot classifier model 424 may communicate the intent indication 422 (i.e., an indication of the identified intent) to the conversation manager 430. In the embodiment of FIG. 4, the conversation manager 430 is shown as local to the skillbot 116. However, the conversation manager 430 may be shared with the masterbot 114 or across multiple skillbots 116. Thus, in some embodiments, the conversation manager 430 is local to the digital assistant or masterbot 114.

[0086] In response to receiving the intent indication 422, the conversation manager 430 may determine an appropriate response 435 to the utterance 202. For example, the response 435 may be an action or message specified in a dialogue flow definition 455 configured for the system 400 of the skillbot 116, and the response 435 may be used as the DA response 112 in the embodiment of FIG. 1. For example, the data store 450 may include various dialogue flow definitions including a respective dialogue flow definition for each intent, and the conversation manager 430 may access the dialogue flow definition 455 based on an identification of an intent corresponding to the dialogue flow definition 455. Based on the dialogue flow definition 455, the conversation manager 430 may determine a dialogue flow state as being the next state to transition to according to the dialogue flow definition 455. The conversation manager 430 may determine the response 435 based on further processing of the input utterance 303. For example, if the input utterance 303 is "Check balance in savings," the conversation manager 430 may transition to a dialogue flow state in which the user is presented with a dialogue related to the user's savings account. The conversation manager 430 may transition to this state based on an intent indication 422 indicating that the identified intent is the "CheckBalance" intent configured for the skill bot 116, and further based on recognizing that a value of "savings" was extracted for the "Account_Type" entity.

[0087] Using feature vectors to describe utterances As mentioned above, the classifier model 324 may use the feature vector in determining whether an input utterance is unrelated or related to any available skillbot 116. For purposes of this disclosure, a feature vector is a vector or set of coordinates that describes the characteristics of an utterance and thereby describes the utterance itself. A feature vector describing an utterance may be used to represent the utterance in a particular context, as described herein.

[0088] The concept of feature vectors is based on the concept of word embedding. Broadly speaking, word embedding is a type of language modeling in which words are mapped to corresponding vectors. A particular word embedding may map semantically similar words to similar regions of a vector space. Thus, similar words will be close to each other in the vector space, and dissimilar words will be far apart. A simple example of word embedding uses "one hot" encoding. In this case, each word in the dictionary is mapped to a vector with an amount of dimensionality equal to the size of the dictionary, such that the vector has a value of one in the dimension corresponding to the word itself and a value of zero in all other dimensions. For example, the first two words of the sentence "Go intelligent bot service artificial intelligence, Oracle" may be represented using the following "one hot" encoding:

[0089] [Table 1]

[0090] Feature vectors can be used to represent words, sentences, or phrases of various types. Considering the above simple example of word embeddings, a corresponding feature vector may map a sequence of words, such as an utterance, to a feature vector that is a collection of word embeddings for that sequence of words. The collection may be, for example, a sum, average, or weighted average.

[0091] For different utterances, the respective feature vectors may be different because such utterances contain words with widely different semantic meanings. However, utterances that are semantically similar and therefore contain words that are the same between utterances or are semantically similar between utterances may have similar feature vectors (i.e., feature vectors that are located close to each other in the vector space). Each feature vector corresponds to a single point in a vector space, also called a feature vector, which is the origin of the feature space plus the feature vector. In some embodiments, the points for semantically similar utterances are located close to each other. Throughout this disclosure, a feature vector and its corresponding point are referred to interchangeably, since the two provide different visual appearances for the same information.

[0092] Exemplary Method for Initializing a Masterbot Classifier Model In some embodiments, the classifier model 324 of the masterbot 114 utilizes the feature vector of the training utterance as a criterion for determining whether the input utterance is unrelated or related to any available skillbot 116. Figure 5 illustrates a method 500 for initializing the classifier model 324 of the masterbot 114 to perform this task, according to some embodiments described herein. For example, this method 500 or a similar method may be performed in block 205 of the method 200 for configuring and using the masterbot 114 to route the input utterance 303. Figure 5 is a schematic method 500, more detailed examples of which are shown and described with reference to Figures 14 and 18.

[0093] The method 500 shown in FIG. 5, as well as other methods described herein, may be implemented in software (e.g., as code, instructions, or programs) executed by one or more processing units (e.g., processors or processor cores), in hardware, or a combination thereof. The software may be stored in a non-transitory storage medium, such as a memory device. The method 500 is intended to be exemplary and non-limiting. FIG. 5 depicts various operations occurring in a particular sequence or order, but this is not intended to be limiting. In certain embodiments, for example, these operations may be performed in a different order, or one or more operations of the method 500 may be performed in parallel. In certain embodiments, the method 500 may be performed by a training system 350, which may be part of the DABP 102.

[0094] In block 505, the training system 350 accesses training utterances (also referred to as example utterances) for the various skillbots 116 associated with the masterbot 114. For example, the training utterances may be stored as skillbot data in training data 354 accessible to the training system 350. In some embodiments, each skillbot 116 available to the masterbot 114 may be associated with a subset of training utterances, and each such subset of training utterances may include a training utterance for each intent for which the skillbot 116 is configured. Thus, the complete set of training utterances may include a training utterance for each intent of each skillbot 116 such that every intent of every skillbot 116 is represented.

[0095] In block 510, the training system 350 generates a training feature vector from the training utterances accessed in block 505. As described above, the training utterances may include a subset associated with each skillbot 116 and may further include training utterances for each intent of each skillbot 116. Thus, in some embodiments, the training feature vector may include a respective subset for each skillbot 116 and may further include a respective feature vector for each intent of each skillbot.

[0096] Figure 6 illustrates the generation of training feature vectors 620 from training utterances 615, according to some embodiments described herein. Specifically, Figure 6 relates to training utterances 615 of skillbot data 358 in training data 354, which are associated with a particular skillbot 116. The skillbot 116 is configured to handle input utterances 303 of multiple intents, including intent A and intent B, and thus the training utterances 615 include training utterances 615 representing intent A and training utterances 615 representing intent B.

[0097] As described above, the training system 350 may generate training feature vectors 620 to describe and represent each training utterance 615. Various techniques are known for converting a sequence of words, such as the training utterance 615, into a feature vector, such as the training feature vector 620, and one or more of such techniques may be used. For example, the training system 350 may, but need not, encode each training utterance 615 into a corresponding training feature vector 620 using one-hot encoding or some other encoding.

[0098] Also, as described above, feature vectors can be represented as points. In some embodiments, each training feature vector 620 can be represented as a point 640 in the feature space 630. In this case, the feature space has a number of dimensions equal to the number of features (i.e., the number of dimensions) in the training feature vector 620. In the example of FIG. 6, two training feature vectors 620 representing the same intent, specifically intent A, are plotted as points 640 that are close to each other because the two training feature vectors 620 are semantically similar. However, it is not necessary that all training feature vectors 620 or all feature vectors for a particular intent be represented as points that are close to each other.

[0099] Returning to FIG. 5, in block 515, the training system 350 generates a plurality of set representations of the training feature vectors 620 generated in block 510. Each set representation represents a set of training feature vectors 620. As described in more detail below, a set representation may be, for example, a cluster of training feature vectors 620, or a composite feature vector that is a collection of multiple training feature vectors 620. In essence, a set representation may be an embodiment that represents multiple training feature vectors 620 grouped together. Each set of training feature vectors represented in a corresponding set representation may share a common intent, a common group of intents, a common skillbot 116, or a common region of the feature space 630, or alternatively, the training feature vectors represented in a single set representation need not have any commonality other than based on the training data 354.

[0100] In block 520, the training system 350 may configure the classifier model 324 to compare the input utterance 303 provided as the user input 110 to the various set representations. For example, the set representations may be stored in a storage device accessible to the classifier model 324 of the masterbot 114. The classifier model 324 may be programmed with rules for how to determine whether the input utterance matches or does not match the various set representations. The definition of a match may depend on the particular set representation being used, as described in more detail below.

[0101] 7 illustrates a method 700 for determining whether an input utterance 303 provided as a user input 110 is unrelated to any available skillbot 116 associated with the masterbot 114 using a classifier model 324 of the masterbot 114 according to some embodiments described herein. The method 700 or a similar method may be performed after initialization of the classifier model 324 and for each received input utterance 303. For example, the method 700 or a similar method may be performed in block 215 of a method 200 for configuring and using the masterbot 114 to route input utterances 303. FIG. 7 is a schematic method 700, more detailed examples of which are shown and described with reference to FIG. 16 and FIG. 21.

[0102] The method 700 shown in FIG. 7 and other methods described herein may be implemented in software (e.g., as code, instructions, or programs) executed by one or more processing units (e.g., processors or processor cores), in hardware, or a combination thereof. The software may be stored in a non-transitory storage medium, such as a memory device. The method 700 is intended to be exemplary and non-limiting. FIG. 7 depicts various operations occurring in a particular sequence or order, but this is not intended to be limiting. In certain embodiments, for example, these operations may be performed in a different order, or one or more operations of the method 700 may be performed in parallel. In certain embodiments, the method 700 may be performed by the masterbot 114.

[0103] At block 705 of the method 700, the masterbot 114 accesses the input utterance 303 provided as the user input 110. For example, in some embodiments, a user may provide the user input 110 in the form of voice input, and the digital assistant 106 may convert the user input 110 into a text input utterance 303 for use by the masterbot 114.

[0104] In block 710, the Masterbot 114 may generate an input feature vector from the input utterance 303 accessed in block 705. The input feature vector may describe and represent the input utterance 303. Various techniques are known for converting a sequence of words, such as an input utterance, into a feature vector, and one or more of such techniques may be used. For example, the training system 350 may, but does not necessarily, encode the input utterance as a corresponding input feature vector using one-hot encoding or some other encoding. However, one embodiment of the Masterbot 114 uses the same technique used to generate the training feature vector 620 from the training utterance in training the classifier model 324.

[0105] In decision block 715, the masterbot 114 may have the classifier model 324 compare the input feature vector generated in block 710 with the set representation determined in block 515 of the method 500 for initializing the classifier model 324, so that the masterbot 114 can determine whether the input feature vector matches any of the skillbots 116 available to the masterbot 114. The particular technique for comparison and matching may depend on the nature of the set representation. For example, as described below, if the set representation is a cluster of training feature vectors 620, the classifier model 324 may compare the input feature vector (i.e., a point representing the input feature vector) with the cluster to determine whether the input feature vector is within any of the clusters and therefore matches at least one skillbot 116, or if the set representation is a composite vector of the training feature vectors, the classifier model 324 may compare the input feature vector with the composite feature vector to determine whether the input feature vector is sufficiently similar to any such composite feature vector and therefore matches at least one skillbot 116. Various implementations are possible and are within the scope of this disclosure.

[0106] If, in decision block 715, the input feature vector is deemed to match at least one skillbot 116, in block 720, the masterbot 114 may route the input utterance to the skillbot 116 with which the input feature vector is deemed to match. However, if, in decision block 715, the input feature vector is deemed not to match any skillbot 116, in block 725, the masterbot 114 may indicate that the utterance cannot be processed by any skillbot 116. This indication may be passed to the digital assistant, which may provide an output to the user indicating that the user input 110 cannot be processed or acted upon by the digital assistant.

[0107] FIG. 8 illustrates another example of a method 800 for determining whether an input utterance 303 provided as a user input 110 is unrelated to any available skillbot 116 associated with the masterbot 114 using a classifier model 324 of the masterbot 114 according to some embodiments described herein. The method 800 or a similar method may be performed for each received input utterance 303 after initialization of the classifier model 324. For example, the method 800 or a similar method may be performed in block 215 of a method 200 for configuring and using the masterbot 114 to route input utterances 303. Similar to the method 700 of FIG. 7, the method 800 of FIG. 8 is a schematic method 800 for which more detailed examples of some method blocks are shown and described with reference to FIG. 16 and FIG. 21. 7, however, this method 800 illustrates preliminary filtering operations at decision block 810 and block 815 that can be used to ensure that certain input utterances 303 that are similar to the training utterances 615 are not classified as belonging to the non-class 316. In other words, this method 800 includes a filter that filters out any input utterances 303 that are deemed sufficiently similar to the training utterances 615 from non-class considerations, thereby ensuring that such input utterances 303 are routed to the skill bot 116.

[0108] The method 800 shown in FIG. 8 and other methods described herein may be implemented in software (e.g., as code, instructions, or programs) executed by one or more processing units (e.g., processors or processor cores), in hardware, or a combination thereof. The software may be stored in a non-transitory storage medium, such as a memory device. The method 800 is intended to be exemplary and non-limiting. FIG. 8 depicts various operations occurring in a particular sequence or order, but this is not intended to be limiting. In certain embodiments, for example, these operations may be performed in a different order, or one or more operations of the method 800 may be performed in parallel. In certain embodiments, the method 800 may be performed by the masterbot 114.

[0109] At block 805 of the method 800, the masterbot 114 accesses the input utterance 303 provided as the user input 110. For example, in some embodiments, a user may provide the user input 110 in the form of voice input, and the digital assistant 106 may convert the user input 110 into a text input utterance 303 for use by the masterbot 114.

[0110] In decision block 810, the masterbot 114 may have the classifier model 324 determine whether all the words, or a predetermined percentage of the words, of the input utterance 303 accessed in block 805 are found in the training utterance 615. For example, in some embodiments, the classifier model 324 utilizes a Bloom filter based on the words of the training utterance 615 and applies the Bloom filter to the input utterance 303.

[0111] If the input utterance 303 contains only words found in the training utterances 615 at decision block 810, the masterbot 114 may determine that the input utterance is relevant to at least one skillbot 116 and therefore does not belong to the non-class 316. In that case, at block 815, the masterbot 114 may route the input utterance 303 to the skillbot 116 that most closely matches the input utterance 303. For example, as described above, the classifier model 324 may be configured to assign a confidence score (e.g., using a logistic regression model) to each skillbot 116 for the input utterance. The masterbot 114 may select the skillbot 116 with the highest confidence score and route the input utterance 303 to that skillbot 116.

[0112] However, if at decision block 810 the input utterance 303 contains any words not in the training utterance 615 or contains a percentage of words greater than a threshold, the method 800 proceeds to block 820. At block 820, the Masterbot 114 may generate an input feature vector from the input utterance 303 accessed at block 805. The input feature vector may describe and represent the input utterance 303. Various techniques are known for converting a sequence of words, such as an input utterance, into a feature vector, and one or more of such techniques may be used. For example, the training system 350 may, but need not, encode each training utterance 615 as a corresponding training feature vector 620 using one-hot encoding or some other encoding. However, one embodiment of the Masterbot 114 uses the same technique used to generate the training feature vector 620 from the training utterance.

[0113] In decision block 825, the masterbot 114 may have the classifier model 324 compare the input feature vector generated in block 820 with the set representation determined in block 515 of the method 500 for initializing the classifier model 324, so that the masterbot 114 can determine whether the input feature vector matches any of the skillbots 116 available to the masterbot 114. The particular technique for comparison and matching may depend on the nature of the set representation. For example, as described below, if the set representation is a cluster of training feature vectors 620, the classifier model 324 may compare the input feature vector (i.e., a point representing the input feature vector) with the cluster to determine whether the input feature vector is within any of the clusters and therefore matches at least one skillbot 116, or if the set representation is a composite vector of the training feature vectors, the classifier model 324 may compare the input feature vector with the composite feature vector to determine whether the input feature vector is sufficiently similar to any such composite feature vector and therefore matches at least one skillbot 116. Various implementations are possible and are within the scope of this disclosure.

[0114] If, in decision block 825, the input feature vector is deemed to match at least one skillbot 116, in block 830, the masterbot 114 may route the input utterance to the skillbot 116 with which the input feature vector is deemed to match. However, if, in decision block 825, the input feature vector is deemed to not match any skillbot 116, in block 835, the masterbot 114 may indicate that the utterance cannot be processed by any skillbot 116. This indication may be passed to the digital assistant, which may provide an output to the user indicating that the user input 110 cannot be processed or acted upon by the digital assistant.

[0115] Examples of the types of clusters that can be used by classifier models As mentioned above, an exemplary type of set representation that can be used by the classifier model 324 is a cluster of training feature vectors 620. In general, input utterances that can be addressed by available skill bots 116 and thus pertain to available skill bots 116 can be assumed to have some semantic similarity to the training utterances 615 for those skill bots 116. Thus, input utterances 303 pertaining to available skill bots 116 can likely be represented as input feature vectors that are proximate to one or more training utterances 615 in the feature space 630.

[0116] Considering the proximity of feature vectors of semantically similar utterances, a boundary can be defined to separate plotted feature vectors to points with a common intent or to separate feature vectors with different intents. In two-dimensional space, the boundary can be a line or a circle, such that points on one side of the line etc. belong to a first intent class (i.e., correspond to utterances with a first intent) and points on the other side of the line belong to a second intent class. In three dimensions, the boundary can be represented as a plane or a sphere. More generally, in different dimensions, the boundary can be a hyperplane, a hypersphere, or a hypervolume. The boundary can have different shapes and does not have to be perfectly spherical or symmetrical.

[0117] 9 illustrates an example of a feature space 630 including points representing feature vectors of example utterances according to some embodiments described herein. In this example, some of the example utterances belong to a balance class and represent a first intent related to a request for account balance information, and the rest of the example utterances belong to a transaction class and represent a second intent related to a request for information about a transaction. In FIG. 9, the example utterances in the balance class are labeled with b, and the example utterances in the transaction class are labeled with t. These intent classes may be defined, for example, for a finance-related skill bot 116.

[0118] For example, the example utterances in the table below may belong to the balance class in the first column and the transaction class in the second column.

[0119] [Table 2]

[0120] FIG. 10 illustrates an example of the feature space 630 of FIG. 9 with a class boundary 1010 between the intent classes of the feature vectors of the example utterances, according to some embodiments described herein. Specifically, as shown in FIG. 10, a line may be drawn as a class boundary 1010 to separate points in the balance class (i.e., feature vectors of the example utterances) from points in the transaction class. This class boundary 1010 is a rough approximation of the division between the two intent classes. To create a more precise boundary 1010 between the intent classes, it may be necessary to create a circle or other geometric volume that defines the clusters of feature vectors, such that each cluster only contains feature vectors in a single corresponding intent class (i.e., has an intent associated with that intent class).

[0121] FIG. 11 illustrates an example of the feature space 630 of FIG. 9 with class boundaries 1010 that separate, specifically, isolate, feature vectors associated with a common intent into respective clusters, according to some embodiments described herein. In this example, as with some embodiments, not all feature vectors of an intent class are in a single cluster, and no cluster includes feature vectors from more than one intent class. Specifically, in the illustrated example, a first cluster 1110a defined by a first class boundary 1010 includes only feature vectors in a balance class, and a second cluster 1110b defined by a second class boundary 1010 includes only feature vectors in a transaction class. As described in more detail below, some embodiments described herein can form class boundaries 1010 to create clusters such as those shown in FIG. 11.

[0122] FIG. 12 illustrates another example of the feature space 630 of FIG. 9 with class boundaries 1010 that separate, specifically, isolate, feature vectors associated with a common intent into respective clusters, according to some embodiments described herein. In this example, as in some embodiments, not all feature vectors of an intent class are in a single cluster, and no cluster includes feature vectors from more than one intent class. Specifically, in the illustrated example, a first cluster 1110c defined by a first class boundary 1010 includes only feature vectors in a balance class, and a second cluster 1110d defined by a second class boundary 1010 includes only feature vectors in a transaction class. However, in contrast to the example of FIG. 11, the class boundaries 1010, and therefore the clusters, overlap. Some embodiments described herein support overlapping clusters as shown in FIG. 12. As described in more detail below, some embodiments described herein can form class boundaries 1010 to create clusters as shown in FIG. 12.

[0123] FIG. 13 illustrates another example of the feature space 630 of FIG. 9 with class boundaries 1010 separating feature vectors into clusters, according to some embodiments described herein. In this example, as in some embodiments, not all feature vectors of an intent class are in a single cluster, and further, a cluster may represent various intents by including feature vectors with different intents. Specifically, in the illustrated example, a first cluster 1110e defined by a first class boundary 1010 includes only feature vectors in the balance class, and a second cluster 1110f defined by a second class boundary 1010 includes some feature vectors in the balance class (i.e., having balance-related intent) and some feature vectors in the transaction class (i.e., having transaction-related intent). Some embodiments described herein support clusters with various intents, as shown in FIG. 13. As described in more detail below, some embodiments described herein may form class boundaries 1010 to create clusters as shown in FIG. 13.

[0124] Clustering to identify unrelated input utterances FIG. 14 illustrates a method 1400 for initializing a classifier model 324 of a masterbot 114 to determine whether an input utterance is irrelevant or relevant to an available skillbot 116 by utilizing clusters of training feature vectors, according to some embodiments described herein. This method 1400 or a similar method can be used in block 205 of the method 200 described above for configuring and using a masterbot 114 to direct input utterances 303 to a skillbot 116, and the method 1400 of FIG. 14 is a more specific variation of the method 500 of FIG. 5. In some embodiments of this method, k-means clustering is performed to configure the classifier model 324, as described below. Although k-means clustering may allow for more accurate formation of clusters compared to other clustering techniques such as k-nearest neighbors, various clustering techniques may be used instead of or in addition to k-means clustering.

[0125] The method 1400 shown in FIG. 14 and other methods described herein may be implemented in software (e.g., as code, instructions, or programs) executed by one or more processing units (e.g., processors or processor cores), in hardware, or a combination thereof. The software may be stored in a non-transitory storage medium, such as a memory device. The method 1400 is intended to be exemplary and non-limiting. FIG. 14 depicts various operations occurring in a particular sequence or order, but this is not intended to be limiting. In certain embodiments, for example, these operations may be performed in a different order, or one or more operations of the method 1400 may be performed in parallel. In certain embodiments, the method 1400 may be performed by a training system 350, which may be part of the DABP 102.

[0126] At block 1405 of the method 1400, the training system 350 accesses training utterances 615, also referred to as example utterances, for various skillbots 116 associated with the masterbot 114. For example, the training utterances 615 may be stored as skillbot data in training data 354 accessible to the training system 350. In some embodiments, each skillbot 116 available to the masterbot 114 may be associated with a subset of training utterances 615, and each such subset of training utterances 615 may include a training utterance 615 for each intent for which the skillbot 116 is configured. Thus, the complete set of training utterances 615 may include a training utterance 615 for each intent of each skillbot 116, such that every intent of every skillbot 116 is represented.

[0127] FIG. 15 illustrates an example of an implementation of aspects of the method 1400 of FIG. 14 according to some embodiments described herein. As illustrated in FIG. 15, the training system 350 may access training data 354. The training data 354 may include skillbot data related to the skillbot 116 available to the masterbot 114 to which the classifier model 324 being trained belongs. In this example, the training data 354 accessed by the training system 350 may include training utterances 615 from the first skillbot data 358d, the second skillbot data 358e, and the third skillbot data 358f, each of which may include training utterances 615 representing a respective skillbot 116 available to the masterbot 114. Furthermore, for a given set of skillbot data, as illustrated with respect to the first skillbot data 358d in FIG. 15, each training utterance 615 may be associated with an intent that the associated skillbot 116 is configured to evaluate and act upon.

[0128] In block 1410 of FIG. 14, the training system 350 generates a training feature vector 620 from the training utterances 615 accessed in block 1405. As described above, the training utterances 615 may include a subset associated with each skillbot 116 and may further include a training utterance 615 for each intent of each skillbot 116. Thus, in some embodiments, the training feature vector 620 may include a respective subset for each skillbot 116 and may further include a respective feature vector for each intent of each skillbot. As described above, the training system 350 may generate the training feature vector 620 to describe and represent each training utterance 615. Various techniques are known for converting a sequence of words, such as the training utterances 615, into a feature vector, such as the training feature vector 620, and one or more of such techniques may be used. For example, the training system 350 may, but need not, encode each training utterance 615 as a corresponding training feature vector 620 using one-hot encoding or some other encoding.

[0129] 15, for example, each training utterance 615 from the various skillbot data 358 may be converted into a respective training feature vector 620. Thus, in some embodiments, the resulting training feature vector 620 may represent all skillbots 116 for which the training utterances 615 were provided, and all intents for which the training utterances 615 were provided.

[0130] 14, the training system 350 may set (i.e., initialize) a count, which is the amount of clusters to be generated. In some embodiments, for example, the count may be initially set to an amount n, where n is the total number of intents across the various skill bots 116 available to the master bot 114. However, various other amounts may be used as the initial value of the count.

[0131] In some embodiments, the training system 350 performs the step of iteratively determining a set of clusters of the training feature vectors 620 one, two, or more times. Although the method 1400 utilizes two rounds of cluster determination, in some embodiments, one or more rounds may be used. At block 1420, the training system 350 begins a first round of cluster determination. The first round utilizes an iterative loop in which the training system 350 generates clusters of the training feature vectors 620 and then determines whether such clusters are sufficient. If the clusters are deemed sufficient, as described below, the training system 350 may terminate the first iteration.

[0132] In some embodiments, block 1425 is the beginning of a first iteration loop for determining clusters. Specifically, in block 1425, the training system 3509 may determine a centroid location for each of the various clusters to be generated in this iteration (i.e., a location for each centroid). The quantity of centroid locations is equal to the count determined in block 1415. In some embodiments, in the first iteration of this loop, the training system 350 may select a set of randomly selected centroid locations in the feature space 630 in which the training utterance 615 falls that have a quantity equal to the count. More specifically, for example, the training system 350 may determine a bounding box, such as a minimum bounding box, for points corresponding to the training feature vector 620. The training system 350 may then randomly select a set of centroid locations within that bounding box. In this case, the number of selected locations is equal to the count determined for the cluster. During iterations other than the first, the count has increased from the previous iteration, and thus, in some embodiments, only the centroid locations of newly added centroids are randomly determined. The centroids carried over from the previous iteration may retain their positions. The respective centroids for the corresponding clusters may be located for each centroid position.

[0133] In block 1430, the training system 350 may determine clusters by assigning each training feature vector 620 to its nearest respective centroid among the various centroids whose locations were determined in block 1425. For example, for each training feature vector 620, the training system 350 may calculate the distance to each centroid location and assign the training feature vector 620 to the centroid having the smallest such distance. The set of training feature vectors 620 assigned to a common centroid may together form a cluster associated with that centroid. Thus, there may be a quantity of clusters equal to the quantity of centroids, which is the value of the count.

[0134] In the example of Figure 15, only a portion of the feature space 630 is shown in which a first centroid 1510a and a second centroid 1510b are visible. The training system 350 has determined that of the three training feature vectors 620 shown, two are closest to the first centroid 1510a and one is closest to the second centroid 1510b. In this example, the two training feature vectors 620 closest to the first centroid 1510a constitute a first cluster 1110g, and the one training feature vector 620 closest to the second centroid 1510b constitutes a second cluster 1110h.

[0135] 14, block 1435, for the clusters determined in block 1430, the training system 350 recalculates the location of the centroid of each cluster. For example, in some embodiments, the centroid of each cluster is calculated to be the average (e.g., the arithmetic mean) of the training feature vectors 620 assigned to that centroid and thus assigned to that cluster.

[0136] At decision block 1440, the training system 350 may determine whether a stopping condition is met. In some embodiments, the training system 350 may utilize this method 1400 or a similar method to iteratively increase the number of centroids (i.e., increase the count) and assign training feature vectors 620 to their closest associated centroids until convergence occurs such that no significant improvement in the clustering is likely to occur. In some embodiments, the stopping condition defines a level of sufficient convergence.

[0137] In some embodiments, the stopping condition may be met if one or both of the following conditions are true: (1) the average cluster cost meets a first threshold, such as the average cluster cost is less than 1.5 or some other predetermined value; or (2) the outlier ratio meets a second threshold, such as the outlier ratio is less than or equal to 0.25 or some other predetermined value. The cluster cost for a particular cluster may be defined as (a) the sum of the squared distances between the centroid of the cluster recalculated in block 1435 and each training feature vector 620 assigned to that centroid, divided by the amount of training feature vectors 620 assigned to that centroid. Thus, the average cluster cost may be the average of the different cluster costs of the different centroids. The outlier ratio may be defined as the total number of outliers among clusters divided by the number of clusters (i.e., count). There are various techniques for defining outliers, and one or more of such techniques may be used by the training system 350. In some embodiments, the stopping condition is met if, and only if, (1) the average cluster cost meets a first threshold (e.g., less than 1.5) and (2) the outlier ratio meets a second threshold (e.g., less than or equal to 0.25).

[0138] In general, the average cluster cost tends to decrease as the value of k (i.e., count) increases, while the outlier ratio tends to increase as k increases. In some embodiments, if the stopping conditions described above that take both factors into account are applied, the maximum k value that meets the respective thresholds for both cluster count and outlier ratio becomes the final k value for the classifier model 324.

[0139] If the stopping condition is not met at decision block 1440, method 1400 may proceed to block 1445. At block 1445, the training system increments the count for the next round of clustering. In some embodiments, the count may be incrementally increased by an amount that increases the likelihood that the stopping condition will be met after no more than a certain number of iterations of the loop. For example, in some embodiments, the count may be increased by:

[0140]

number

[0141] The step size may be increased by a value of step size equal to n, where n is the total number of intents across all available skill bots 116, and u is the total number of training utterances 615 across all available skill bots 116. This step size will ensure that the stopping condition is eventually met. Specifically, for this step size, starting with a count equal to n, the twentieth iteration will have a count that is equal to or greater than u, the amount of utterances. If the count is equal to or greater than u, each utterance may have its own cluster, or if not, there is still a chance that the average cluster cost will be less than 1.5 and the outlier ratio will be equal to or less than 0.25, which satisfies the exemplary stopping condition. More generally, the step size may be selected to ensure that the iterations do not waste computing resources by looping for an unreasonable amount of time. After the counts are updated in block 1445, the method 1400 returns to block 1425 to perform another loop iteration.

[0142] However, if a stopping condition is met at decision block 1440, the method 1400 may end the current iterative loop and proceed to block 1450. At block 1450, the training system 350 begins a second round of clustering, where clusters may be further defined based on the work performed in the first round.

[0143] In some embodiments, block 1455 is the beginning of a second iteration loop to determine clusters. Specifically, in block 1455, the training system 3509 may determine the respective centroid locations (i.e., the respective locations for each centroid) for the various clusters to be generated in this iteration. The quantity of centroid locations is equal to the current value of the count. In some embodiments, in the first iteration of this loop, the training system 350 uses the centroid locations as recalculated in block 1435 before the end of the first iteration. In iterations other than the first, the count is incremented from the previous iteration. In that case, the centroids from the previous iteration may retain their centroid locations, and in some embodiments, the centroid locations of the newly added centroids due to the increment in the count may be determined randomly. The respective centroids for the corresponding clusters may be located for each centroid location.

[0144] In block 1460, the training system 350 may determine clusters by assigning each training feature vector 620 to its nearest respective centroid among the various centroids whose locations were determined in block 1455. For example, for each training feature vector 620, the training system 350 may calculate the distance to each centroid location and assign the training feature vector 620 to the centroid having the smallest such distance. The set of training feature vectors 620 assigned to a common centroid may together form a cluster associated with that centroid. Thus, there may be a quantity of clusters equal to the quantity of centroids, which is the value of the count.

[0145] At block 1465, for the clusters determined at block 1460, the training system 350 recalculates the location of the centroid of each cluster. For example, in some embodiments, the centroid of each cluster is calculated to be the average (e.g., the arithmetic mean) of the training feature vectors 620 assigned to that centroid and thus assigned to that cluster.

[0146] At decision block 1470, the training system 350 may determine whether a stopping condition is met. In some embodiments, the training system 350 may utilize this method 1400 or a similar method to iteratively increase the number of centroids (i.e., increase the count) and assign training feature vectors 620 to their closest associated centroids until convergence occurs such that no significant improvement in the clustering is likely to occur. In some embodiments, the stopping condition defines a level of sufficient convergence.

[0147] In some embodiments, the stopping condition may be met if one or both of the following conditions are true: (1) the average cluster cost meets a first threshold, such as the average cluster cost is less than 1.5 or some other predetermined value; or (2) the outlier ratio meets a second threshold, such as the outlier ratio is less than or equal to 0.25 or some other predetermined value. The cluster cost for a particular cluster may be defined as (a) the sum of the squared distances between the centroid of the cluster recalculated in block 1465 and each training feature vector 620 assigned to that centroid, divided by the amount of training feature vectors 620 assigned to that centroid. Thus, the average cluster cost may be the average of the different cluster costs of the different centroids. The outlier ratio may be defined as the total number of outliers among clusters divided by the number of clusters (i.e., count). There are various techniques for defining outliers, and one or more of such techniques may be used by the training system 350. In some embodiments, the stopping condition is met if, and only if, (1) the average cluster cost meets a first threshold (e.g., less than 1.5) and (2) the outlier ratio meets a second threshold (e.g., less than or equal to 0.25).

[0148] If the stopping condition is not met at decision block 1470, method 1400 may proceed to block 1475. At block 1475, the training system increments the count for the next round of clustering. In some embodiments, the count may be incrementally increased by an amount that increases the likelihood that the stopping condition will be met after no more than a certain number of iterations of the loop. For example, in some embodiments, the count may be increased by:

[0149]

number

[0150] The count may be increased by a step size value equal to . This step size ensures that the stopping condition will eventually be met. Specifically, with this step size, the stopping condition is likely to be met by the end of the fifth iteration. More generally, the step size may be selected to ensure that the iterations do not waste computing resources by looping for an unreasonable amount of time. After the count is updated in block 1475, method 1400 returns to block 1455 to perform another loop iteration.

[0151] However, if a stopping condition is met at decision block 1470, the method 1400 may terminate the current iterative loop and proceed to block 1480. At block 1480, the training system 350 determines a respective boundary 1010 for each cluster determined above. In some embodiments, the boundary 1010 for a cluster is defined to be centered on the cluster's centroid and to include all training feature vectors 620 assigned to the cluster. In some embodiments, for example, the cluster boundary 1010 is a hypersphere (e.g., a circle or sphere) centered at the centroid. In some embodiments, the radius of the boundary 1010 may be a margin value (i.e., padding amount) plus the greater of (1) the maximum distance from the center to the training feature vector 620 in the cluster furthest from the centroid, or (2) the average of each distance from the training feature vector 620 in the cluster to the centroid, plus three times the standard deviation of such distance. That is, the radius may be set as radius = margin + max(max(distances), mean(distances) + 3σ(distances)), where distance is the set of distances from the training feature vectors of the clusters to the cluster centroids, max(distances) is the maximum of the set, mean(distances) is the mean of the set, and σ(distances) is the standard deviation of the set. Furthermore, the margin value may be a margin of error and may have a value greater than or equal to zero.

[0152] In some embodiments, the margins may be used to define a boundary 1010 that encompasses a broader scope than would otherwise be the case, so as to reduce the likelihood that the associated input utterance 303 falls outside of all clusters and is therefore labeled as a member of a non-class 316. In other words, the margins may pad the boundary 1010. For example, the margins may be

[0153]

number

[0154] where u is the total number of training utterances 615 used. The margin can take into account the following situations: potentially the amount of training utterances is too low (e.g., 20-30) so that the training feature vectors 620 do not cover a significant portion of the feature space 630, and thus the clusters are too small to capture the relevant input utterances 303.

[0155] Returning to the example of FIG. 15, the training system 350 determines a respective boundary for each cluster determined based on the assignment of training feature vectors 620. Specifically, in this example, a first boundary 1010a is determined for a first cluster 1110g that includes two training feature vectors 620, and a second boundary 1010b is determined for a second cluster 1110h that includes one training feature vector 620. The amount of training feature vectors may be small, as in this example, or large, such as the number of training feature vectors 620 per cluster. It will be appreciated that this simplified example is non-limiting and is provided for illustrative purposes only.

[0156] 14, in block 1485, the training system 350 may configure the classifier model 324 of the masterbot 114 to utilize the boundaries 1010, also referred to as cluster boundaries, determined in block 1480. For example, the training system 350 may store an indication of the cluster boundaries in a storage device accessible to the classifier model (e.g., on the digital assistant 106). The classifier model 324 may be configured to compare input utterances to such cluster boundaries 1010, which serve as a set representation of the training feature vectors 620.

[0157] Various modifications may be made to the method 1400 described above and are within the scope of the present disclosure. For example, some examples of the training system 350 perform only one round of cluster refinement. In that case, the operations from block 1445 to block 1475 may be skipped such that the method 1400 proceeds to block 1480 when a stopping condition is met at decision block 1440. Some other examples of the training system 350 perform more than two rounds of cluster refinement. In that case, the operations from block 1445 to block 1475 may be repeated for each additional round after the second run. These and other implementations are within the scope of the present disclosure.

[0158] In some embodiments, the k value (i.e., the value of the count and thus the number of clusters) determined in the above method 1400 depends on the total number of training utterances 615 and the distribution of the training feature vectors 620 across the feature space 630. Thus, the k value may vary for each masterbot 114 depending on the available skillbots 116 and the training utterances 615 available to express the intents of those skillbots 116. The optimal k value is a balanced value such that each cluster is large enough that the input utterances 303 related to the cluster fall within the boundary 1010 of the cluster while the irrelevant utterances fall outside the boundary 1010. If the clusters are too large, the risk of false matches increases. If the clusters are too small (e.g., consisting of a single training utterance 615), the usefulness of such clusters is limited because the classifier model 324 may be overfitted.

[0159] In the above example of the method 1400, the training feature vectors 620 are not divided or grouped based on intent, so a cluster may include training feature vectors 620 representing different skillbots 116 or different intents of one or more skillbots 116. Additionally or alternatively, an embodiment of the training system 350 may ensure that each cluster includes only training feature vectors 620 representing a single skillbot 116, a single intent, or a single subbot. In this case, a subbot is associated with a subset of training utterances that represent a single skillbot 116. Various techniques may be used to limit the clusters in this way. For example, the training utterances may be separated into groups based on intent, subbot, or skillbot 116, and each instance of the above method 1400 may be performed for each group. In this manner, a cluster determined during one instance of the method 1400 for a corresponding group (e.g., training utterances 615 representing a particular skillbot 116) may include only training feature vectors 620 from that corresponding group, which may be limited to training utterances of a single intent, subbot, or skillbot 116. Various other implementations are possible and are within the scope of this disclosure.

[0160] FIG. 16 illustrates a method 1600 for using a classifier model 324 of a masterbot 114 to determine whether an input utterance 303 provided as a user input 110 is unrelated to any available skillbot 116 associated with the masterbot 114, according to some embodiments described herein. This method 1600 or a similar method can be used in block 215 of the method 200 described above for configuring and using a masterbot 114 to direct the input utterance 303 to a skillbot 116. The method 1600 of FIG. 16 is a more specific variation of the method 700 of FIG. 7, and like the method 700 of FIG. 7, the method 1600 of FIG. 16 can be used with the preliminary filtering operation described with respect to the method 800 of FIG. 8. More specifically, as described below, the classifier model 324 of the masterbot 114 can utilize clusters of training feature vectors 620 to determine whether the input utterance 303 belongs to a non-class 316 (i.e., is unrelated to an available skillbot 116).

[0161] The method 1600 shown in FIG. 16 and other methods described herein may be implemented in software (e.g., as code, instructions, or programs) executed by one or more processing units (e.g., processors or processor cores), in hardware, or a combination thereof. The software may be stored in a non-transitory storage medium, such as a memory device. The method 1600 is intended to be exemplary and non-limiting. FIG. 16 depicts various operations occurring in a particular sequence or order, but this is not intended to be limiting. In certain embodiments, for example, these operations may be performed in a different order, or one or more operations of the method 1600 may be performed in parallel. In certain embodiments, the method 1600 may be performed by a masterbot 114 associated with a set of available skillbots 116.

[0162] At block 1605 of the method 1600, the masterbot 114 accesses the input utterance 303 provided as the user input 110. For example, in some embodiments, a user may provide the user input 110 in the form of voice input, and the digital assistant 106 may convert the user input 110 into a text input utterance 303 for use by the masterbot 114.

[0163] In block 1610, the Masterbot 114 may generate an input feature vector from the input utterance 303 accessed in block 1605. More specifically, in some embodiments, the Masterbot 114 causes the classifier model 324 of the Masterbot 114 to generate an input feature vector from the input utterance 303. The input feature vector may describe and represent the input utterance 303. Various techniques are known for converting a sequence of words, such as an input utterance, into a feature vector, and one or more of such techniques may be used. For example, the training system 350 may, but does not necessarily, encode the input utterance 303 as a corresponding input feature vector using one-hot coding or some other coding. However, one embodiment of the Masterbot 114 uses the same technique used to generate the training feature vector 620 from the training utterance 615 in training the classifier model 324.

[0164] In decision block 1615, the masterbot 114 compares the input feature vector generated in block 1610 to the clusters. More specifically, in some embodiments, the masterbot 114 has the classifier model 324 compare the input feature vector generated in block 1610 to the clusters determined during training, specifically the boundaries of the clusters. As described above, each cluster may include a set of training feature vectors 620 and may include a boundary 1010 based on the training feature vectors 620 of that cluster. For example, the boundary 1010 includes all the training feature vectors 620 assigned to the cluster and possibly some additional space outside the training feature vectors 620. Specifically, in some embodiments, the classifier model 324 determines whether the input feature vector (i.e., the point corresponding to the input feature vector) is within any boundary 1010 of any cluster of the training feature vectors 620. Various techniques exist in the art for determining whether a point is within a boundary, and one or more such techniques may be used to determine whether an input feature vector falls within any of the cluster boundaries 1010.

[0165] At decision block 1620, the classifier model 324 makes a decision based on comparing the input feature vector to the cluster boundaries. If the input feature vector is not within any cluster boundary, and is therefore outside all of the cluster boundaries 1010, the method 1600 proceeds to block 1625.

[0166] FIG. 17 illustrates an example of performing the method 1600 when the input feature vector 1710 is outside all cluster boundaries, according to some embodiments described herein. In some embodiments, the masterbot 114 provides the input utterance 303 to the classifier model 324, which causes the classifier model 324 to convert the input utterance 303 into an input feature vector 1710 and compare the input feature vector with the cluster boundaries 1010. In the example of FIG. 17, five clusters 1110 are shown in the feature space 630. However, a greater or lesser number of clusters 1110 may be used. In this example, the input feature vector 1710 is outside all cluster boundaries 1010, and therefore the classifier model 324 outputs an indication to the masterbot 114 that the input utterance 303 belongs to the non-class 316.

[0167] 16, in block 1625, the masterbot 114 indicates that it cannot process the input utterance 303 (i.e., cannot process it further) based on the classifier model 324 indicating that the input feature vector is outside of all cluster boundaries 1010. For example, the digital assistant 106 may respond to the user to request clarification or to report that the user input 110 is not relevant to the digital assistant's skills.

[0168] However, if the input feature vector is within one or more cluster boundaries 1010, the method 1600 skips to block 1630. In block 1630, the masterbot 114 determines a skillbot 116 to address the input utterance 303 (i.e., to further process the input utterance 303 and determine a response 435 to the input utterance 303) by selecting one of the skillbots 116 from among the available skillbots. The determination of the skillbot 116 can be performed in a variety of ways, and the technique used may depend on the configuration of one or more cluster boundaries 1010 into which the input feature vector falls.

[0169] To select a skillbot 116 for an input utterance 303, the masterbot 114 may take into account various training utterances represented by training feature vectors 620 that share clusters with the input feature vector 1710 (i.e., training feature vectors that are members of clusters 1110 within the boundary 1010 in which the input feature vector 1710 falls). For example, if a training feature vector 620 falls within two or more overlapping clusters 1110, training utterances 615 that have corresponding training feature vectors 1710 in any of those two or more clusters 1110 may be considered. Similarly, if a training feature vector 620 falls only within a single cluster 1110, training utterances 615 that have corresponding training feature vectors 620 in that cluster 1110 are considered. If all of the training utterances 615 under consideration represent a single skillbot 116, in some cases, for example if the input feature vector 1710 falls within a cluster 1110 that contains training feature vectors 620 associated with only a single skillbot 116, the masterbot 114 may select that skillbot 116 to address the input utterance 303.

[0170] In some embodiments, the classifier model 324 may be able to identify a particular intent of a particular skillbot 116 to address the input utterance. For example, if one or more clusters 1110 into which the training feature vector 620 falls only include a training feature vector 620 of a training utterance 615 that represents a single intent of a single skillbot 116, the classifier model 324 may identify that the particular intent is applicable to the input utterance 303. If the classifier model 324 can identify the particular intent of a particular skillbot 116, the masterbot 114 may route the input utterance 303 to that skillbot 116 and may indicate the intent to the skillbot 116. As a result, the skillbot 116 may be able to dispense with performing its own classification of the input utterance 303 to infer the intent, but rather may infer the intent indicated by the masterbot 114.

[0171] However, if the training utterance 615 being considered represents multiple skillbots 116, in some cases, such as when the input feature vector 1710 falls within a cluster 1110 composed of training feature vectors 620 representing multiple skillbots 116, or when the input feature vector 1710 falls within multiple overlapping clusters 1110, the masterbot 114 may need to further classify the input utterance 303 to select a skillbot 116. Various techniques may be used to further classify the input utterance 303. In some embodiments, the classifier model 324 implements a machine learning mode to calculate a confidence score for the input utterance 303 (e.g., using a logistic regression model) for each skillbot 116 that has an associated training feature vector 620 within one or more clusters 1110 in which the input feature vector 1710 falls. For example, the machine learning model may be trained using the composite feature vector. The masterbot 114 may then select the skillbot 116 with the highest confidence score to address the input utterance 303. In contrast to conventional uses of confidence scores to identify relevant skill bots 116, in some embodiments, it has already been determined that the input utterance 303 is relevant to the skill bot 116. Thus, the risk of routing the input utterance 303 to an irrelevant skill bot 116 is reduced or eliminated.

[0172] In additional or alternative embodiments, the classifier model 324 may utilize a k-nearest neighbor technique to select a skillbot 116 from among two or more skillbots 116 whose associated training feature vectors 620 share one or more clusters 1110 with the input feature vector 1710. For example, the classifier model 324 may select a value of k and identify the k-nearest training feature vectors 620 to the input feature vector 1710 from among the training feature vectors 620 that fall within one or more clusters 1110 of the input feature vector 1710. The classifier model 324 may identify the skillbot 116 that has the greatest number of associated training feature vectors 620 in the set of k-nearest training feature vectors 620. The masterbot 114 may select the skillbot 116 to address the input utterance. Various other implementations for selecting a skillbot 116 are possible and are within the scope of this disclosure.

[0173] In block 1635, the masterbot 114 may forward the input utterance 303 to the skillbot selected in block 1630. The skillbot 116 may then process the input utterance 303 to respond to the user input 110.

[0174] Using composite vectors to identify irrelevant input utterances As noted above, generally, some embodiments described herein utilize a set representation of the training feature vectors 620 to determine whether an input utterance 303 is relevant to a set of available skill bots 116. Also, as noted above, the set representation may be a cluster 1110. However, additionally or alternatively, the set representation may be a higher level feature vector, referred to herein as a composite feature vector.

[0175] 18 illustrates a method 1800 for initializing a classifier model 324 of a masterbot 114 to determine whether an input utterance 303 is unrelated or related to an available skillbot 116 utilizing a synthetic feature vector according to some embodiments described herein. The method 1800 or a similar method can be used in block 205 of the method 200 described above for configuring and using a masterbot 114 to direct an input utterance 303 to a skillbot 116, and further, the method 1800 of FIG. 18 is a more specific variation of the method 500 of FIG. 5.

[0176] The method 1800 shown in FIG. 18 and other methods described herein may be implemented in software (e.g., as code, instructions, or programs) executed by one or more processing units (e.g., processors or processor cores), in hardware, or a combination thereof. The software may be stored in a non-transitory storage medium, such as a memory device. The method 1800 is intended to be exemplary and non-limiting. FIG. 18 depicts various operations occurring in a particular sequence or order, but this is not intended to be limiting. In certain embodiments, for example, these operations may be performed in a different order, or one or more operations of the method 1800 may be performed in parallel. In certain embodiments, the method 1800 may be performed by a training system 350, which may be part of the DABP 102.

[0177] At block 1805 of the method 1800, the training system 350 accesses training utterances 615 (also referred to as example utterances) for various skillbots 116 associated with the masterbot 114. For example, the training utterances 615 may be stored as skillbot data in training data 354 accessible to the training system 350. In some embodiments, each skillbot 116 available to the masterbot 114 may be associated with a subset of training utterances 615, and each such subset of training utterances 615 may include a training utterance 615 for each intent for which the skillbot 116 is configured. Thus, the complete set of training utterances 615 may include a training utterance 615 for each intent of each skillbot 116 such that every intent of every skillbot 116 is represented.

[0178] In block 1810, the training system 350 generates a training feature vector 620 from the training utterances 615 accessed in block 1805. As described above, the training utterances 615 may include a subset associated with each skillbot 116 and may further include a training utterance 615 for each intent of each skillbot 116. Thus, in some embodiments, the training feature vector 620 may include a respective subset for each skillbot 116 and may further include a respective feature vector for each intent of each skillbot. As described above, the training system 350 may generate the training feature vector 620 to describe and represent each training utterance 615. Various techniques are known for converting a sequence of words, such as the training utterances 615, into a feature vector, such as the training feature vector 620, and one or more of such techniques may be used. For example, the training system 350 may, but need not, encode each training utterance 615 as a corresponding training feature vector 620 using one-hot encoding or some other encoding.

[0179] In block 1815, the training system 350 divides the training feature vectors 620 and thus the corresponding training utterances 615 into conversation categories. The conversation categories can be defined, for example, based on intents, subbots, or skillbots 116. In some embodiments, if the conversation categories are based on intents, each conversation category includes a training feature vector 620 that represents a single corresponding intent for which the skillbot 116 is configured. In that case, the number of conversation categories can be equal to the number of intents across the various skillbots 116 available to the masterbot 114. In some embodiments, if the conversation categories are based on skillbots 116, each conversation category includes a training feature vector 620 that represents a single corresponding skillbot 116. In that case, the number of conversation categories can be equal to the number of skillbots 116 available to the masterbot 114. In some embodiments, if the conversation categories are based on subbots, each conversation category includes a subset of training feature vectors 620 that represent a single corresponding skillbot 116. In that case, the number of conversation categories may be equal to or greater than the number of skillbots 116 available to the masterbot 114.

[0180] In block 1820, the training system 350 generates a composite feature vector from the training feature vectors 620, such that each training feature vector 620 in a conversation category is aggregated into a composite feature vector that represents and corresponds to that conversation category. In other words, for each conversation category to which a training feature vector 620 is assigned in block 1815, the training system 350 may combine each training feature vector 620 into a composite feature vector for the conversation category. Various techniques can be used for aggregation. For example, the composite feature vector can be the average (e.g., arithmetic mean) of the training feature vectors 620 in each category. Based on the conversation category, the composite feature vector can be an intent vector, and a subbot vector, or a bot vector, depending on whether the conversation category is defined based on an intent, a subbot, or a skillbot 116.

[0181] Although the composite feature vector may be generated as an arithmetic average of the training feature vectors 620, other mathematical functions, including other types of linear combinations, may additionally or alternatively be used to generate the composite feature vector. For example, in some embodiments, the composite feature vector may be a weighted average of the training feature vectors 620 in a conversation category. The weighting may be based on various factors, such as the priority of a particular keyword in the corresponding training utterances 615, such that a training utterance 615 having a particular keyword is given a greater weight. When generating a sub-bot or bot vector, the training feature vectors 620 may be aggregated as a weighted average, such that a training feature vector 620 corresponding to a particular intent is given a greater weight than other training feature vectors 620. For example, a training feature vector 620 corresponding to an intent with a larger number of representative training utterances 615 may be given a greater weight in the aggregate compared to a training feature vector 620 corresponding to an intent with a smaller number of representative training utterances 615. Various implementations are possible and are within the scope of this disclosure.

[0182] 19 and 20 illustrate the concept of a composite feature vector. Specifically, FIG. 19 illustrates the generation of a composite feature vector using intent-based speech categories according to some embodiments described herein. As a result, the composite feature vector illustrated in FIG. 19 is at the intent level and therefore an intent vector. In the example of FIG. 19, at least two skillbots 116 are available to the masterbot 114. The first skillbot 116 is associated with a first skillbot data 358g that includes training utterances 615 representing a first intent, intent A, and other training utterances 615 representing a second intent, intent B. Thus, the first skillbot 116 is configured to respond to input utterances associated with intent A or intent B. The second skillbot 116 is associated with a second skillbot data 358h that includes training utterances 615 representing a third intent, intent C. Thus, the second skill bot 116 is configured to act on input utterances associated with intent C.

[0183] In the example of FIG. 19, the training system 350 converts all training utterances 615 for the skill bot 116 into respective feature vectors 620. The training system 350 groups the training feature vectors into intent-based conversational categories. Thus, the training feature vectors 620 corresponding to the training utterances 615 for intent A form a first conversational category, the training feature vectors 620 corresponding to the training utterances 615 for intent B form a second conversational category, and the training feature vectors 620 corresponding to the training utterances 615 for intent C form a third conversational category. In this example, as described above, the training feature vectors 620 for a given intent-based conversational category are aggregated (e.g., averaged) into a composite feature vector for the conversational category and, therefore, for the associated intent. Specifically, the training system 350 aggregates the training feature vectors 620 corresponding to the training utterances 615 for intent A into a first composite feature vector 1910a, the training system 350 aggregates the training feature vectors 620 corresponding to the training utterances 615 for intent B into a second composite feature vector 1910b, and the training system 350 aggregates the training feature vectors 620 corresponding to the training utterances 615 for intent C into a third composite feature vector 1910c. Thus, in this example, each composite feature vector represents a respective intent of the skill bot 116.

[0184] FIG. 20 illustrates the generation of a composite feature vector using skillbot-based conversation categories, also referred to as bot-based conversation categories, according to some embodiments described herein. As a result, the composite feature vector illustrated in FIG. 20 is at the skillbot level, and thus a bot vector. In the example of FIG. 20, at least two skillbots 116 are available to the masterbot 114. The first skillbot 116 is associated with a first skillbot data 358g including training utterances 615 representing a first intent, intent A, and other training utterances 615 representing a second intent, intent B. Thus, the first skillbot 116 is configured to respond to input utterances associated with intent A or intent B. The second skillbot 116 is associated with a second skillbot data 358h including training utterances 615 representing a third intent, intent C. Thus, the second skillbot 116 is configured to respond to input utterances associated with intent C.

[0185] In the example of FIG. 20, the training system 350 converts all training utterances 615 for the skillbot 116 into respective feature vectors 620. The training system 350 groups the training feature vectors into skillbot-based conversational categories. Thus, the training feature vectors 620 corresponding to the training utterances 615 for intent A together with the training feature vectors 620 corresponding to the training utterances 615 for intent B form a first conversational category, and the training feature vectors 620 corresponding to the training utterances 615 for intent C form a second conversational category. In this example, as described above, the training feature vectors 620 for a given skillbot-based conversational category are aggregated (e.g., averaged) into a composite feature vector for the conversational category and therefore for the associated skillbot 116. Specifically, the training system 350 aggregates the training feature vectors 620 corresponding to the training utterances 615 for intent A and the training feature vectors 620 corresponding to the training utterances 615 for intent B into a first composite feature vector 1910d, and the training system 350 aggregates the training feature vectors 620 corresponding to the training utterances 615 for intent C into a second composite feature vector 1910e. Thus, in this example, each composite feature vector represents a respective skill bot 116.

[0186] In some embodiments, the training system 350 is not limited to generating only one type of composite feature vector. For example, the training system 350 may generate intent vectors, subbot vectors, and skillbot vectors, or the training system 350 may generate some other combination of these or other types of composite feature vectors.

[0187] 18, in block 1825, the training system 350 may configure the classifier model 324 of the masterbot 114 to utilize the composite feature vector determined in block 1820. For example, the training system 350 may store an indication of the composite feature vector in a storage device (e.g., on the digital assistant 106) accessible by the classifier model 324. The classifier model 324 may be configured to compare input utterances to such composite feature vectors, which serve as a set representation of the training feature vectors 620.

[0188] FIG. 21 illustrates a method 2100 for using a classifier model 324 of the masterbot 114 to determine whether an input utterance 303 provided as a user input 110 is unrelated to any available skillbot 116 associated with the masterbot 114, according to some embodiments described herein. The method 2100 or a similar method can be used in block 215 of the method 200 described above for configuring and using the masterbot 114 to direct the input utterance 303 to the skillbot 116. The method 2100 of FIG. 21 is a more specific variation of the method 700 of FIG. 7, and like the method 700 of FIG. 7, the method 2100 of FIG. 21 can be used with the preliminary filtering operation described with respect to the method 800 of FIG. 8. More specifically, as described below, the classifier model 324 of the masterbot 114 can utilize a composite feature vector to determine whether the input utterance 303 belongs to a non-class 316 (i.e., is unrelated to an available skillbot 116).

[0189] The method 2100 shown in FIG. 21 and other methods described herein may be implemented in software (e.g., as code, instructions, or programs) executed by one or more processing units (e.g., processors or processor cores), in hardware, or a combination thereof. The software may be stored in a non-transitory storage medium such as a memory device. The method 2100 is intended to be exemplary and non-limiting. FIG. 21 shows various operations occurring in a particular sequence or order, but this is not intended to be limiting. In certain embodiments, for example, these operations may be performed in a different order, or one or more operations of the method 2100 may be performed in parallel. In certain embodiments, the method 200 may be performed by a masterbot 114 associated with a set of available skillbots 116.

[0190] In block 2105 of the method 2100, the masterbot 114 accesses the input utterance 303 provided as the user input 110. For example, in some embodiments, a user may provide the user input 110 in the form of voice input, and the digital assistant 106 may convert the user input 110 into a text input utterance 303 for use by the masterbot 114.

[0191] In block 2110, the Masterbot 114 may generate an input feature vector 1710 from the input utterance 303 accessed in block 2105. More specifically, in some embodiments, the Masterbot 114 causes the classifier model 324 of the Masterbot 114 to generate the input feature vector 1710 from the input utterance 303. The input feature vector 1710 may describe and represent the input utterance 303. Various techniques are known for converting a sequence of words, such as the input utterance 303, into a feature vector, and one or more of such techniques may be used. For example, the training system 350 may, but need not, encode the input utterance 303 as the corresponding input feature vector 1710 using one-hot encoding or some other encoding. However, one embodiment of the Masterbot 114 uses the same technique used to generate the training feature vector 620 from the training utterance 615 in training the classifier model 324.

[0192] In block 2115, the masterbot 114 compares the input feature vector 1710 generated in block 2110 to the composite feature vector. More specifically, in some embodiments, the masterbot 114 has the classifier model 324 compare the input feature vector 1710 determined in block 2110 to the composite feature vector, which may be determined as described above with respect to FIG. 18. In comparing the input feature vector 1710 to the composite feature vector, the classifier model 324 may determine a similarity or distance between the input feature vector 1710 and each composite feature vector previously constructed for the classifier model 324.

[0193] The similarity or distance between the input feature vector 1710 and another feature vector, specifically a composite feature vector, can be calculated in a variety of ways. For example, to determine the similarity, the classifier model 324 may calculate the absolute value of the arithmetic difference (i.e., Euclidean distance) between the composite feature vector and the input feature vector 1710 by subtracting one from the other to obtain the absolute value. For another example, the classifier model 324 may multiply the input feature vector 1710 by the composite feature vector. For example, if one-hot coding is used for the input feature vector 1710 and the composite feature vector, each vector entry (i.e., each dimension) will have a value of 1 or 0. In this case, a value of 1 may represent the presence of a particular feature. If the input feature vector 1710 and the composite feature vector have most of the same features, the result of the vector-vector multiplication will be a vector with nearly as many 1 values ​​as either of the two vectors being multiplied. If not, the resulting vector will be mostly 0s. As another example, cosine similarity can be used. For example, the cosine similarity between the input feature vector 1710 and the composite feature vector can be calculated as the dot product of the two vectors divided by the product of the Euclidean norms of both vectors. Various other techniques for measuring similarity are possible and are within the scope of this disclosure.

[0194] In decision block 2120, the classifier model 324 of the masterbot 114 determines whether the input feature vector 1710 is sufficiently similar to any of the composite feature vectors. The classifier model 324 may use a predefined threshold such that the similarity is deemed sufficient if it meets the threshold. As with the distance metric, if the similarity metric used provides a small value when the two vectors are similar, the threshold may be, for example, an upper threshold such that the input feature vector 1710 is sufficiently close to the composite feature vector if the similarity is below the threshold. However, if the similarity metric used provides a large value when the two vectors are similar, the threshold may be, for example, a lower threshold such that the input feature vector 1710 is sufficiently close to the composite feature vector if the similarity is above the threshold.

[0195] In some embodiments, the determination of sufficient similarity may be determined using a hierarchy of composite feature vectors. For example, the input feature vector 1710 may be compared to the bot vectors, and then, if necessary, the sub-bot vectors, and then, if necessary, the intent vectors, until a sufficient similarity is found or until it is determined that the input feature vector 1710 is not sufficiently similar to any such composite feature vector. Because there are fewer bot vectors than intent vectors, and fewer intent vectors than sub-bot vectors, the computing resources required to determine the similarity to the bot vectors may be less than the computing resources required to determine the similarity to the intent vectors, which may be less than the computing resources required to determine the similarity to the sub-bot vectors. Thus, by comparing the input feature vector 1710 to the composite feature vectors at various levels, starting from the highest level available (e.g., the skill bot level), the classifier model 324 can determine which skill bot 116, if any, is best suited to deal with the input utterance 303 while using less computationally intensive processing before employing more computationally intensive processing.

[0196] In some embodiments, an input utterance 303 may be deemed unrelated to any of the available skill bots 116 if the input feature vector 1710 is far enough away from all bot vectors (e.g., beyond a predetermined distance considered excessive). However, in additional or alternative embodiments, the input utterance 303 is not deemed unrelated solely based on differences from the bot vectors. For example, if the input feature vector 1710 is not similar (i.e., not similar enough) to all bot vectors, this is not necessarily a clear indication that provides a clue that the input utterance 303 belongs to a non-class 316. There may be situations where the input feature vector 1710 is far away (i.e., not similar) from any bot vector but is still close to an intent vector because the corresponding intent is not similar to other intents for which the skill bot 116 is configured. In some embodiments, comparing the input feature vector 1710 to both the bot vectors and the intent vectors is typically sufficient to identify an unrelated input utterance 303. Thus, an input utterance 303 may be considered irrelevant to available skill bots 116 if the input feature vector 1710 is not similar to all bot vectors and is not similar to all intent vectors. In some embodiments, the input feature vector 1710 may additionally or alternatively be compared to sub-bot vectors or other composite feature vectors as part of determining whether the input utterance 303 is a member of a non-class. Various implementations are possible and are within the scope of the present disclosure.

[0197] If, at decision block 2120, based on whatever comparisons are performed, the classifier model 324 determines that the input feature vector 1710 is not sufficiently similar to any of the composite feature vectors, the method 2100 proceeds to block 2125. At block 2125, the masterbot 114 determines that the input utterance is unrelated to any of the available skillbots 116, and thus the masterbot 114 may indicate that the input utterance 303 cannot be processed. As a result, for example, the digital assistant may ask the user to clarify the user input 110.

[0198] However, if in decision block 2120 the classifier model 324 determines that the input feature vector 1710 is sufficiently similar to one or more composite feature vectors, the method 2100 proceeds to block 2130. In block 2130, based on the comparison with the composite feature vector, the masterbot 114 determines (i.e., selects) a skillbot 116 to address the input utterance 303. For example, the masterbot 114 may select the skillbot 116 associated with the composite feature vector deemed most similar to the input feature vector 1710. For example, if the most similar composite feature vector is an intent vector, the masterbot 114 may select the skillbot 116 configured to address the intent corresponding to the intent vector. Furthermore, if the most similar composite feature vector is a sub-bot vector or a bot vector, the masterbot 114 may select the skillbot 116 to which the sub-bot vector or the bot vector corresponds. In another example, the masterbot 114 may use a k-nearest neighbor technique to select the skillbot 116, as further described below.

[0199] In block 2135, the masterbot 114 may route the input utterance to the skillbot 116 selected in block 2130. In some embodiments, if the masterbot 114 identifies a particular intent for the input utterance (e.g., if the input feature vector 1710 is deemed close enough to a single intent vector), the masterbot 114 may indicate that intent to the skillbot 116, thereby allowing the skillbot 116 to skip the process of inferring the intent. The skillbot 116 may then process the input utterance 303 to respond to the user input 110.

[0200] 22 illustrates a method 2200 for selecting a skillbot 116 to address an input utterance, according to some embodiments described herein. This method 2200 or a similar method may be used in block 2130 of the method 2100 described above after determining that the input utterance 303 is relevant to at least one available skillbot 116. This method 2200 provides a k-nearest neighbor approach to selecting a skillbot 116, although other techniques may be used in addition to or instead of this approach. In particular, this method 2200 utilizes the k-nearest neighbor training feature vector 620 to the input feature vector 1710 to determine which skillbot 116 should be selected.

[0201] The method 2200 shown in FIG. 22 and other methods described herein may be implemented in software (e.g., as code, instructions, or programs) executed by one or more processing units (e.g., processors or processor cores), in hardware, or a combination thereof. The software may be stored in a non-transitory storage medium, such as a memory device. The method 2200 is intended to be exemplary and non-limiting. FIG. 22 depicts various operations occurring in a particular sequence or order, but this is not intended to be limiting. In certain embodiments, for example, these operations may be performed in a different order, or one or more operations of the method 2200 may be performed in parallel. In certain embodiments, the method 2200 may be performed by the masterbot 114.

[0202] 22, in block 2205, the masterbot 114 determines a value for k, where k is the number of neighbors that will be considered. In some embodiments, the value of k may be a factor of the total number of training feature vectors 620 (i.e., the total number of training utterances 615).

[0203] In block 2210, using the value of k determined in block 2205, the Masterbot 114 determines a set of k training feature vectors 620 that are closest (i.e., most similar) to the input feature vector 1710. The Masterbot 114 may use the same or a different similarity metric as used above when determining whether the input feature vector 1710 is sufficiently similar to any of the composite feature vectors. For example, the similarity metric used may be Euclidean distance, vector multiplication, or cosine similarity.

[0204] In block 2215, the masterbot 114 selects the skillbot 116 that has the most training feature vectors 620 in the set determined in block 2210. In some embodiments, it is not necessary that the training feature vectors 620 of the skillbot 116 constitute the majority of the set, but rather, it is only necessary that no other skillbot 116 has a larger number of training feature vectors 620 in the set. As described above, after selecting this skillbot 116, the masterbot 114 may route the input utterance 303 to the selected skillbot 116 for processing.

[0205] In addition to or instead of considering the training feature vector 620 closest to the input feature vector 1710, some embodiments of the masterbot 114 may take into account the closest synthesis feature vector, such as the closest intent vector. FIG. 23 illustrates another exemplary method 2300 for selecting a skillbot 116 to address an input utterance, according to some embodiments described herein. This method 2300 or a similar method may be used in block 2130 of the method 2100 described above after determining that the input utterance 303 is relevant to at least one available skillbot 116. This method 2300 provides a k-nearest neighbor approach to selecting a skillbot 116, although other techniques may be used in addition to or instead of this approach. Specifically, in contrast to the method 2200 of FIG. 22, this method 2300 utilizes the k-nearest neighbor intent vectors to the input feature vector 1710 to determine which skillbot 116 should be selected.

[0206] The method 2300 shown in FIG. 23 and other methods described herein may be implemented in software (e.g., as code, instructions, or programs) executed by one or more processing units (e.g., processors or processor cores), in hardware, or a combination thereof. The software may be stored in a non-transitory storage medium, such as a memory device. The method 2300 is intended to be exemplary and non-limiting. FIG. 23 depicts various operations occurring in a particular sequence or order, but this is not intended to be limiting. In certain embodiments, for example, these operations may be performed in a different order, or one or more operations of the method 2300 may be performed in parallel. In certain embodiments, the method 2300 may be performed by the masterbot 114.

[0207] As shown in FIG. 23, in block 2305, the masterbot 114 determines a value for k, where k is the number of neighbors that will be considered. In some embodiments, the value of k may be a factor of the total number of intent vectors. The total number of intent vectors may be equal to the total number of intents represented in the training utterances 615 or the total number of intents that the available skillbots 116 are configured to address, and the value of k may be selected as a factor of that amount.

[0208] In block 2310, the Masterbot 114 uses the value of k determined in block 2305 to determine the set of k intent vectors that are closest (i.e., most similar) to the input feature vector 1710. The Masterbot 114 may use the same or a different similarity metric as used above when determining whether the input feature vector 1710 is sufficiently similar to any of the composite feature vectors in the method 2100. For example, the similarity metric used may be Euclidean distance, vector multiplication, or cosine similarity.

[0209] In block 2315, the masterbot 114 selects the skillbot 116 with the most intent vectors in the set determined in block 2310. In some embodiments, it is not necessary that the intent vectors of the skillbot 116 constitute the majority of the set, but rather, it is only necessary that no other skillbot 116 has more intent vectors in the set. As described above, after selecting this skillbot 116, the masterbot 114 may route the input utterance 303 to the selected skillbot 116 for processing.

[0210] Example of implementation 24 shows a simplified diagram of a distributed system 2400 for implementing one embodiment. In the illustrated embodiment, the distributed system 2400 includes one or more client computing devices 2402, 2404, 2406, and 2408 coupled to a server 2412 via one or more communication networks 2410. The client computing devices 2402, 2404, 2406, and 2408 may be configured to run one or more applications.

[0211] In various examples, the server 2412 may be adapted to run one or more services or software applications that enable the processes described in this disclosure.

[0212] In particular embodiments, the server 2412 may also provide other services or software applications, which may include non-virtualized and virtualized environments. In some embodiments, these services may be provided as web-based or cloud services, such as under a Software as a Service (SaaS) model, to users of the client computing devices 2402, 2404, 2406, and / or 2408. Users operating the client computing devices 2402, 2404, 2406, and / or 2408 may utilize the services provided by these components by interacting with the server 2412 utilizing one or more client applications.

[0213] In the configuration shown in Figure 24, the server 2412 may include one or more components 2418, 2420, and 2422 that implement the functions performed by the server 2412. These components may include software components that may be executed by one or more processors, hardware components, or combinations thereof. It should be appreciated that a wide variety of system configurations that may differ from the distributed system 2400 are possible. Thus, the embodiment shown in Figure 24 is an example of a distributed system for implementing the system of the embodiments and is not intended to be limiting.

[0214] A user may use client computing devices 2402, 2404, 2406 and / or 2408 to interact with server 2412 in accordance with the teachings of this disclosure. The client devices may provide an interface that allows a user of the client device to interact with the client device. The client devices may also output information to the user via this interface. Although FIG. 24 shows only four client computing devices, any number of client computing devices may be supported.

[0215] Client devices may include various types of computing systems, such as portable handheld devices, general purpose computers such as personal computers and laptops, workstation computers, wearable devices, gaming systems, thin clients, various messaging devices, sensors or other sensing devices, etc. These computing devices may run various types and versions of software applications and operating systems (e.g., Microsoft Windows®, Apple Macintosh®, UNIX® or UNIX-like operating systems, Linux® or Linux-like operating systems, e.g., Google Chrome® OS, including various mobile operating systems (e.g., Microsoft Windows Mobile®, iOS®, Windows Phone®, Android®, BlackBerry®, Palm OS®). Portable handheld devices may include cellular phones, smartphones (e.g., iPhone®), tablets (e.g., iPad®), personal digital assistants (PDAs), etc. Wearable devices may include Google Glass® head-mounted displays and other devices. The gaming systems may include various handheld gaming devices, Internet-enabled gaming devices (e.g., Microsoft Xbox® gaming consoles with or without Kinect® gesture input devices, Sony PlayStation® systems, various gaming systems offered by Nintendo®, etc.) The client devices may be capable of running a wide variety of applications, such as various Internet-related applications, communication applications (e.g., email applications, short message service (SMS) applications), and may use a variety of communication protocols.

[0216] Network 2410 may be any type of network known to those skilled in the art capable of supporting data communications using any of a variety of available protocols, including, but not limited to, TCP / IP (Transmission Control Protocol / Internet Protocol), SNA (Systems Network Architecture), IPX (Internet Packet Exchange), AppleTalk, etc. By way of example only, network 2410 may include a local area network (LAN), an Ethernet-based network, a token ring, a wide area network (WAN), the Internet, a virtual network, a virtual private network (VPN), an intranet, an extranet, a public switched telephone network (PSTN), an infrared network, a wireless network (e.g., a wireless network operating under any of the Institute of Electrical and Electronics Engineers (IEEE) 802.11 protocol suite, Bluetooth, and / or any other wireless protocol), and / or any combination of these and / or other networks.

[0217] The servers 2412 may be comprised of one or more general purpose computers, dedicated server computers (including, by way of example, PC (personal computer) servers, UNIX servers, mid-range servers, mainframe computers, rack-mounted servers, etc.), server farms, server clusters, or other suitable configurations and / or combinations. The servers 2412 may include one or more virtual machines running a virtual operating system, or other computing architectures involving virtualization, such as one or more flexible pools of logical storage that can be virtualized to maintain virtual storage for the servers. In various embodiments, the servers 2412 may be adapted to run one or more services or software applications that provide the functionality described in the disclosure above.

[0218] The computing systems in the servers 2412 may run one or more operating systems, including any of those mentioned above, as well as commercially available server operating systems. The servers 2412 may also run any of a variety of additional server and / or mid-tier applications, including a HyperText Transfer Protocol (HTTP) server, a File Transfer Protocol (FTP) server, a Common Gateway Interface (CGI) server, a JAVA server, a database server, and the like. Exemplary database servers include, but are not limited to, those commercially available from Oracle, Microsoft, Sybase, IBM (International Business Machines), and the like.

[0219] In some implementations, the server 2412 may include one or more applications for parsing and consolidating data feeds and / or event updates received from users of the client computing devices 2402, 2404, 2406, and 2408. By way of example, the data feeds and / or event updates may include, but are not limited to, Twitter® feeds, Facebook® updates, or real-time updates received from one or more third party sources and continuous data streams that may include real-time events related to sensor data applications, financial stock ticker boards, network performance measurement tools (e.g., network monitoring and traffic management applications), clickstream analysis tools, automobile traffic monitoring, etc. The server 2412 may also include one or more applications for displaying the data feeds and / or real-time events via one or more display devices of the client computing devices 2402, 2404, 2406, and 2408.

[0220] The distributed system 2400 may also include one or more data repositories 2414, 2416. In certain embodiments, these data repositories may be used to store data and other information. For example, one or more of the data repositories 2414, 2416 may be used to store data or information generated by and / or used for the processes described herein. The data repositories 2414, 2416 may be in a variety of locations. For example, the data repository used by the server 2412 may be local to the server 2412 or may be remote from the server 2412 and communicate with the server 2412 via a network-based or dedicated connection. The data repositories 2414, 2416 may be of different types. In certain embodiments, the data repository used by the server 2412 may be a database, for example, a relational database such as those provided by Oracle Corporation® and other manufacturers. One or more of these databases may be adapted to allow data to be stored, updated, and retrieved from the database in response to SQL-formatted commands.

[0221] In particular embodiments, one or more of the data repositories 2414, 2416 may be used by an application to store application data. The data repositories used by an application may be of various types, such as, for example, a key-value store repository, an object store repository, or a general-purpose storage repository supported by a file system.

[0222] In certain embodiments, the functionality described in this disclosure may be provided as a service via a cloud environment. FIG. 25 is a simplified block diagram of a cloud-based system environment that may provide the functionality described herein as a cloud service, according to certain embodiments. In the embodiment shown in FIG. 25, a cloud infrastructure system 2502 may provide one or more cloud services that users may request using one or more client computing devices 2504, 2506, and 2508. The cloud infrastructure system 2502 may include one or more computers and / or servers, which may include those previously described with respect to server 2412. The computers in the cloud infrastructure system 2502 may be organized as general purpose computers, dedicated server computers, server farms, server clusters, or any other suitable arrangement and / or combination.

[0223] The network 2510 may facilitate communication and exchange of data between the clients 2504, 2506, and 2508 and the cloud infrastructure system 2502. The network 2510 may include one or more networks. The networks may be of the same or different types. The network 2510 may support one or more communication protocols, including wired and / or wireless protocols, to facilitate communication.

[0224] The embodiment shown in Figure 25 is merely one example of a cloud infrastructure system and is not intended to be limiting. It should be understood that in other embodiments, cloud infrastructure system 2502 may have more or fewer components than those shown in Figure 25, may combine two or more components, or may have components in a different configuration or arrangement. For example, while Figure 25 shows three client computing devices, in alternative embodiments, any number of client computing devices may be supported.

[0225] The term cloud services is generally used to refer to services made available to users on demand via a communication network such as the Internet by a service provider's system (e.g., cloud infrastructure system 2502). Typically, in a public cloud environment, the servers and systems that make up the cloud service provider's system are different from the customer's own on-premise servers and systems. The cloud service provider's systems are managed by the cloud service provider. Thus, customers can use cloud services provided by the cloud service provider without having to purchase separate licenses, support, or hardware and software resources for the services. For example, the cloud service provider's system may host applications, and users may order and use the applications on demand via the Internet without having to purchase infrastructure resources to run the applications. Cloud services are designed to provide easy and scalable access to applications, resources, and services. Several providers offer cloud services. For example, several cloud services, such as middleware services, database services, and Java cloud services, are offered by Oracle Corporation of Redwood Shores, California.

[0226] In particular embodiments, cloud infrastructure system 2502 may provide one or more cloud services using a variety of models, such as a Software as a Service (SaaS) model, a Platform as a Service (PaaS) model, an Infrastructure as a Service (IaaS) model, etc., including hybrid service models. Cloud infrastructure system 2502 may include a suite of applications, middleware, databases, and other resources that enable the provisioning of various cloud services.

[0227] The SaaS model allows applications or software to be delivered as a service to customers over a communications network such as the Internet without the customer having to purchase hardware or software for the underlying application. For example, the SaaS model may be used to provide customers with access to on-demand applications hosted by the cloud infrastructure system 2502. Examples of SaaS services offered by Oracle Corporation include, but are not limited to, various services for human resource / capital management, customer relationship management (CRM), enterprise resource planning (ERP), supply management (SCM), enterprise performance management (EPM), analytical services, social applications, and the like.

[0228] The IaaS model is commonly used to provide flexible computing and storage capabilities by providing infrastructure resources (e.g., servers, storage, hardware and networking resources) to customers as cloud services. Various IaaS services are offered by Oracle Corporation.

[0229] The PaaS model is generally used to provide platform and environment resources as a service that enable customers to develop, run, and manage applications and services without having to procure, build, or manage the environment resources. Examples of PaaS services provided by Oracle Corporation (registered trademark) include, but are not limited to, Oracle Java Cloud Service (JCS), Oracle Database Cloud Service (DBCS), data management cloud services, various application development solution services, etc.

[0230] Cloud services are generally provided on an on-demand self-service basis, subscription-based, elastically scalable, reliable, highly available, and secure manner. For example, a customer may order one or more services offered by cloud infrastructure system 2502 via a subscription order. Cloud infrastructure system 2502 then provides the services requested in the customer's subscription order by executing processing. For example, in certain embodiments, chatbot-related functionality described herein may be provided as a user / subscriber-provided cloud service. Cloud infrastructure system 2502 may be configured to provide one cloud service or even multiple cloud services.

[0231] Cloud infrastructure system 2502 may provide cloud services through a variety of deployment models. In a public cloud model, cloud infrastructure system 2502 may be owned by a third-party cloud service provider and cloud services are provided to customers in the general public. The customers may be individuals or businesses. In other particular embodiments, under a private cloud model, cloud infrastructure system 2502 may function within an organization (e.g., within a corporate organization) and services are provided to customers within the organization. For example, the customers may be various departments of a company, such as human resources, payroll, etc., or may be individuals within the company. In some other embodiments, under a community cloud model, cloud infrastructure system 2502 and the services provided may be shared among various organizations within an associated community. Various other models may be used, including hybrids of the above models.

[0232] Client computing devices 2504, 2506, and 2508 may be of different types (e.g., devices 2402, 2404, 2406, and 2408 shown in FIG. 24) and may be capable of operating one or more client applications. Users may use the client devices to interact with cloud infrastructure system 2502, such as to request services provided by cloud infrastructure system 2502. For example, a user may use a client device to request a chatbot-related service described in this disclosure.

[0233] In some embodiments, the processing performed by cloud infrastructure system 2502 may include big data analytics. This analytics may include using, analyzing, and processing large data sets to detect and visualize various trends, behaviors, relationships, etc. within the data. This analytics may be performed by one or more processors, possibly processing the data in parallel, running simulations with the data, etc. The data used in this analytics may include structured data (e.g., data stored in a database or structured according to a structured model) and / or unstructured data (e.g., data blobs (binary large objects)).

[0234] 25 embodiment, cloud infrastructure system 2502 may include infrastructure resources 2530 utilized to facilitate the provision of various cloud services offered by cloud infrastructure system 2502. Infrastructure resources 2530 may include, for example, processing resources, storage or memory resources, networking resources, etc.

[0235] In certain embodiments, to facilitate efficient provisioning of these resources to support the various cloud services offered by cloud infrastructure system 2502 for different customers, resources may be organized into resource sets or resource modules (also referred to as "pods"). Each resource module or pod may include a pre-integrated and optimized combination of one or more types of resources. In certain embodiments, different pods may be pre-provisioned for different types of cloud services. For example, a first set of pods may be provisioned for database services, a second set of pods may be provisioned for Java services, etc., which may include a different combination of resources than the pods in the first set of pods. For some services, resources allocated to provision these services may be shared between the services.

[0236] Cloud infrastructure system 2502 itself may use services 2532 internally that are shared by various components of cloud infrastructure system 2502 and that facilitate provisioning of services by cloud infrastructure system 2502. These internal shared services may include, but are not limited to, security and identity services, integration services, enterprise repository services, enterprise manager services, virus scanning and whitelist services, high availability, backup and recovery services, services enabling cloud support, email services, notification services, file transfer services, etc.

[0237] Cloud infrastructure system 2502 may include multiple subsystems. These subsystems may be implemented in software, or hardware, or a combination thereof. As shown in FIG. 25, the subsystems may include a user interface subsystem 2512 that allows users or customers of cloud infrastructure system 2502 to interact with cloud infrastructure system 2502. User interface subsystem 2512 may include a variety of different interfaces, such as a web interface 2514, an online store interface 2516 through which cloud services provided by cloud infrastructure system 2502 are advertised and available for purchase by consumers, and other interfaces 2518. For example, a customer may use a client device to request (service request 2534) one or more services provided by cloud infrastructure system 2502 using one or more of interfaces 2514, 2516, and 2518. For example, a customer may access an online store, browse cloud services provided by cloud infrastructure system 2502, and place a subscription order for one or more services provided by cloud infrastructure system 2502 that the customer wishes to subscribe to. The service request may include information identifying the customer and one or more services that the customer wishes to request.

[0238] 25, the cloud infrastructure system 2502 may include an order management subsystem (OMS) 2520 configured to process new orders. As part of this processing, the OMS 2520 may be configured to create an account for the customer if not already created, receive billing and / or account information from the customer that will be used to charge the customer for providing the requested services to the customer, verify the customer information, and once verified, reserve the order for the customer, and prepare the order for provisioning by coordinating various workflows.

[0239] Upon proper validation, the OMS 2520 may invoke an order provisioning subsystem (OPS) 2524 configured to provision resources for the order, including processing, memory, and networking resources. Provisioning may include allocating resources for the order and configuring the resources to facilitate the service requested by the customer order. The manner in which resources are provisioned for the order and the type of resources provisioned may depend on the type of cloud service ordered by the customer. For example, following a workflow, the OPS 2524 may be configured to determine the specific cloud service being requested and identify the number of pods that may be pre-configured for the specific cloud service. The number of pods allocated for an order may depend on the size / amount / level / scope of the service requested. For example, the number of pods to allocate may be determined based on the number of users the service should support, the duration for which the service is requested, etc. The allocated pods may then be customized to the specific requesting customer to provide the requested service.

[0240] The cloud infrastructure system 2502 may send a response or notification 1044 to the requesting customer to indicate when the requested service will be available for use. In some examples, information (e.g., a link) may be sent to the customer that enables the customer to begin using and availing the benefits of the requested service.

[0241] Cloud infrastructure system 2502 may provide services to multiple customers. For each customer, cloud infrastructure system 2502 is responsible for managing information associated with one or more subscription orders received from the customer, maintaining customer data associated with the orders, and providing the requested services to the customer. Cloud infrastructure system 2502 may also collect usage statistics regarding the customer's use of the subscribed services. For example, statistics may be collected on the amount of storage used, the amount of data transferred, the number of users, and the amount of system uptime and system downtime, etc. This usage information may be used to bill the customer. Billing may be on a monthly basis, for example.

[0242] Cloud infrastructure system 2502 may provide services to multiple customers in parallel. Cloud infrastructure system 2502 may store information about these customers, possibly including copyright information. In certain embodiments, cloud infrastructure system 2502 includes an identity management subsystem (IMS) 2528 configured to manage information about customers and isolate the managed information so that information about one customer is not accessed by another customer. IMS 2528 may be configured to provide various security-related services, such as identity services such as information access management, authentication and authorization services, services for managing customer identities and roles and associated capabilities, and the like.

[0243] FIG. 26 illustrates an example computer system 2600 that may be used to implement certain embodiments. For example, in some embodiments, the computer system 2600 may be used to implement any of the systems and subsystems of the chatbot system, as well as the various servers and computer systems described above. As shown in FIG. 26, the computer system 2600 includes various subsystems, including a processing subsystem 2604 that communicates with several other subsystems via a bus subsystem 2602. These other subsystems may include a processing acceleration unit 2606, an I / O subsystem 2608, a storage subsystem 2618, and a communication subsystem 2624. The storage subsystem 2618 may include a non-transitory computer-readable storage medium, including a storage medium 2622 and a system memory 2610.

[0244] Bus subsystem 2602 provides a mechanism for allowing the various components and subsystems of computer system 2600 to communicate with each other as intended. Although bus subsystem 2602 is shown diagrammatically as a single bus, alternative embodiments of the bus subsystem may utilize multiple buses. Bus subsystem 2602 may be any of several types of bus structures including a memory bus or memory controller, a peripheral bus, a local bus, etc., using any of a variety of bus architectures. For example, such architectures may include an Industry Standard Architecture (ISA) bus, a Micro Channel Architecture (MCA) bus, an Enhanced ISA (EISA) bus, a Video Electronics Standards Association (VESA) local bus, a Peripheral Component Interconnect (PCI) bus, which may be implemented as a mezzanine bus manufactured in accordance with the IEEE P1386.1 standard, and the like.

[0245] The processing subsystem 2604 controls the operation of the computer system 2600 and may include one or more processors, application specific integrated circuits (ASICs), or field programmable gate arrays (FPGAs). The processors may include single-core or multi-core processors. The processing resources of the computer system 2600 may be organized into one or more processing units 2632, 2634, etc. The processing units may include one or more processors, one or more cores from the same or different processors, combinations of cores and processors, or other combinations of cores and processors. In some embodiments, the processing subsystem 2604 may include one or more dedicated co-processors, such as graphics processors, digital signal processors (DSPs), etc. In some embodiments, some or all of the processing units of the processing subsystem 2604 may be implemented using customized circuitry, such as application specific integrated circuits (ASICs) or field programmable gate arrays (FPGAs).

[0246] In some embodiments, the processing units in the processing subsystem 2604 may execute instructions stored in the system memory 2610 or the computer-readable storage medium 2622. In various embodiments, the processing units may execute various program or code instructions as well as maintain multiple programs or processes executing simultaneously. At any given time, some or all of the program code to be executed may reside in the system memory 2610 and / or the computer-readable storage medium 2622, potentially including one or more storage devices. Through appropriate programming, the processing subsystem 2604 may provide the various functions described above. In an example where the computer system 2600 is running one or more virtual machines, one or more processing units may be assigned to each virtual machine.

[0247] In particular embodiments, a processing acceleration unit 2606 may optionally be provided to accelerate the overall processing performed by the computer system 2600, to perform customized processing, or to offload some of the processing performed by the processing subsystem 2604.

[0248] The I / O subsystem 2608 may include devices and mechanisms for inputting information to the computer system 2600 and / or outputting information from or through the computer system 2600. In general, use of the term "input device" is intended to include all conceivable types of devices and mechanisms for inputting information to the computer system 2600. User interface input devices may include, for example, keyboards, pointing devices such as mice or trackballs, touchpads or touchscreens integrated into displays, scroll wheels, click wheels, dials, buttons, switches, keypads, voice input devices with voice command recognition systems, microphones, and other types of input devices. User interface input devices may also include motion sensing and / or gesture recognition devices, such as Microsoft Kinect® motion sensors that allow a user to control and interact with the input device, Microsoft Xbox® 360 game controllers, devices with interfaces for receiving input using gestures and voice commands. The user interface input devices may also include eye gesture recognition devices, such as a Google Glass® blink detector, that detects eye movements from the user (e.g., "blinking" while taking a picture and / or making a menu selection) and translates the eye gestures as input to the input device (e.g., Google Glass®). Additionally, the user interface input devices may include a voice recognition sensing device that allows the user to interact with a voice recognition system (e.g., the Siri® navigator) via voice commands.

[0249] Other examples of user interface input devices include, but are not limited to, three-dimensional (3D) mice, joysticks or pointing sticks, game pads and graphic tablets, as well as audio / visual devices such as speakers, digital cameras, digital camcorders, portable media players, webcams, image scanners, fingerprint scanners, barcode readers 3D scanners, 3D printers, laser range finders, and eye-tracking devices. In addition, user interface input devices may include medical imaging input devices, such as, for example, computed tomography, magnetic resonance imaging, position emission tomography, and medical ultrasound devices. User interface input devices may also include audio input devices, such as, for example, MIDI keyboards, digital musical instruments, and the like.

[0250] In general, use of the term output device is intended to include all conceivable types of devices and mechanisms for outputting information from computer system 2600 to a user or to another computer. User interface output devices may include display subsystems, indicator lights, or non-visual displays such as audio output devices, and the like. Display subsystems may be flat panel devices such as those using cathode ray tubes (CRTs), liquid crystal displays (LCDs) or plasma displays, projection devices, touch screens, and the like. For example, user interface output devices may include, but are not limited to, a variety of display devices that visually convey text, graphics, and audio / visual information, such as monitors, printers, speakers, headphones, automobile navigation systems, plotters, audio output devices, and modems.

[0251] The storage subsystem 2618 provides a repository or data store for storing information and data used by the computer system 2600. The storage subsystem 2618 provides a tangible, non-transitory computer-readable storage medium for storing the basic programming and data constructs that provide the functionality of some embodiments. Software (e.g., programs, code modules, instructions) that, when executed by the processing subsystem 2604, provide the functionality described above may be stored in the storage subsystem 2618. The software may be executed by one or more processing units of the processing subsystem 2604. The storage subsystem 2618 may also provide a repository for storing data used in accordance with the teachings of the present disclosure.

[0252] The storage subsystem 2618 may include one or more non-transitory memory devices, including volatile and non-volatile memory devices. As shown in FIG. 26, the storage subsystem 2618 includes a system memory 2610 and a computer-readable storage medium 2622. The system memory 2610 may include several memories, including a volatile main random access memory (RAM) for storing instructions and data during program execution, and a non-volatile read-only memory (ROM) or flash memory in which fixed instructions are stored. In some implementations, a basic input / output system (BIOS), containing the basic routines that help transfer information between elements within the computer system 2600, such as during start-up, may typically be stored in a ROM. Typically, the RAM contains data and / or program modules currently being operated on and executed by the processing subsystem 2604. In some implementations, the system memory 2610 may include several different types of memories, such as static random access memory (SRAM), dynamic random access memory (DRAM), and the like.

[0253] 26, system memory 2610 may load running application programs 2612, program data 2614, and operating system 2616, which may include various applications such as a web browser, a middle tier application, a relational database management system (RDBMS), etc. By way of example, operating system 2616 may include Microsoft Windows®, Apple Macintosh®, and / or Linux operating systems, various commercially available UNIX® or UNIX-like operating systems (including but not limited to various GNU / Linux operating systems, Google Chrome® OS, etc.), and / or various versions of mobile operating systems such as iOS®, Windows Phone, Android® OS, BlackBerry® OS, Palm® OS operating systems, etc.

[0254] The computer-readable storage medium 2622 may store programming and data configurations that provide functionality of some embodiments. The computer-readable storage medium 2622 may provide storage of computer-readable instructions, data structures, program modules, and other data for the computer system 2600. Software (programs, code modules, instructions) that provide the above-mentioned functionality when executed by the processing subsystem 2604 may be stored in the storage subsystem 2618. By way of example, the computer-readable storage medium 2622 may include non-volatile memory such as a hard disk drive, a magnetic disk drive, a CD ROM, a DVD, an optical disk drive such as a Blu-Ray® disk, or other optical media. The computer-readable storage medium 2622 may include, but is not limited to, a Zip® drive, a flash memory card, a Universal Serial Bus (USB) flash drive, a Secure Digital (SD) card, a DVD disk, a digital video tape, and the like. The computer-readable storage medium 2622 may also include solid-state drives (SSDs) based on non-volatile memory such as flash memory-based SSDs, enterprise flash drives, solid-state ROM, etc., SSDs based on volatile memory such as solid-state RAM, dynamic RAM, static RAM, DRAM-based SSDs, magnetoresistive RAM (MRAM) SSDs, and hybrid SSDs that use a combination of DRAM and flash memory-based SSDs.

[0255] In particular embodiments, storage subsystem 2618 may also include a computer-readable storage medium reader 2620 that may be further connected to a computer-readable storage medium 2622. Reader 2620 may be configured to receive and read data from a memory device such as a disk, a flash drive, or the like.

[0256] In certain embodiments, computer system 2600 may support virtualization techniques, including but not limited to virtualization of processing and memory resources. For example, computer system 2600 may provide support for running one or more virtual machines. In certain embodiments, computer system 2600 may execute a program, such as a hypervisor, that facilitates configuration and management of virtual machines. Each virtual machine may be assigned memory, computing (e.g., processors, cores), I / O, and networking resources. Each virtual machine typically runs independently from other virtual machines. A virtual machine typically runs its own operating system, which may be the same or different from the operating systems run by other virtual machines executed by computer system 2600. Thus, potentially multiple operating systems may be run simultaneously by computer system 2600.

[0257] The communications subsystem 2624 provides an interface to other computer systems and networks. The communications subsystem 2624 serves as an interface for sending and receiving data between other systems and the computer system 2600. For example, the communications subsystem 2624 may enable the computer system 2600 to establish a communications channel over the Internet to one or more client devices to send information to or receive information from the one or more client devices.

[0258] The communications subsystem 2624 may support both wired and / or wireless communication protocols. For example, in certain embodiments, the communications subsystem 2624 may include a radio frequency (RF) transceiver component for accessing wireless voice and / or data networks (e.g., using cellular telephone technology, advanced data network technologies such as 3G, 4G, or EDGE (Enhanced Data Rates for Global Evolution), WiFi (IEEE 802.XX family of standards, or other mobile communications technologies, or any combination thereof), a global positioning system (GPS) receiver component, and / or other components. In some embodiments, the communications subsystem 2624 may provide a wired network connection (e.g., Ethernet) in addition to or in place of a wireless interface.

[0259] The communications subsystem 2624 may receive and transmit data in a variety of formats. For example, in some embodiments, the communications subsystem 2624 may receive incoming communications in the form of structured and / or unstructured data feeds 2626, event streams 2628, event updates 2630, etc., in addition to other formats. For example, the communications subsystem 2624 may be configured to receive (or transmit) data feeds 2626 in real time from users of social media networks and / or other communications services, such as web feeds, such as Twitter® feeds, Facebook® updates, Rich Site Summary (RSS) feeds, and / or real-time updates from one or more third party sources.

[0260] In particular embodiments, the communications subsystem 2624 may be configured to receive data in the form of a continuous data stream, which may include an event stream 2628 of real-time events and / or event updates 2630, which may be continuous or infinite in nature without a clear end. Examples of applications that generate continuous data may include, for example, sensor data applications, financial stock ticker boards, network performance measurement tools (e.g., network monitoring and traffic management applications), clickstream analysis tools, automobile traffic monitoring, and the like.

[0261] The communications subsystem 2624 may be configured to communicate data from the computer system 2600 to other computer systems or networks. This data may be communicated in a variety of different formats, such as structured and / or unstructured data feeds 2626, event streams 2628, event updates 2630, etc., to one or more databases that may be in communication with one or more streaming data source computers coupled to the computer system 2600.

[0262] The computer system 2600 may be one of a variety of types, including a handheld portable device (e.g., an iPhone® cellular phone, an iPad® computing tablet, a PDA), a wearable device (e.g., a Google Glass® head-mounted display), a personal computer, a workstation, a mainframe, a kiosk, a server rack, or other data processing system. Due to the ever-changing nature of computers and networks, the description of the computer system 2600 shown in FIG. 26 is intended only as a specific example. Many other configurations having more or fewer components than the system shown in FIG. 26 are possible. Those skilled in the art will recognize other aspects and / or methods for implementing various embodiments based on the disclosure and teachings herein.

[0263] Although specific embodiments have been described, various modifications, variations, alternative configurations, and equivalents are possible. The embodiments are not limited to operation in a specific data processing environment, but may freely operate in multiple data processing environments. In addition, while specific embodiments have been described using a specific sequence of transactions and steps, it should be apparent to one skilled in the art that this is not intended to be limiting. Although some flow charts describe operations as a sequential process, many of these operations may be performed in parallel or simultaneously. In addition, the order of operations may be re-specified. A process may have additional steps not included in the figures. Various features and aspects of the above-described embodiments may be used individually or together.

[0264] Additionally, while certain embodiments have been described using particular combinations of hardware and software, it should be understood that other combinations of hardware and software are possible. Certain embodiments may be implemented exclusively in hardware, exclusively in software, or using a combination thereof. The various processes described herein may be implemented on the same processor or on separate processors in any combination.

[0265] Where a device, system, component or module is described as being configured to perform a particular operation or function, such configuration may be achieved, for example, by designing an electronic circuit to perform the operation, by programming a programmable electronic circuit (such as a microprocessor) to perform the operation, by executing, for example, computer instructions or code, or a processor or core programmed to execute code or instructions stored in a non-transitory memory medium or any combination thereof. Processes may communicate using a variety of techniques, including, but not limited to, conventional techniques for inter-process communication, and different pairs of processes may use different techniques, and the same pair of processes may use different techniques at different times.

[0266] In the present disclosure, specific details are set forth to allow the embodiments to be fully understood. However, the embodiments may be practiced without these specific details. For example, well-known circuits, processes, algorithms, structures, and techniques are shown without unnecessary detail so as not to obscure the embodiments. This specification provides only exemplary embodiments and is not intended to limit the scope, applicability, or configuration of other embodiments. Rather, the above description of the embodiments provides one skilled in the art with an enabling description for implementing various embodiments. Various changes are possible within the function and configuration of the elements.

[0267] The specification and accompanying drawings are therefore to be regarded in an illustrative and not restrictive sense. It will be apparent, however, that additions, subtractions, deletions, and other modifications and alterations may be made thereto without departing from the broader spirit and scope as set forth in the claims. Thus, although specific embodiments have been described, they are not intended to be limiting. Various modifications and equivalents, as well as any relevant combination of the disclosed features, are within the scope of the appended claims.

Claims

1. 1. A system comprising: a training system configured to train a classifier model, training the classifier model comprising: accessing training utterances associated with a skill bot, the training utterances including respective training utterances associated with each of the skill bots, each of the skill bots being configured to provide a dialogue with a user; and training the classifier model further comprising: generating training feature vectors from the training utterances, the training feature vectors including a respective training feature vector associated with each of the skill bots; and training the classifier model further comprises: generating a plurality of set representations of the training feature vector, each set representation of the plurality of set representations corresponding to a subset of the training feature vector, and training the classifier model further comprises: configuring the classifier model to compare an input feature vector to the plurality of set representations of the training feature vector, the system further comprising: A master bot, the master bot comprising: accessing an input utterance as user input; generating an input feature vector from the input utterance; comparing the input feature vector with the plurality of sets of representations of the training feature vectors using the classifier model; and outputting an indication that the user input cannot be addressed by the skill bot based on the input feature vector being outside of the plurality of set representations.

2. generating the plurality of set representations of the training feature vectors includes generating clusters to which the training feature vectors are assigned; The system of claim 1 , wherein comparing the input feature vector to the plurality of set representations comprises determining that the input feature vector does not fall within a boundary of the cluster.

3. generating the clusters determining respective centroid positions for the initial clusters in the feature space; and assigning each training feature vector to an initial cluster having a centroid position to which the training feature vector is closest among the initial clusters, wherein generating the clusters further includes: determining boundaries of the initial clusters, each of the initial clusters including a corresponding assigned training feature vector, and generating the clusters further includes: updating the initial clusters in response to determining that a stopping condition is not satisfied, wherein updating the initial clusters includes: determining an incremental count of a cluster such that the incremental count of the cluster is greater than an initial count of one of the initial clusters; determining respective centroid positions for the clusters in the feature space; assigning each training feature vector to one of the clusters having a respective centroid position to which the training feature vector is closest; and determining the boundaries of the clusters, each of the clusters including a corresponding assigned training feature vector.

4. The action of the master bot further comprises: accessing a second input utterance as a second user input; generating a second input feature vector from the second input utterance; determining that the second input feature vector is within a boundary of a cluster of the clusters; and forwarding the second input utterance to a skill bot associated with the certain cluster for processing based on the second input feature vector being within the boundary of the certain cluster.

5. The system of claim 4 , further comprising the skillbot, the skillbot configured to process the input utterance to perform an action in response to the user input.

6. Generating the plurality of set representations of the training feature vectors comprises: Segmenting the training utterances into conversational categories; and generating a composite feature vector corresponding to the conversation categories, wherein generating the composite feature vector comprises: generating, for each conversation category of the conversation categories, a respective composite feature vector as a collection of training feature vectors of respective training utterances in the conversation category.

7. 7. The system of claim 6, wherein for each conversation category, generating the composite feature vector as a collection of the respective training feature vectors of the training utterances in the conversation category comprises averaging the respective training feature vectors of the training utterances in the conversation category.

8. 7. The system of claim 6, wherein the conversation categories are defined based on intents for which the skillbot is configured, such that each conversation category corresponds to a respective skillbot intent and includes training utterances that represent the respective skillbot intent.

9. The system of claim 6 , wherein each of the conversation categories corresponds to a respective one of the skill bots and includes training utterances representative of the respective skill bot.

10. 7. The system of claim 6, wherein comparing the input feature vector to the multiple set representations of the training feature vector comprises determining that the input feature vector is not sufficiently similar to any of the synthetic feature vectors.

11. The action of the master bot further comprises: accessing a second input utterance as a second user input; generating a second input feature vector from the second input utterance; determining that the second input feature vector is sufficiently similar to a composite feature vector of the composite feature vectors; and based on the second input feature vector being sufficiently similar to the synthetic feature vector, forwarding the second input utterance to a skill bot associated with the synthetic feature vector for processing.

12. The system of claim 11 , further comprising the skillbot, the skillbot configured to process the input utterance to perform an action in response to the user input.

13. 1. A method comprising: accessing, by a computer system, training utterances associated with skill bots, the training utterances including a respective subset of training utterances for each of the skill bots, each of the skill bots being configured to provide an interaction with a user, the method further comprising: generating training feature vectors from the training utterances, the training feature vectors including a respective training feature vector for each training utterance of the training utterances, the method further comprising: determining centroid positions for the clusters in the feature space; assigning each training feature vector to a respective one of the clusters having a respective centroid position to which the training feature vector is closest; and iteratively modifying the clusters until a stopping condition is met, the step of modifying the clusters comprising: incrementing the count of the clusters to an updated count; determining a new centroid position for the cluster by an amount equal to the updated count; and reassigning the training feature vectors to the clusters based on their proximity to the new centroid locations, the method further comprising: determining boundaries of the clusters, the boundaries including a respective boundary for each of the clusters; accessing an input utterance; converting the input utterance into an input feature vector; determining that the input feature vector is outside the boundary of the cluster by comparing the input feature vector to the boundary of the cluster; and outputting an indication that the input utterance cannot be addressed by the skill bot based on the input feature vector being outside of the cluster in the feature space.

14. accessing a second input utterance as a second user input; generating a second input feature vector from the second input utterance; determining that the second input feature vector is within one or more of the clusters; and forwarding the second input utterance to a skill bot associated with the one or more clusters for processing based on the second input feature vector being within the range of the one or more clusters.

15. The one or more clusters include respective training utterances of the skill bot and respective training utterances of a second skill bot, and the method further comprises: The method of claim 14 , comprising selecting the skill bot from between the skill bot and the second skill bot to process the input utterance based on respective confidence scores calculated for the skill bot and the second skill bot.

16. The one or more clusters include respective training utterances of the skill bot and respective training utterances of a second skill bot, and the method further comprises:

15. The method of claim 14, comprising selecting a skillbot from among the skillbot and the second skillbot to process the input utterance based on applying a k-nearest neighbor technique to the respective training utterances of the skillbot and the respective training utterances of the second skillbot.

17. 1. A method comprising: accessing, by a computer system, training utterances associated with skill bots, the training utterances including a respective subset of training utterances for each of the skill bots, each of the skill bots configured to provide an interaction with a user, the method further comprising: generating training feature vectors from the training utterances, the training feature vectors including a respective training feature vector for each training utterance of the training utterances, the method further comprising: Segmenting the training utterances into speech categories; and generating composite feature vectors corresponding to the conversation categories, wherein generating the composite feature vectors comprises, for each conversation category of the conversation categories, generating a respective composite feature vector as a collection of training feature vectors for each of the training utterances in the conversation category, the method further comprising: accessing an input utterance; converting the input utterance into an input feature vector; determining that the input feature vector is not sufficiently similar to the composite feature vector by comparing the input feature vector to the composite feature vector; and outputting an indication that the input utterance cannot be addressed by the skill bot based on the input feature vector being not sufficiently similar to the synthetic feature vector.

18. accessing a second input utterance; converting the second input utterance into a second input feature vector; determining that the second input feature vector is sufficiently similar to a composite feature vector of the composite feature vectors by comparing the second input feature vector to the composite feature vectors; and forwarding the second input utterance to a skill bot associated with the synthetic feature vector for processing based on the second input feature vector being sufficiently similar to the synthetic feature vector.

19. the conversation categories are defined based on the intents that the skill bot is configured to address, such that each conversation category corresponds to a respective one or more skill bot intents and includes training utterances representative of the respective one or more skill bot intents; The method of claim 18 , wherein the composite feature vector corresponds to a skillbot intent that the skillbot is configured to address.

20. determining that the second input feature vector is sufficiently similar to the composite feature vector by comparing the second input feature vector to the composite feature vector, determining that the second input feature vector is sufficiently similar to one or more additional composite feature vectors corresponding to one or more additional skillbot intents of the skillbot; and performing a k-nearest neighbor analysis, the performing the k-nearest neighbor analysis comprising: The step of performing a k-nearest neighbor analysis further comprises identifying neighboring composite feature vectors in a predefined amount, the neighboring composite feature vectors being closest to the input feature vector, the step of performing a k-nearest neighbor analysis further comprising: determining that a majority of the composite feature vectors of the neighborhood correspond to a skillbot intent that the skillbot is configured to address; and selecting the skillbot based on the majority of the composite feature vectors of the neighborhood that correspond to a skillbot intent that the skillbot is configured to address.

21. A program for causing a computer to execute the method according to any one of claims 13 to 20.

Images