Detection of irrelevant utterances in chatbot systems

Abstract

To provide a system and method for determining whether or not an input utterance is irrelevant to a set of skill bots associated with a master bot.SOLUTION: A system comprises a training system 350 and a master bot 114. The training system 350 trains a classifier of the master bot 114, in which the training includes accessing a training utterance associated with the skill bot and generating a training feature vector from the training utterance. The master bot 114 executes generating an input feature vector together with accessing the input utterance and comparing the input feature vector with a plurality of sets of expressions using the classifier, to thereby determine whether or not the input feature is out of a range and hence it is impossible to cope therewith by the skill bot.SELECTED DRAWING: Figure 3

Description

BACKGROUND ART

[0001] Cross-Reference to Related Applications This application claims the benefit of and priority to U.S. Provisional Application No. 62 / 899,700, filed September 12, 2019, entitled "Detecting Unrelated Utterances in a Chatbot System" under 35 U.S.C. § 119(e), the content of which is incorporated herein by reference in its entirety for all purposes.

[0002] Background A chatbot is an artificial intelligence-based software application or device that provides an interface for conversation with a human user. Chatbots can be programmed to perform various tasks in response to user inputs provided during a conversation. User inputs can be provided in various forms including, for example, voice inputs and text inputs. Accordingly, natural language understanding, speech-to-text, and other language processing techniques may be employed as part of the processing performed by chatbots. In some computing environments, multiple chatbots are available to interact with users, and each chatbot addresses a respective different set of tasks. SUMMARY OF THE INVENTION

[0003] Overview The techniques described in this specification are for determining that an input utterance from a user is not related to any skill bot, also referred to as a chatbot, within a set of one or more skill bots available to a master bot. In some embodiments, the master bot may evaluate the input utterance, may determine that the input utterance is unrelated to the skill bots, or may route the input utterance to an appropriate skill bot. MEANS FOR SOLVING THE PROBLEMS

[0004] In some embodiments, the system described herein includes a training system and a master bot. The training system is configured to train a classifier model. Training the classifier model includes accessing training utterances associated with the skill bots, the training utterances including the respective training utterances associated with each skill bot. Each skill bot is configured to provide interaction with a user. Training further includes generating training feature vectors from the training utterances. The training feature vectors include the respective training feature vectors associated with each skill bot. Training further includes generating multiple set representations of the training feature vectors. Each set representation in the multiple set representations corresponds to a subset of the training feature vectors. Training further includes configuring the classifier model to compare an input feature vector with the multiple set representations. The master bot is configured to access input utterances as user input and to generate an input feature vector from the input utterances. The master bot is further configured to use the classifier model to compare the input feature vector with multiple set representations of the training feature vectors and to output an instruction that the skill bot cannot address the user input based on the input feature vector falling outside the range of the multiple set representations.

[0005] In additional or alternative embodiments, the method described herein involves a computer system accessing training utterances associated with the skillbot. The training utterances include a subset of the training utterances for each skillbot. Each skillbot is configured to provide interaction with a user. The method further includes the step of generating training feature vectors from the training utterances, the training feature vectors including a separate training feature vector for each training utterance. The method further includes the step of determining the centroid positions for clusters in feature space, and assigning each training feature vector to each cluster having the centroid position to which the training feature vector is closest. The method further includes the step of iteratively modifying the clusters until a stopping condition is met. The step of modifying the clusters includes increasing the count of the clusters to an updated count, determining a new centroid position for the clusters by an amount equal to the updated count, and reallocating the training feature vectors to the clusters based on their proximity to the new centroid positions. The method further includes the step of determining the boundaries of the clusters, the boundaries including a separate boundary for each cluster of the clusters. The method further includes the steps of accessing an input utterance, converting the input utterance into an input feature vector, and determining that the input feature vector is outside the range of the cluster boundary by comparing the input feature vector to the cluster boundary. In addition, the method includes the step of outputting an instruction that the skill bot cannot process the input utterance based on the fact that the input feature vector is outside the range of the cluster in the feature space.

[0006] In further additional or alternative embodiments, the method described herein includes the step of accessing training utterances associated with a skillbot by a computer system. The training utterances include a respective subset of training utterances for each skillbot. Each skillbot is configured to provide interaction with a user. The method further includes the step of generating training feature vectors from the training utterances, the training feature vectors including a respective training feature vector for each training utterance, and the method further includes the step of dividing the training utterances into conversation categories. The method further includes the step of generating composite feature vectors corresponding to the conversation categories. The step of generating composite feature vectors includes, for each conversation category, generating a respective composite feature vector as a set of the respective training feature vectors of the training utterances within that conversation category. The method further includes the steps of accessing an input utterance, converting the input utterance into an input feature vector, and comparing the input feature vector to the composite feature vector to determine that the input feature vector is not sufficiently similar to the composite feature vector. In addition, the method includes the step of outputting an instruction that the skill bot cannot process the input utterance based on the fact that the input feature vector is not sufficiently similar to the composite feature vector.

[0007] In further additional or alternative embodiments, the system described herein includes a master bot. The master bot is configured to perform operations including accessing an input utterance as user input, generating an input feature vector from the input utterance, comparing the input feature vector to a set of representations of training feature vectors using a classifier model, and outputting an instruction that the skill bot cannot address the user input based on the input feature vector being outside the range of the set of representations. Each skill bot is configured to provide interaction with the user.

[0008] In further additional or alternative embodiments, the method described herein is an input utterance. The method includes the step of accessing the input utterance. The method further includes the step of converting the input utterance into an input feature vector. The method further includes the step of determining that the input feature vector is outside the boundaries of a cluster by comparing the input feature vector to the boundaries of a cluster in the feature space. The method further includes the step of outputting an instruction that the skill bot cannot process the input utterance based on the fact that the input feature vector is outside the boundaries of the cluster in the feature space. Each skill bot is configured to provide interaction with the user.

[0009] In further additional or alternative embodiments, the method described herein includes the step of accessing an input utterance. The method further includes the step of converting the input utterance into an input feature vector. The method further includes the step of comparing the input feature vector to a composite feature vector and determining that the input feature vector is not sufficiently similar to the composite feature vector, which corresponds to a conversation category. The method further includes the step of outputting an instruction that the skillbot cannot address the input utterance based on the fact that the input feature vector is not sufficiently similar to the composite feature vector. Each skillbot is configured to provide interaction with a user.

[0010] In further additional or alternative embodiments, the method described herein includes the step of accessing an input utterance as user input. The method further includes the step of generating an input feature vector from the input utterance. The method further includes the step of comparing the input feature vector to a set of representations of the training feature vector using a classifier model. The method further includes the step of outputting an instruction that the skillbot cannot address the user input based on the input feature vector being outside the range of the set of representations. Each skillbot is configured to provide interaction with the user.

[0011] In further additional or alternative embodiments, the method described herein may be used to train a classifier model. The method includes accessing training utterances associated with a skillbot, the training utterances including the respective training utterances associated with each skillbot of the skillbot, each skillbot of the skillbot configured to provide interaction with a user. The method further includes generating training feature vectors from the training utterances, the training feature vectors including the respective training feature vectors associated with each skillbot of the skillbot. The method further includes generating a plurality of set representations of the training feature vectors, each set representation of the plurality of set representations corresponding to a subset of the training feature vectors. The method further includes configuring the classifier model to compare an input feature vector with the plurality of set representations of the training feature vectors.

[0012] In further additional or alternative embodiments, the method described herein is used to generate clusters that can be used to determine whether an input utterance can be addressed by a skillbot. The method includes the step of a computer system accessing training utterances associated with a skillbot, the training utterances including a subset of training utterances for each skillbot of the skillbot. Each skillbot of the skillbot is configured to provide interaction with a user. The method further includes the step of generating training feature vectors from the training utterances, the training feature vectors including a respective training feature vector for each training utterance of the training utterances. The method further includes the step of determining the centroid position of clusters in feature space. The method further includes the steps of assigning each training feature vector of the training feature vector to each cluster among the clusters having the closest centroid locations to the training feature vector, and iteratively modifying the clusters until a stopping condition is met. The steps of modifying the clusters include increasing the count of the clusters to an updated count, determining a new centroid location for the clusters by an amount equal to the updated count, and reallocating the training feature vectors to the clusters based on their proximity to the new centroid locations. The method further includes determining the boundaries of the clusters, the boundaries of which include a boundary for each cluster within the clusters.

[0013] In further additional or alternative embodiments, the method described herein is used to generate a composite feature vector that can be used to determine whether an input utterance can be handled by a skillbot. The method further includes the step of accessing training utterances associated with a skillbot by a computer system, the training utterances comprising each subset of training utterances for each skillbot of the skillbot. Each skillbot of the skillbot is configured to provide interaction with a user. The method further includes the step of generating a training feature vector from the training utterances, the training feature vector comprising each training feature vector for each training utterance of the training utterances. The method further includes the step of dividing the training utterances into conversation categories. The method further includes the step of generating a composite feature vector corresponding to the conversation categories. The step of generating the composite feature vector includes, for each conversation category, generating a composite feature vector as a set of each training feature vector of the training utterances within that conversation category.

[0014] In further additional or alternative embodiments, the systems described herein include means for carrying out any of the methods described above.

[0015] The foregoing, along with other features and embodiments, will become clearer upon reference to the following specification, the appended claims, and the appended drawings. [Brief explanation of the drawing]

[0016] [Figure 1] This block diagram shows an environment including a master bot that communicates with various skill bots, also known as chatbots, according to some embodiments described herein. [Figure 2] This flowchart illustrates how to configure and use a master bot to direct input utterances to a skill bot and, where applicable, to determine that a particular input utterance is unrelated to any available skill bot, according to some embodiments described herein. [Figure 3] This block diagram shows a master bot configured to determine whether an input utterance is unrelated to an available skill bot, according to a specific embodiment described herein. [Figure 4] This figure shows a skillbot according to a specific embodiment described herein. [Figure 5] This flowchart illustrates a method for initializing a master bot's classifier model to determine whether an input utterance is unrelated to an available skill bot, according to some embodiments described herein. [Figure 6] This figure shows the generation of training feature vectors from training utterances used to train a classifier model, according to some embodiments described herein. [Figure 7] This flowchart illustrates an example of how to determine whether an input utterance is unrelated to any available skillbot associated with the masterbot, using a masterbot classifier model according to some embodiments described herein. [Figure 8] FIG. 6 is a flow diagram illustrating another example of a method for determining whether an input utterance is unrelated to any available skill bot associated with a master bot, using a classifier model of the master bot, in accordance with some embodiments described in the present specification. [Figure 9] FIG. 9 is a diagram illustrating an example of a feature space including points representing feature vectors of exemplary utterances, in accordance with some embodiments described in the present specification. [Figure 10] FIG. 10 is a diagram illustrating an example of the feature space of FIG. 9 having class boundaries between intent classes of feature vectors of exemplary utterances, in accordance with some embodiments described in the present specification. [Figure 11] FIG. 11 is a diagram illustrating an example of the feature space of FIG. 9 having class boundaries that separate feature vectors associated with a common intent into respective clusters, in accordance with some embodiments described in the present specification. [Figure 12] FIG. 12 is a diagram illustrating an example of the feature space of FIG. 9 having overlapping class boundaries that separate feature vectors associated with a common intent into respective clusters, in accordance with some embodiments described in the present specification. [Figure 13] FIG. 13 is a diagram illustrating another example of the feature space of FIG. 9 having class boundaries that separate feature vectors into clusters, in accordance with some embodiments described in the present specification. [Figure 14] FIG. 14 is a flow diagram illustrating a method for initializing a classifier model of a master bot to determine, using clusters, whether an input utterance is unrelated to an available skill bot, in accordance with some implementations described in the present specification. [Figure 15] FIG. 15 is a diagram illustrating an implementation example of certain aspects of the method of FIG. 14, in accordance with some embodiments described in the present specification. [Figure 16] FIG. 16 is a flow diagram illustrating a method for determining whether an input utterance is unrelated to any available skill bot associated with a master bot, using a classifier model of the master bot, in accordance with some embodiments described in the present specification. [Figure 17](FIG. 12 is a diagram illustrating an example of performing the method when an input feature vector is outside the range of all cluster boundaries, in accordance with some embodiments described in this specification.) [Figure 18] FIG. 6 is a flow diagram illustrating a method for initializing a classifier model of a master bot to determine whether an input utterance is unrelated to available skill bots, using synthetic feature vectors, in accordance with some embodiments described in this specification. [Figure 19] FIG. 7 is a diagram illustrating generation of synthetic feature vectors using intent-based conversation categories, in accordance with some embodiments described in this specification. [Figure 20] FIG. 8 is a diagram illustrating generation of synthetic feature vectors using bot-based conversation categories, in accordance with some embodiments described in this specification. [Figure 21] FIG. 9 is a flow diagram illustrating a method for determining whether an input utterance is unrelated to any of the available skill bots associated with the master bot, using the classifier model of the master bot, in accordance with some embodiments described in this specification. [Figure 22] FIG. 10 is a flow diagram illustrating an example of a method for selecting a skill bot to handle an input utterance, in accordance with some embodiments described in this specification. [Figure 23] FIG. 11 is a diagram illustrating another example of a method for selecting a skill bot to handle an input utterance, in accordance with some embodiments described in this specification. [Figure 24] FIG. 13 is a diagram illustrating a distributed system for implementing some embodiments described in this specification. [Figure 25] FIG. 14 is a diagram illustrating a cloud-based system environment in which various chatbot-related services may be provided as cloud services, in accordance with some embodiments described in this specification. [Figure 26] FIG. 15 is a diagram illustrating an example of a computer system that may be used to implement some embodiments described in this specification. [Description of Embodiments]

[0017] Detailed explanation The following description is intended to provide a complete understanding of specific embodiments for illustrative purposes. Specific details are provided below. However, it will be clear that various embodiments can be carried out without these specific details. The figures and descriptions are not intended to be limiting. The term “exemplary” is used herein to mean “serving as an example, case, or illustration.” No embodiment or design described herein as “exemplary” should be construed as necessarily preferable or advantageous to other embodiments or designs.

[0018] This specification describes various embodiments, including methods, systems, and non-temporary computer-readable storage media for storing programs, code, or instructions that can be executed by one or more processors.

[0019] As described above, a particular environment includes multiple chatbots, also referred to herein as skillbots, where each chatbot is specialized to handle a specific set of tasks or skills. In this case, it would be advantageous to automatically direct user input from a user to the chatbot best suited to handle that input, and to quickly identify when the user input is irrelevant to a chatbot and therefore cannot be handled by any of the available chatbots. Several embodiments described herein allow a computing system, such as a master bot, to perform preliminary processing on user input to determine whether the user input is irrelevant to a chatbot (i.e., cannot be handled by a chatbot). As a result, computing resources are not wasted by chatbots attempting to process irrelevant user input. Thus, certain embodiments described herein conserve the computing resources of chatbots and prevent chatbots in a given environment from processing irrelevant user input by detecting irrelevant user input early. Furthermore, certain embodiments conserve network resources by preventing the master bot from sending such user input to one or more chatbots that cannot handle it.

[0020] In an environment with multiple chatbots, the master bot may include a classifier that determines which chatbot should process the input utterance. For example, the classifier could be implemented as a neural network or some other ML model that outputs a set of probabilities. Each probability is associated with a chatbot and indicates a confidence level that the chatbot can handle the input utterance. This type of master bot selects the chatbot with the highest confidence level and forwards the input utterance to that chatbot. However, the master bot may also incorrectly provide a high confidence score for irrelevant input utterances because the confidence is effectively divided among the available chatbots without any consideration of whether the chatbot is configured to handle the input utterance. This can result in a chatbot processing input utterances that it is not configured to handle. The chatbot may also ultimately provide an indication that it is unable to handle the input utterance or the user's request for clarification. However, this can occur after the chatbot has consumed resources to process the input utterance.

[0021] Alternatively, developers may train a master bot to recognize utterances that none of the available chatbots can handle. For example, a master bot may be trained with labeled training data containing both relevant utterances (i.e., those that can be handled by the chatbot) and irrelevant utterances (i.e., those that cannot be handled by the chatbot) in order to educate the master bot to recognize irrelevant utterances. However, to enable the master bot to recognize all irrelevant utterances, training data The range of input is unlikely to be sufficient. Input utterances that do not resemble any of the irrelevant utterances used during training may ultimately be forwarded to the chatbot for processing. As a result, the computational resources required to train the master bot will increase, and the master bot will still fail to recognize a wide range of irrelevant utterances.

[0022] Certain embodiments described herein address the shortcomings of the techniques described above and may be used instead of or in conjunction with such techniques. In certain embodiments, a trained machine-learning (ML) model, hereafter referred to as a classifier model, is used to determine whether an input utterance (i.e., a natural language phrase that may be in text form) provided as user input is related to any of the skillbots in the set of available skillbots. This determination may be performed, for example, by a master bot that utilizes the classifier model. The master bot generates an input feature vector from the input utterance. In one example, the master bot's classifier model compares the feature vector to a set of clusters of training feature vectors, which are feature vectors from the training data, to determine whether the input feature vector falls within the range of any of the clusters. If the input feature vector falls outside the range of all clusters, the master bot determines that the input utterance is unrelated to any skillbot. In another example, the master bot's classifier model compares the input feature vector to a set of composite feature vectors, each of which represents one or more training feature vectors belonging to their respective categories. If the input feature vector is not sufficiently similar to any of the composite feature vectors, the master bot determines that the input utterance is unrelated to any skill bot. If the input utterance is determined to be unrelated to any bot, it may be considered "none class" and will not be routed to any bot for handling. Instead, processing of the input utterance may terminate, or the master bot may prompt the user to clarify what the user intended.

[0023] This improved routing can prevent computing resources from being diverted to handling queries that may not be effectively addressed by the system in the end. This can improve the overall responsiveness of the system. In addition, the user experience can be improved because users may be instructed to rephrase or clarify queries so that they can be more easily handled by a well-trained, specialized chatbot or set of chatbots. Thus, in addition to saving processing and network resources, the average time to address user queries can be reduced by such improved routing through a bot network.

[0024] If an input utterance is determined to be related to at least one bot, the input utterance may undergo intent classification. Intent classification determines the intent that most closely matches the input utterance in order to initiate the conversational flow associated with that intent. For example, each intent of a skill bot may be associated with a state machine that defines various conversational states related to a conversation with a user. Intent classification can be performed at the individual bot level. For example, each skill bot registered with a master bot may have its own classifier (e.g., an ML-based classifier) ​​that is trained on a given utterance associated with that particular bot. The input utterance may be fed into the intent classifier of the bot most closely related to the input utterance in order to determine which of the bot's intents best matches the utterance.

[0025] This can, as a result, improve the user experience because the user's query may be better prepared to handle that user's query than a non-specialized general bot, or it may be able to handle that user's query more quickly than a non-specialized general bot. This is because it allows for quick routing to a selected skill bot or a set of selected skill bots. Improved routing can, additionally or alternatively, result in fewer computing resources being used because the processing resources consumed by a selected skill bot or a selected set of skill bots when handling user queries may be less than those consumed by a non-specialized, general-purpose bot.

[0026] Overview of an exemplary chatbot system Figure 1 is a block diagram of an environment including a master bot 114 that communicates with various skill bots 116, also referred to as chatbots, according to some embodiments described herein. This environment allows developers to use a digital assistant (DA) 106 Alternatively, it includes a digital assistant builder platform (DABP) 102 that enables the programming and deployment of a chatbot system. The DA 106 includes one or more skillbots 116, or a master bot 114 that accesses one or more skillbots 116, or accesses said master bot 114. Each skillbot 116 is configured to provide a user with one or more skills or tasks. In some embodiments, the master bot 114 and the skillbots 116 run on the DA 106 itself. However, alternatively, only the master bot 114 runs on the DA 106 and communicates with the skillbots 116 running elsewhere (for example, on another computing device).

[0027] In some embodiments, DABP102 can be used to program one or more DA106. For example, as shown in Figure 1, a developer can use DABP102 to create and deploy a digital assistant 106 for user access. For example, DABP102 may be used by a bank to create one or more digital assistants for use by its customers. The same DABP102 platform may be used by multiple companies to create digital assistants 106. As another example, a restaurant owner (e.g., a pizzeria) may use DABP102 to create and deploy a digital assistant that allows restaurant customers to order food (e.g., order a pizza). Additionally or alternatively, for example, DABP102 may be used by a developer to deploy one or more skillbots 116 so that one or more skillbots 116 can access the master bot 114 of an existing digital assistant 106.

[0028] Additionally or alternatively, in some embodiments, as further described below, the DABP 102 is configured to train the digital assistant's master bot 114 so that the master bot 114 recognizes when an input utterance is unrelated to any of the available skill bots 116.

[0029] For the purposes of this disclosure, “digital assistant” is an entity that helps users of a digital assistant accomplish various tasks through natural language conversation. A digital assistant may be implemented using software alone (for example, a digital assistant is a digital entity implemented using programs, code, or instructions executable by one or more processors), using hardware, or using a combination of hardware and software. A digital assistant may be embodied or implemented in a variety of physical systems or devices, which may include general-purpose or dedicated hardware such as computers, mobile phones, watches, electronic devices, and vehicles. A digital assistant may also be referred to as a chatbot system. Therefore, for the purposes of this disclosure, the terms digital assistant and chatbot system may be synonymous.

[0030] In some embodiments, the digital assistant 106 can be used to perform various tasks through natural language-based conversation between the digital assistant and its user 108. As part of the conversation, the user may provide a response 112 to a user input 110 and optionally provide a user input 110 to the digital assistant 106 which can perform one or more tasks related to the user input 110 (for example, if the user input includes instructions for performing a task). The conversation or dialogue may include one or more user inputs 110 and responses 112. Through the conversation, the user may request one or more tasks to be performed by the digital assistant 106, and the digital assistant 106 is configured to perform the tasks requested by the user and respond to the user with appropriate responses.

[0031] User input 110 may be in natural language form, referred to as utterances. Utterances may be in text form, such as when the user types a phrase such as a sentence, question, text fragment, or even a single word and provides the text to the digital assistant 106 as input. In some embodiments, user utterances may be in voice input or utterance form, such as when the user utters something that is provided as input to the digital assistant 106. The digital assistant 106 may include or have access to a microphone for capturing such utterances. Utterances are typically in the language spoken by the user. If the utterance is in the form of voice input, the voice input may be converted to text utterances in the same language, and the digital assistant 106 may process the text utterances as user input. Various speech-to-text processing techniques may be used to convert voice input to text utterances. In some embodiments, speech-to-text conversion is performed by the digital assistant 106 itself, but various implementations are within the scope of this disclosure. For the purposes of this disclosure, the input utterance (i.e., the utterance provided as user input) is assumed to be either a text utterance directly provided by the user 108 of the digital assistant 106, or the result of converting an input voice utterance into text format. However, this is not intended to limit or restrict to any particular form.

[0032] An utterance may be a fragment, a sentence, multiple sentences, one or more words, one or more questions, or a combination of the above types. In some embodiments, the digital assistant 106, including the corresponding master bot 114 and skill bot 116, is configured to apply natural language understanding (NLU) techniques to the utterance to understand the meaning of the user input. As part of the NLU processing for the utterance, the digital assistant 106 may perform processing to understand the meaning of the utterance, which involves identifying one or more intents and one or more entities corresponding to the utterance. Once the meaning of the utterance is understood, the digital assistant 106, including its master bot 114 and skill bot 116, may perform one or more actions or behaviors depending on the understood meaning or intent.

[0033] For example, user input may request a pizza order by providing an input utterance such as "I want to order a pizza." Upon receiving such an utterance, the digital assistant 106 determines the meaning of the utterance and takes an appropriate action. An appropriate action may include responding to the user with a question that requests user input such as the type of pizza the user wants to order, the size of the pizza, or any toppings for the pizza. The response 112 provided by the digital assistant 106 may also be in natural language form and may typically be in the same language as the input utterance. As part of generating these responses 112, the digital assistant 106 may perform natural language generation (NLG). Through a conversation with Assistant 106, the digital assistant may guide the user to provide all the information necessary to order a pizza, and furthermore, at the end of the conversation, The user may be asked to order a pizza. The digital assistant 106 may end the conversation by outputting information to the user indicating that a pizza has been ordered.

[0034] At a conceptual level, the digital assistant 106, together with its master bot 114 and associated skill bots 116, performs various processes in response to utterances received from the user. In some embodiments, this process includes a series or sequence of processing steps, such as understanding the meaning of the input utterance (using NLU), determining what action should be taken in response to the utterance, taking the action at the appropriate time, generating a response to be output to the user in response to the utterance, and outputting the response to the user. NLU processing may include parsing the received input utterance to understand its structure and meaning, and refining and improving the utterance to develop a form that is more easily parsed and understood (e.g., a logical form). Generating a response may include using NLG technology. Thus, the natural language processing performed by the digital assistant 106 may include a combination of NLU processing and NLG processing.

[0035] The NLU processing performed by the digital assistant 106 may include various NLU operations such as sentence analysis (e.g., tokenization, classification by headword, identification of part-of-speech tags, identification of named entities, generation of a dependency tree to represent sentence structure, splitting of utterances into clauses, analysis of individual clauses, decomposition of anaphora, performing chunking, etc.). In certain embodiments, the NLU processing or a portion of it is performed by the digital assistant 106 itself. Additionally or alternatively, the digital assistant 106 may use other resources to perform some parts of the NLU processing. For example, the syntax and structure of the input utterance may be identified by processing the input utterance using a parser, a part-of-speech tagging tool, or a named entity recognizer separate from the digital assistant 106.

[0036] The various examples provided in this disclosure demonstrate English utterances, but these are intended as examples only. In certain embodiments, the digital assistant 106 may, additionally or alternatively, handle utterances in languages ​​other than English. The digital assistant 106 may provide subsystems (e.g., components that implement NLU functionality) configured to perform processing for various languages. These subsystems may be implemented as pluggable units that can be invoked using service calls from an NLU core server that may run on the digital assistant 106. This makes the NLU processing flexible and extensible for each language, including enabling processing in various orders. Language packs may be provided for individual languages. Language packs may register a list of subsystems that may be provided from the NLU core server.

[0037] In some embodiments, the digital assistant 106 may be available or accessible to its user 108 through various different channels, for example, through a specific application, through a social media platform, through various messaging services and applications (e.g., an instant messaging application), or through other applications or channels. A single digital assistant may have several channels configured so that it can be run on and simultaneously accessed by different services. Additionally or alternatively, the digital assistant 106 may be implemented on a device local to the user, and for this purpose may be a personal digital assistant used by the user or other nearby users.

[0038] The digital assistant 106 may be associated with one or more skills. In certain embodiments, these skills are accessed through individual chatbots referred to as skillbots 116. Each of these is implemented and configured to interact with the user to perform a specific type of task, such as tracking inventory, submitting a time card, creating expense reports, ordering food, checking bank accounts, making reservations, purchasing widgets, or other tasks. For example, in the embodiment shown in Figure 1, the digital assistant 106 includes or accesses three skillbots 116, each performing a specific skill or set of skills. However, varying amounts of skillbots 116 may be supported by the digital assistant 106. Each skillbot 116 may be implemented as hardware, software, or a combination of both.

[0039] Each skill associated with the digital assistant 106 and thereby realized as a skillbot 116 is configured to assist the user in completing a task through a conversation with the user of the digital assistant 106. The conversation may include a combination of user input 110, either text or voice, provided by the user, and responses 112 provided by the skillbot 116 via the digital assistant 106. These responses 112 may be in the form of text or voice messages to the user, or they may be provided using a simple user interface element (e.g., a selection list) that is presented to the user for the user to make a selection.

[0040] There are various ways in which a SkillBot 116 can be associated with or added to a digital assistant. In some cases, a SkillBot 116 may be developed by an enterprise and can also be added to a digital assistant 106 using DABP 102, for example, through a user interface provided by DABP 102 to register the SkillBot 116 with the digital assistant 106. In other cases, a SkillBot 116 may be developed and created using DABP 102 and can also be added to a digital assistant 106 using DABP 102. In yet another case, DABP 102 provides an online digital store (referred to as the "Skill Store") that offers multiple skills aimed at a wide range of tasks. Skills offered through the Skill Store can also expose various cloud services. To add a skill to a digital assistant 106 using DABP 102, a developer can access the Skill Store via DABP 102, select the desired skill, and indicate that the selected skill should be added to the digital assistant 106. Skills from the Skill Store can be added to the digital assistant as is or in a modified form. DABP102, which is added to the digital assistant 106 in the form of a skillbot 116 to add skills, can configure the master bot 114 of the digital assistant 106 to communicate with the skillbot 116. In addition, DABP102 can configure the master bot 114 with skillbot data that enables the master bot 114 to recognize utterances that the skillbot 116 can use to deal with data used by the master bot 114, thereby updating the data, in order to determine whether the input utterance is unrelated to any of the skillbots 116 of the digital assistant 106. The operation for configuring the master bot 114 in this way is described in more detail below.

[0041] The skill bots 116 available to the digital assistant 106 and the skills they provide can vary considerably. For example, in the case of a digital assistant 106 developed for a business, the digital assistant's master bot 114 may interface with skill bots 116 that have specific functions, such as a CRM bot to perform functions related to customer relationship management (CRM), an ERP bot to perform functions related to enterprise resource planning (ERP), or an HCM bot to perform functions related to human capital management (HCM). A variety of other skills may also be available to the digital assistant. It is possible, and it depends on the intended use of the digital assistant 106.

[0042] Various architectures can be used to implement the digital assistant 106. In certain embodiments, the digital assistant 106 may be implemented using a master-child paradigm or architecture. According to this paradigm, the digital assistant 106 functions as a master bot 114 by including it or by accessing it, and interacts with one or more child bots which are skill bots 116. The skill bots 116 may or may not run directly on the digital assistant 106. In the example shown in Figure 1, the digital assistant 106 includes (i.e., accesses and uses) the master bot 114 and three skill bots 116. However, an excessive number of skill bots 116 may also be used, and this number may change over time as skills are added to or removed from the digital assistant 106.

[0043] The digital assistant 106, implemented according to a master-child architecture, enables the user of the digital assistant 106 to interact with multiple skillbots 116, thereby enabling the use of multiple skills that may be implemented separately through an integrated user interface, i.e., via the master bot 114. In some embodiments, when a user engages with the digital assistant 106, user input is received by the master bot 114. The master bot 114 then performs preliminary processing to determine the meaning of the input utterance that acts as user input. In this case, the input utterance may be, for example, the user input itself or a text version of the user input. The master bot 114 determines whether the input utterance is irrelevant to any skillbot 116 available for that input utterance. This may be, for example, if the input utterance requires a skill other than the skills of the skillbot 116. If the input utterance is irrelevant to the skillbot 116, the master bot 114 may return an instruction to the user that the input utterance is irrelevant to the skillbot 116. For example, the digital assistant 106 may request clarification from the user or report that the input utterance was not understood. However, if the master bot 114 identifies the appropriate skill bot 116, the master bot 114 may route the input utterance and the conversation in progress to that skill bot 116. This allows the user to interact with the digital assistant 106, which has multiple skill bots 116, through a common interface.

[0044] The embodiment shown in Figure 1 illustrates a digital assistant 106 including a master bot 114 and a skill bot 116, but this is not intended to be limiting. In some embodiments, the digital assistant 106 may include various other components, such as other systems or subsystems that provide the functionality of the digital assistant 106. These systems and subsystems may be implemented in software only (for example, as code stored on a computer-readable medium and executable by one or more processors), in hardware, or in implementations using a combination of software and hardware.

[0045] In certain embodiments, the master bot 114 is configured to recognize available skill bots 116. For example, the master bot 114 may have access to metadata that identifies various available skill bots 116 and the capabilities of each skill bot 116, including tasks that can be performed by that skill bot 116. Upon receiving a user request in the form of an input utterance, the master bot 114 is configured to identify or predict from among several available skill bots 116 which is best suited to or best able to address the user request. Alternatively, in the alternative example, the system is configured to determine that the input utterance is unrelated to any of the skillbots 116. If it is determined that a skillbot 116 can handle the input utterance, the masterbot 114 may route the input utterance or at least a portion of it to the skillbot 116 for further handling. Thus, control proceeds from the masterbot 114 to the skillbot 116.

[0046] In some embodiments, DABP102 provides infrastructure and various services and features that enable DABP102 developer users to create a digital assistant 106 that includes one or more skillbots 116. In some cases, a skillbot 116 can be created by cloning an existing skillbot 116, for example, by cloning a skillbot 116 provided in a skill store. As described above, DABP102 can provide a skill store that provides multiple skillbots 116 for performing various tasks. Users of DABP102 can clone a skillbot 116 from the skill store, and the cloned skillbot 116 may be modified or customized as needed. In some other cases, DABP102 developer users create a skillbot 116 from scratch, for example, by using the tools and services provided by DABP102.

[0047] In certain embodiments, creating or customizing the SkillBot 116 at a high level involves the following actions:

[0048] (1) Configure the settings for the new skill bot. (2) Configure one or more intents for SkillBot.

[0049] (3) Constitute one or more entities for one or more intents. (4) Train the skillbot.

[0050] (5) Create a dialogue flow for the skill bot. (6) Add custom components to the skill bot as needed.

[0051] (7) Test and deploy the skillbot. The following is a brief explanation of each of the actions described above.

[0052] (1) Configure the settings for the new skillbot 116. Various settings can be configured for the skillbot 116. For example, a skillbot developer can specify one or more invocation names for the skillbot 116 being created. These invocation names, which serve as identifiers for the skillbot 116, can also be used by the user of the digital assistant 106 to explicitly refer to the skillbot 116. For example, a user can include an invocation name in their input utterance to explicitly refer to the corresponding skillbot 116.

[0053] (2) Constitute one or more intents and associated exemplary utterances for SkillBot 116. The designer of SkillBot 116 specifies one or more intents, also called chatbot intents, for the SkillBot 116 being created. SkillBot 116 is then trained based on these specified intents. These intents represent the categories or classes for which SkillBot 116 is trained to infer about input utterances. Upon receiving an utterance, the trained SkillBot 116 infers (i.e., determines) an intent about the utterance. In this case, the inferred intent is used to train SkillBot 116 A predefined set of intents is selected. Skillbot 116 then takes the appropriate action for the utterance based on the intent inferred in relation to the utterance. In some cases, the intents for Skillbot 116 represent tasks that Skillbot 116 can perform on behalf of the digital assistant user. Each intent is given an intent identifier or intent name. For example, for Skillbot 116 trained on banking, intents specified for Skillbot 116 might include "CheckBalance," "TransferMoney," and "DepositCheck."

[0054] For each intent defined for SkillBot 116, the designer of SkillBot 116 may also provide one or more exemplary utterances that represent and illustrate the intent. These exemplary utterances are intended to represent utterances that a user may input to SkillBot 116 about that intent. For example, for the CheckBalance intent, exemplary utterances might be, "What's the balance of my savings account?" and "How much is in my checking account?" This could include phrases like "How much is in my checking account?" or "How much money do I have in my account?". Therefore, various rearrangements of typical user utterances can be specified as utterance examples for intents.

[0055] Intents and their associated exemplary utterances, also referred to as training utterances, are used as training data to train SkillBot 116. Various different training techniques may be used. As a result of this training, a predictive model is produced that takes an utterance as input and outputs an inferred intent for that utterance. In some cases, an input utterance provided as user input is fed into an intent analysis engine (for example, a rule-based or ML-based classifier run by SkillBot 116) which is configured to predict or infer an intent about the input utterance using the trained model. SkillBot 116 can then take one or more actions based on the inferred intent.

[0056] (3) Configure entities for one or more intents of SkillBot 116. In some cases, additional context may be needed to enable SkillBot 116 to respond appropriately to input utterances. For example, there may be situations where multiple input utterances result in the same intent for SkillBot 116. For instance, the utterances "What is the balance of my savings account?" and "How much is in my checking account?" both result in the same CheckBalance intent, but these utterances are different requests asking for different things. To clarify such requests, one or more entities can be added to the intent. An entity called Account_Type, which defines values ​​referred to as "checking" and "savings," can enable SkillBot 116 to parse user requests and respond appropriately, using the example of the banking SkillBot 116. In the example above, multiple utterances result in the same intent, but the values ​​associated with the Account_Type entity are different for these two utterances. This allows SkillBot 116 to perform different actions for these two utterances, even though they result in the same intent. One or more entities may be specified for a particular intent configured for SkillBot 116. Therefore, entities are used to add context to the intent itself. Entities help to better describe the intent and enable SkillBot 116 to fulfill user requests.

[0057] In certain embodiments, (a) an embedded entity that may be provided by DABP102 There are two types of entities: (i) and (2) custom entities that can be specified by the developer. Embedded entities are general-purpose entities that can be used with a wide variety of skillbots 116. Examples of embedded entities include entities related to time, date, address, number, email address, duration, cycle period, currency, telephone number, uniform resource locator (URL), etc. Custom entities are used for more customized purposes. For example, in the case of a banking skill, the Account_Type entity may be defined by the developer to support various banking transactions by checking user input for keywords such as checking, savings, and credit card.

[0058] (4) Train the SkillBot 116. The SkillBot 116 is configured to receive user input in the form of utterances, to parse or process the received user input, and to identify or select intents related to the received user input. In some embodiments, the SkillBot 116 must be trained for this purpose, as described above. In certain embodiments, the SkillBot 116 is trained on intents configured for the SkillBot 116 and exemplary utterances associated with those intents (i.e., training utterances), so that the SkillBot 116 can decompose an input utterance into one of its configured intents. In certain embodiments, the SkillBot is enabled to identify what the user is saying (or, in some cases, what the user is trying to say) using a predictive model trained with the training data. DABP 102 may provide a variety of different training techniques that developers can use to train the SkillBot 116, including various ML-based training techniques, rule-based training techniques, or combinations thereof. In certain embodiments, a portion of the training data (e.g., 80%) is used to train the skillbot model, and another portion (e.g., the remaining 20%) is used to test or validate the model. Once trained, the trained model, also referred to as the trained skillbot 116, can be used to address and respond to input utterances. In certain cases, the input utterance may be a question that requires only a single answer and does not require any further conversation. To address such situations, a question-and-answer (Q&A) intent may be defined for the skillbot 116. In some embodiments, Q&A While intents are created in a similar manner to other intents, the interaction flow for Q&A intents can differ from that for regular intents. For example, the interaction flow for a Q&A intent may not require prompting the user for additional information (e.g., values ​​about a specific entity), unlike with other intents.

[0059] (5) Create a dialogue flow for SkillBot 116. The dialogue flow specified for SkillBot 116 describes how SkillBot 116 will respond when various intents about SkillBot 116 are broken down in response to the received user input 110. The dialogue flow defines the actions or behaviors that SkillBot 116 will take, such as how SkillBot 116 responds to user utterances, how SkillBot 116 prompts the user for input, and how SkillBot 116 returns data. The dialogue flow is similar to a flowchart that SkillBot 116 will follow. The designer of SkillBot 116 specifies the dialogue flow using a language such as Markdown. In certain embodiments, a version of YAML called OBotML may be used to specify the dialogue flow for SkillBot 116. The dialogue flow definition for SkillBot 116 serves as a model for the conversation itself, allowing the designer of SkillBot 116 to choreograph the dialogue between SkillBot 116 and the user it corresponds to.

[0060] In a particular embodiment, the dialogue flow definition for the skillbot 116 includes the following three sections:

[0061] (a) Context section (b) Default transition section (c) State section Context Section: In the Context Section, SkillBot 116 developers can define variables used in the conversation flow. Other variables that may be designated in the Context Section include, for example, variables for error handling, variables for built-in or custom entities, and user variables that allow SkillBot 116 to recognize and maintain user preferences.

[0062] Default Transition Section: Transitions for SkillBot 116 can be defined in the Dialogue Flow State Section or the Default Transition Section. Transitions defined in the Default Transition Section act as fallbacks and are triggered when there are no applicable transitions defined within a given state, or when the conditions necessary to trigger a state transition are not met. The Default Transition Section can be used to define routing that allows SkillBot 116 to handle unexpected user actions effectively.

[0063] State Section: The dialogue flow and its associated behavior are defined as a set of transient states that govern the logic within the dialogue flow. Each state node in the dialogue flow definition designates a component that provides the functionality required at that point in the dialogue. In this way, states are built around the components. The states contain component-specific characteristics and define transitions to other states that are triggered after the component has been executed.

[0064] Special case scenarios can be addressed using state sections. For example, it may be desirable to give the user the option to temporarily leave the first skill in which the user is involved and do something with a second skill within the digital assistant 106. For instance, if the user is engaged in a conversation with a shopping skill (e.g., the user has made some selection for a purchase), the user may want to jump to a banking skill (e.g., want to make sure the user has enough money for the purchase) and then return to the shopping skill to complete the user's order. To address this, the state section within the dialogue flow definition of the first skill may be configured to initiate a conversation with a second different skill within the same digital assistant and then return to the original dialogue flow.

[0065] (6) Adding custom components to SkillBot 116: As described above, the states specified in the dialogue flow for SkillBot 116 designate components that provide the necessary functionality corresponding to those states. Components enable SkillBot 116 to perform functions. In certain embodiments, DABP 102 provides a set of pre-configured components for performing a wide range of functions. Developers can select one or more of these pre-configured components and associate them with states in the dialogue flow for SkillBot 116. Developers can also use the tools provided by DABP 102 to create custom or new components and associate the custom components with one or more states in the dialogue flow for SkillBot 116.

[0066] (7) Test and deploy SkillBot 116: DABP102 was developed by the developer It may offer several features that allow testing of the SkillBot 116 inside. Furthermore, the SkillBot 116 can be deployed and included in the Digital Assistant 106.

[0067] While the above describes how to create a skillbot 116, similar techniques may be used to create a digital assistant 106 or a masterbot 114. At the masterbot level or digital assistant level, embedded system intents may be configured for the digital assistant 106. In some embodiments, these embedded system intents are used to identify common tasks that the masterbot 114 can handle without calling a skillbot 116. Examples of system intents defined for the masterbot 114 include: (1) Exit: when the user wants to end the current conversation or context with the digital assistant; (2) Help: when the user is seeking assistance or direction; (3) UnresolvedIntent: when user input does not fit well with the Exit intent or the Help intent. The masterbot 114 may store information about one or more skillbots 116 associated with the digital assistant 106. This information allows the master bot 114 to select a specific skill bot 116 to address the utterance, or alternatively, to determine that the utterance is unrelated to any of the skill bots 116 of the digital assistant 106.

[0068] When a user inputs a phrase or utterance into the digital assistant 106, the master bot 114 is configured to perform a process that determines how to route the utterance and the associated conversation. The master bot 114 makes this determination using a routing model that may be rule-based, ML-based, or a combination of both. Using the routing model, the master bot 114 determines whether the conversation corresponding to the utterance should be routed to a specific skill bot 116 for handling, handled by the digital assistant 106 or the master bot 114 itself on a per-built-system intent basis, handled as a different state in the current conversation flow, or unrelated to any of the skill bots 116 associated with the digital assistant 106.

[0069] In certain embodiments, as part of this process, the master bot 114 determines whether the input utterance explicitly identifies the skill bot 116 using its invocation name. If an invocation name is present in the input utterance, the invocation name is treated as an explicit invocation of the skill bot 116 corresponding to the invocation name. In such scenarios, the master bot 114 may route the input utterance to the explicitly invocationed skill bot 116 for further action. If there is no specific invocation or explicit invocation, in certain embodiments, the master bot 114 evaluates the input utterance and calculates confidence scores for the system intent and skill bot 116 associated with the digital assistant 106 (for example, using a logistic regression model). The scores calculated for the skill bot 116 or system intent represent the likelihood that the input utterance represents a task or system intent configured for the skill bot 116 to perform. Any system intent or skill bot 116 whose associated calculated confidence score exceeds a threshold may be selected as a candidate for further evaluation. Next, the master bot 114 selects a specific system intent or skill bot 116 from the identified candidates to further address the input utterance. In a particular embodiment, after one or more skill bots 116 are identified as candidates, the intents associated with those candidate skill bots 116 are evaluated for each skill bot 116, for example, by using a trained model, and a confidence score is determined for each intent. Generally, confidence scores exceeding a threshold (e.g., 70%) are... Any intent containing "A" is treated as a candidate intent. If a specific skillbot 116 is selected, the input utterance is routed to that skillbot 116 for further processing. If a system intent is selected, one or more actions are performed by the masterbot 114 itself according to the selected system intent.

[0070] Some embodiments of the master bot 114 described herein are configured, where applicable, to not only direct input utterances to the appropriate skill bot 116, but also to determine whether to prompt some indication that an irrelevant input utterance is irrelevant when the input utterance is irrelevant to an available skill bot 116. Figure 2 is a flowchart illustrating a method 200 of configuring and using a master bot to direct input utterances to a skill bot 116 and, where applicable, to determine that a particular input utterance is irrelevant to an available skill bot 116, according to some embodiments described herein. The method 200 in Figure 2 is an overview of various specific cases, which are described in detail below.

[0071] Method 200 shown in Figure 2 and other methods described herein can be implemented in software (e.g., as code, instructions, or programs) executed by one or more processing units (e.g., processors or processor cores), in hardware, or in a combination thereof. The software may be stored in a non-temporary storage medium such as a memory device. Method 200 is intended to be illustrative and non-limiting. Figure 2 shows various operations occurring in a particular order or sequence, but this is not intended to be limiting. In particular embodiments, for example, these operations may be performed in a different order, or one or more operations of Method 200 may be performed in parallel. In particular embodiments, Method 200 may be performed by a training system and a master bot 114.

[0072] As shown in Figure 2, in block 205 of method 200, the classifier model of the master bot 114 is initialized. In some embodiments, the classifier model is configured to determine whether the input utterance is unrelated to any of the available skill bots 116. In addition, in some embodiments, the classifier model may also be configured to route the input utterance to a suitable skill bot 116 for processing if the input utterance is deemed to be related to a skill bot 116.

[0073] In some embodiments, the initialization of a classifier model may include training the classifier model on how to recognize irrelevant input utterances. For this purpose, a training system, which may be part of DABP102, may have access to training utterances (i.e., exemplary utterances) for each skillbot 116. The training system may generate respective training feature vectors that describe and represent each training utterance of various skillbots 116 that are available to or may become available to the master bot 114. Each training feature vector is the feature vector of the corresponding training utterance. The training system may then generate various set representations from the training feature vectors. Each set representation may represent a set of training feature vectors. For example, such a set may be a cluster or composite feature vector, as will be described in detail below. To initialize the classifier model, the training system may configure the classifier model to compare input vectors with various set representations that represent sets of training feature vectors.

[0074] In block 210, the master bot 114 receives user input 110 in the form of input utterance. For example, the user may have typed the input utterance as user input 110, or the user may have spoken user input 110 which the digital assistant 106 converted into input utterance. The input utterance was converted by the digital assistant 106, however This may include user requests that should be handled by master bot 114.

[0075] In block 215, the master bot 114 uses a classifier model to determine whether any of the skill bots 116 associated with the master bot 114 can handle the input utterance. In some embodiments, the classifier model of the master bot 114 generates an input feature vector that describes and represents the input utterance. The classifier model may compare the input feature vector to a set representation of the training feature vector, and based on this comparison, the classifier model determines whether the input utterance is irrelevant to any of the skill bots 116 available to the master bot 114 (i.e., the skill bot 116 in which the training utterance is represented in the set representation).

[0076] Examples of classification at the digital assistant or skillbot level Figure 3 is a block diagram showing a master bot 114, also referred to as a master bot (MB) system, according to a particular embodiment described herein. The master bot 114 can be implemented in software only, in hardware only, or in a combination of hardware and software. In some embodiments, the master bot 114 includes a preprocessing subsystem 310 and a routing subsystem 320. The master bot 114 shown in Figure 3 is merely one example of the arrangement of components within the master bot 114. Those skilled in the art will recognize many possible variations, alternatives, and modifications. For example, in some implementations, the master bot 114 may have more or fewer systems or components than those shown in Figure 3, may be a combination of two or more subsystems, or may have different configurations or arrangements of subsystems.

[0077] In some embodiments, the language processing subsystem 310 is configured to process user input 110 provided by the user. Such processing may include, for example, converting the user input 110 into a text input utterance 303 using automatic speech recognition or some other tool if the user input is in some form other than speech or text.

[0078] In some embodiments, the routing subsystem 320 is configured to determine (a) whether the input utterance 303 is unrelated to any of the available skillbots 116, and (b) if the input utterance 303 is related to at least one skillbot 116, which skillbot 116 is best suited to address the input utterance 303. In particular, the classifier model 324 of the routing subsystem 320 may be rule-based, ML-based, or a combination thereof, and may be configured to determine whether the input utterance 303 is unrelated to any of the available skillbots 116, represents a specific skillbot 116, or represents a specific intent configured for a specific skillbot 116. For example, as described above, the skillbot 116 may consist of one or more chatbot intents. Each chatbot intent may have its own dialogue flow and may be associated with one or more tasks that the skillbot 116 can perform. If it is determined that the input utterance 303 represents a specific skillbot 116 or an intent configured for a specific skillbot 116, the routing subsystem 320 can invoke the specific skillbot 116 and transmit the input utterance 303 as input 335 for that specific skillbot 116. However, if the input utterance 303 is deemed to be unrelated to any of the available skillbots 116, the input utterance 303 is considered to belong to a non-class 316, which is a class of utterances that cannot be handled by any of the available skillbots. In that case, the digital assistant 106 may indicate to the user that it cannot handle the input utterance 303.

[0079] In some embodiments, the classifier model 324 may be implemented using a rule-based model, an ML-based model, or both. For example, in some embodiments, the classifier model 324 may include a rule-based model trained on training data 354 containing exemplary utterances. Thus, the rule-based model is used to determine whether the input utterance is unrelated to any of the available skillbots 116 (i.e., any skillbot 116 associated with the digital assistant 106). Additionally or alternatively, in some embodiments, the classifier model 324 may include a neural network trained on the training data 354. The training data 354 may include a corresponding set of exemplary utterances for each skillbot 116 (for example, two or more exemplary utterances for each intent in which the skillbot 116 is configured). For example, in the example in Figure 3, the available skillbots 116 include a first skillbot 116a, a second skillbot 116b, and a third skillbot 116c, and the training data 354 includes the first skillbot data 358a corresponding to the first skillbot 116a, the second skillbot data 358b corresponding to the first skillbot 116b, and the third skillbot data 358c corresponding to the first skillbot 116c, with each such skillbot 116 having its own skillbot data. The skillbot data for skillbot 116 may include training utterances (i.e., exemplary utterances) that represent the utterances that skillbot 116 can handle. Thus, the training data 354 includes training utterances for each of the different skillbots 116. The training system 350 may be incorporated into DABP 102, but is not required. Alternatively, the training data 354 may be used to train the classifier model 324 of the master bot 114 and perform these tasks.

[0080] In some embodiments, the training system 350 trains a classifier model 324 using training data 354 to determine whether an input utterance is unrelated to any of the available skillbots 116. Generally, the training system 350 may generate training feature vectors to describe and represent the training utterances in the training data 354. As described in detail below, the training system 350 may generate set representations, each representing a set of training feature vectors. After training and during operation, the classifier model 324 may compare an input utterance 303 to a set representation to determine whether the input utterance 303 is unrelated to any of the available skillbots 116.

[0081] In some embodiments, the classifier model 324 includes one or more submodels, each submodel configured to perform different tasks related to classifying the input utterance 303. In addition to, or instead of, the classifier model 324 may include, for example, an ML model or other type of model configured to determine which skillbot 116 is best suited to the input utterance if the input utterance 303 is considered to be related to (i.e., not unrelated to) at least one available skillbot 116. In some embodiments, the classifier model 324 may determine that the input utterance is related to at least one skillbot 116, while determining the most suitable skillbot 116. In this case, no further determination is needed to identify the best-suited skillbot 116. However, alternatively, the classifier model 324 may determine, for each skillbot 116, a relevant confidence score (for example, using a logistic regression model) indicating the likelihood that the skillbot 116 is best suited to processing the input utterance 303 (i.e., best suited to the input utterance). For this purpose, the neural network included in the classifier model 324 can be trained by a training system to determine the likelihood that the input utterance 303 represents each skillbot 116 or one of the intents configured for the skillbot 116. The neural network may then determine and output a confidence score associated with each skillbot 116. In this case, the confidence score for a skillbot indicates the likelihood that skillbot 116 can handle the input utterance 303, or the likelihood that it is the best available skillbot 116 to handle the input utterance. Taking the confidence scores into consideration, the routing subsystem 320 may select the skillbot 116 associated with the highest confidence score and route the input utterance 303 to that skillbot 116 for processing.

[0082] Figure 4 shows a SkillBot 116, also referred to as a SkillBot system, according to a particular embodiment described herein. An instance of SkillBot 116 shown in Figure 4 can be used as SkillBot 116 in Figure 1 and can be implemented in software only, in hardware only, or in a combination of hardware and software. As shown in Figure 4, SkillBot 116 may include a bot classifier model 424 that determines the intent of an input utterance 303 and a conversation manager 430 that generates a response 435 based on the intent.

[0083] The bot classifier model 424 can be implemented using a rule-based model, an ML-based model, or both, and can take as input the input utterance 303 routed to the skill bot 116 by the master bot 114. The bot classifier model 424 may have access to rules 452 and intent data 454 in the data store 450 on the skill bot 116, or it may have access to the skill bot 116. For example, the intent data 454 may include exemplary utterances or other data for each intent, and rule 452 may describe how to use the intent data to determine the intent for the input utterance 303. The bot classifier model 424 may apply these rules to the input utterance 303 and the intent data 454 to determine the intent for the input utterance 303.

[0084] More specifically, in some embodiments, the bot classifier model 424 may operate in a similar manner to how the classifier model 324 in Figure 3 determines which skill bot 116 to address the input utterance 303. For example, the bot classifier model 424 may determine each confidence level (for example, using a logistic regression model) and assign such confidence levels to each chatbot intent that constitutes the skill bot 116 in a similar manner to how the classifier model 324 may assign confidence levels to the skill bot 116. Thus, the confidence levels indicate the highest likelihood that each chatbot intent is applicable to the input utterance 303. The bot classifier model 424 may then select the intent to which the highest confidence level has been assigned. In additional or alternative embodiments, the bot classifier model 424 determines intents for an input utterance 303 by comparing the input feature vector describing the input utterance 303 with either or both of the following: (a) a composite feature vector containing one composite feature vector to represent a training feature vector for each intent, or (b) clusters of training feature vectors. In this case, each cluster represents an intent or multiple intents. The use of such feature vectors is described in more detail below.

[0085] Once the intent that best represents utterance 202 is identified, the bot classifier model 424 may transmit an intent instruction 422 (i.e., an instruction for the identified intent) to the conversation manager 430. In the embodiment of Figure 4, the conversation manager 430 is shown as local to skillbot 116. However, the conversation manager 430 may be shared with the master bot 114 or across multiple skillbots 116. Thus, in some embodiments, the conversation manager 430 may be shared with the digital assistant as well. This is local to masterbot 114.

[0086] In response to receiving an intent instruction 422, the conversation manager 430 may determine an appropriate response 435 for the utterance 202. For example, the response 435 may be an action or message specified in a dialogue flow definition 455 configured for the system 400 of the skillbot 116, and the response 435 may be used as the DA response 112 in the embodiment of Figure 1. For example, the data store 450 may contain various dialogue flow definitions, each containing its own dialogue flow definition for each intent, and the conversation manager 430 may access the dialogue flow definition 455 based on the identification of the intent corresponding to that dialogue flow definition 455. Based on the dialogue flow definition 455, the conversation manager 430 may determine a certain dialogue flow state as the next state to transition to, in accordance with the dialogue flow definition 455. The conversation manager 430 may determine the response 435 based on yet another processing of the input utterance 303. For example, if the input utterance 303 is "Check balance in savings", the conversation manager 430 may transition to a conversation flow state in which the user is presented with a dialogue related to the user's savings account. The conversation manager 430 may transition to this state based on intent instruction 422 indicating that the identified intent is the "CheckBalance" intent configured for the skillbot 116, and further based on the recognition that the value "saving" has been extracted for the "Account_Type" entity.

[0087] Using feature vectors to describe utterances As described above, the classifier model 324 may use feature vectors when determining whether an input utterance is unrelated to or related to any of the available skillbots 116. For the purposes of this disclosure, a feature vector is a set of vectors or coordinates that describe the features of an utterance and thereby describe the utterance itself. A feature vector describing an utterance may be used to represent an utterance in a particular context, as described herein.

[0088] The concept of feature vectors is based on the concept of word embeddings. Generally, word embeddings are a type of language modeling where words are mapped to corresponding vectors. A particular word embedding can map semantically similar words to similar regions in a vector space. As a result, similar words will be close to each other in the vector space, while dissimilar words will be far apart. A simple example of word embeddings is "one-hot" coding. In this case, each word in the dictionary is a quantity with dimensions equal to the size of the dictionary. It is mapped to a vector that has a value of 1 in the dimension corresponding to the word itself, and a value of zero in all other dimensions. For example, the first two words of the sentence "Go intelligent bot service artificial intelligence, Oracle" can be represented using the following "one-hot" coding.

[0089] [Table 1]

[0090] Feature vectors can be used to represent words, sentences, or various types of phrases. Considering the simple example of word embeddings above, the corresponding feature vector maps a set of words, such as an utterance, to a feature vector which is a set of word embeddings for that set of words. It is possible. That set could be, for example, the sum, the average, or the weighted average.

[0091] In the case of utterances that are different from each other, their feature vectors may also be different because they contain words with significantly different semantic meanings. However, utterances that are semantically similar and therefore contain the same or semantically similar words between them may have similar feature vectors (i.e., feature vectors that are located close to each other in the vector space). Each feature vector corresponds to a single point in the vector space, also called the feature space, which is the origin of the feature space plus the feature vector. In some embodiments, points for semantically similar utterances are located close to each other. Throughout this disclosure, feature vectors and their corresponding points are referred to interchangeably because they provide different visual appearances with respect to the same information.

[0092] An exemplary method for initializing the masterbot classifier model. In some embodiments, the classifier model 324 of the master bot 114 utilizes the feature vectors of the training utterance as a criterion for determining whether an input utterance is unrelated to or related to any of the available skill bots 116. Figure 5 shows a method 500 for initializing the classifier model 324 of the master bot 114 to perform this task, according to some embodiments described herein. For example, this method 500 or a similar method may be performed in block 205 of method 200 for configuring and using the master bot 114 to route an input utterance 303. Figure 5 is a schematic method 500, and more detailed examples thereof are illustrated and described with reference to Figures 14 and 18.

[0093] The method 500 shown in Figure 5, and other methods described herein, can be implemented in software (e.g., as code, instructions, or programs) executed by one or more processing units (e.g., processors or processor cores), in hardware, or in a combination thereof. The software may be stored in a non-temporary storage medium such as a memory device. The method 500 is intended to be illustrative and non-limiting. Figure 5 shows various operations that occur in a particular order or sequence, but this is not intended to limit them. In particular embodiments, for example, these operations may be performed in a different order, or one or more operations of the method 500 may be performed in parallel. In particular embodiments, the method 500 may be performed by a training system 350, which may be part of the DABP 102.

[0094] In block 505, the training system 350 accesses training utterances (also referred to as exemplary utterances) for various skillbots 116 associated with the masterbot 114. For example, training utterances may be stored as skillbot data in training data 354 accessible to the training system 350. In some embodiments, each skillbot 116 available to the masterbot 114 may be associated with a subset of training utterances, each such subset of training utterances may contain a training utterance for each intent in which the skillbot 116 is composed. Thus, a set of training utterances may contain a training utterance for each intent of each skillbot 116, so as to represent every intent of any skillbot 116.

[0095] In block 510, the training system 350 generates training feature vectors from the training utterances accessed in block 505. As described above, the training utterances may include a subset associated with each skillbot 116, and may also include training utterances for each intent of each skillbot 116. Therefore, in some embodiments, the training feature vectors may include a subset for each skillbot 116, and further, a feature vector for each intent of each skillbot.

[0096] Figure 6 illustrates the generation of training feature vectors 620 from training utterances 615 according to several embodiments described herein. Specifically, Figure 6 relates to training utterances 615 of skillbot data 358 within training data 354, where skillbot data 358 is associated with a particular skillbot 116. Skillbot 116 is configured to handle input utterances 303 with multiple intents, including intent A and intent B. For this reason, training utterances 615 include training utterances 615 representing intent A and training utterances 615 representing intent B.

[0097] As described above, the training system 350 may generate training feature vectors 620 to describe and represent each training utterance 615. Various techniques are known for converting a set of words, such as training utterances 615, into feature vectors, such as training feature vectors 620, and one or more such techniques may be used. For example, the training system 350 may, but does not necessarily, encode each training utterance 615 as a corresponding training feature vector 620 using one-hot coding or some other coding method.

[0098] Furthermore, as mentioned above, feature vectors can be represented as points. In some embodiments, each training feature vector 620 can be represented as a point 640 in the feature space 630. In this case, the feature space has a number of dimensions equal to the number of features (i.e., the number of dimensions) in the training feature vector 620. In the example in Figure 6, two training feature vectors 620 representing the same intent, specifically intent A, are plotted as points 640 that are close to each other because these two training feature vectors 620 are semantically similar. However, it is not necessary for all training feature vectors 620 or all feature vectors with respect to a particular intent to be represented as points that are close to each other.

[0099] Returning to Figure 5, in block 515, the training system 350 generates multiple set representations of the training feature vectors 620 generated in block 510. Each set representation represents a set of training feature vectors 620. As will be described in detail below, a set representation can be, for example, a cluster of training feature vectors 620, or a composite feature vector which is a set of multiple training feature vectors 620. Essentially, a set representation can be a representation of multiple training feature vectors 620 that are grouped together. Each set of training feature vectors represented in a corresponding set representation may share a common intent, a group of common intents, a common skillbot 116, or a common region of the feature space 630, or alternatively, the training feature vectors represented in a single set representation do not need to have any commonalities other than being based on the training data 354.

[0100] In block 520, the training system 350 may configure a classifier model 324 to compare an input utterance 303, provided as user input 110, with various set representations. For example, set representations may be stored in a memory accessible to the classifier model 324 of the master bot 114. The classifier model 324 may be programmed with rules on how to determine whether an input utterance matches or does not match various set representations. The definition of a match may depend on the specific set representation being used, as will be described in more detail below.

[0101] Figure 7 shows a method 700, according to some embodiments described herein, that uses a classifier model 324 of a master bot 114 to determine whether an input utterance 303 provided as user input 110 is unrelated to any of the available skill bots 116 associated with the master bot 114. This method 700 or a similar method may be performed after the initialization of the classifier model 324, and further for each input utterance 303 received. For example, this method 700 or a similar method may be performed in block 215 of method 200 for configuring and using the master bot 114 to route input utterances 303. Figure 7 is a schematic example of method 700, and more detailed examples thereof are illustrated and described with reference to Figures 16 and 21.

[0102] The method 700 shown in Figure 7 and other methods described herein may be implemented in software (e.g., as code, instructions, or programs) executed by one or more processing units (e.g., processors or processor cores), in hardware, or in a combination thereof. The software may be stored in a non-temporary storage medium such as a memory device. The method 700 is intended to be illustrative and non-limiting. Figure 7 shows various operations occurring in a particular order or sequence, but this is not intended to be limiting. In particular embodiments, for example, these operations may be performed in a different order, or one or more operations of the method 700 may be performed in parallel. In particular embodiments, the method 700 may be performed by a master bot 114.

[0103] In block 705 of method 700, the master bot 114 accesses the input utterance 303 provided as user input 110. For example, in some embodiments, the user may provide user input 110 in the form of voice input, and the digital assistant 106 may convert the user input 110 into a text input utterance 303 for use by the master bot 114.

[0104] In block 710, the master bot 114 may generate an input feature vector from the input utterance 303 accessed in block 705. The input feature vector can describe and represent the input utterance 303. Various techniques are known for converting a set of words, such as an input utterance, into a feature vector, and one or more of these techniques may be used. For example, the training system 350 may, but does not necessarily, encode the input utterance as a corresponding input feature vector using one-hot coding or some other coding method. However, one embodiment of the master bot 114 uses the same technique used to generate the training feature vector 620 from the training utterance when training the classifier model 324.

[0105] In the decision block 715, the master bot 114 may have the classifier model 324 compare the input feature vector generated in block 710 with the set representation determined in block 515 of method 500 for initializing the classifier model 324, so that the master bot 114 can determine whether the input feature vector matches any of the skill bots 116 available to the master bot 114. The specific techniques for comparison and matching may depend on the nature of the set representation. For example, if the set representation is a cluster of training feature vectors 620, as described below, the classifier model 324 may determine whether the input feature vector falls within one of the clusters and therefore matches at least one skill bot 116 by comparing the input feature vector (i.e., the points representing the input feature vector) with the clusters, or if the set representation is a composite vector of training feature vectors, the classifier model 324 may determine whether the input feature vector is sufficiently similar to any of those composite feature vectors and therefore matches at least one skill It is possible to determine whether it matches Bot 116. Various implementations are possible and fall within the scope of this disclosure.

[0106] In the determination block 715, if the input feature vector is deemed to match at least one skillbot 116, in block 720, the master bot 114 may route the input utterance to the skillbot 116 that is deemed to match the input feature vector. However, in the determination block 715, if the input feature vector is deemed not to match any skillbot 116, in block 725, the master bot 114 may indicate that the utterance cannot be processed by any skillbot 116. This instruction may be passed to a digital assistant, which may provide the user with an output indicating that the user input 110 cannot be processed or addressed by the digital assistant.

[0107] Figure 8 shows another example of a method 800 that uses a classifier model 324 of a master bot 114 to determine whether an input utterance 303 provided as user input 110 is unrelated to any available skill bot 116 associated with the master bot 114, according to some embodiments described herein. This method 800 or a similar method may be performed for each received input utterance 303 after the initialization of the classifier model 324. For example, this method 800 or a similar method may be performed in block 215 of method 200 for configuring and using the master bot 114 to route input utterances 303. Similar to method 700 in Figure 7, method 800 in Figure 8 is a schematic method 800, and more detailed examples of several method blocks thereof are illustrated and described with reference to Figures 16 and 21. However, in contrast to method 700 in Figure 7, method 800 illustrates preliminary filtering operations in decision blocks 810 and 815, which can be used to ensure that certain input utterances 303 that are similar to the training utterance 615 are not classified as belonging to a non-class 316. In other words, method 800 includes a filter that filters out any input utterances 303 that are considered sufficiently similar to the training utterance 615 from non-class considerations, thereby ensuring that such input utterances 303 are routed to skillbot 116.

[0108] The method 800 shown in Figure 8 and other methods described herein may be implemented in software (e.g., as code, instructions, or programs) executed by one or more processing units (e.g., processors or processor cores), in hardware, or in a combination thereof. The software may be stored in a non-temporary storage medium such as a memory device. The method 800 is intended to be illustrative and non-limiting. Figure 8 shows various operations that occur in a particular order or sequence, but this is not intended to limit them. In particular embodiments, for example, these operations may be performed in a different order, or one or more operations of the method 800 may be performed in parallel. In particular embodiments, the method 800 may be performed by a master bot 114.

[0109] In block 805 of method 800, the master bot 114 accesses the input utterance 303 provided as user input 110. For example, in some embodiments, the user may provide user input 110 in the form of voice input, and the digital assistant 106 may convert the user input 110 into a text input utterance 303 for use by the master bot 114.

[0110] In the determination block 810, the master bot 114 instructs the classifier model 324 to determine whether all words from the input utterance 303 accessed in block 805, or a predetermined percentage of the words, can be found in the training utterance 615. For example, in some embodiments, the classifier model 324 utilizes a Bloom filter based on the words of the training utterances 615, and this Bloom filter is used. This applies to force speech 303.

[0111] If the input utterance 303 contains only words found in the training utterance 615 in the determination block 810, the master bot 114 may determine that the input utterance relates to at least one skill bot 116 and therefore does not belong to a non-class 316. In that case, in block 815, the master bot 114 may route the input utterance 303 to the skill bot 116 that best matches the input utterance 303. For example, as described above, the classifier model 324 may be configured to assign a confidence score to each skill bot 116 for the input utterance (for example, using a logistic regression model). The master bot 114 may select the skill bot 116 with the highest confidence score and route the input utterance 303 to that skill bot 116.

[0112] However, if in the determination block 810 the input utterance 303 contains any word not present in the training utterance 615, or contains a percentage of words greater than a threshold, method 800 proceeds to block 820. In block 820, the master bot 114 may generate an input feature vector from the input utterance 303 accessed in block 805. The input feature vector can describe and represent the input utterance 303. Various techniques are known for converting a set of words, such as an input utterance, into a feature vector, and one or more such techniques may be used. For example, the training system 350 may, but does not necessarily, encode each training utterance 615 as a corresponding training feature vector 620 using one-hot coding or some other coding method. However, one embodiment of the master bot 114 uses the same technique used to generate the training feature vector 620 from the training utterance.

[0113] In decision block 825, the master bot 114 may have the classifier model 324 compare the input feature vector generated in block 820 with the set representation determined in block 515 of method 500 for initializing the classifier model 324, thereby enabling the master bot 114 to determine whether the input feature vector matches any of the skill bots 116 available to the master bot 114. Specific techniques for comparison and matching may depend on the nature of the set representation. For example, if the set representation is a cluster of training feature vectors 620, as described below, the classifier model 324 may determine whether the input feature vector falls within one of the clusters and therefore matches at least one skill bot 116 by comparing the input feature vector (i.e., the points representing the input feature vector) with the clusters; or, if the set representation is a composite vector of training feature vectors, the classifier model 324 may determine whether the input feature vector is sufficiently similar to any such composite feature vector and therefore matches at least one skill bot 116 by comparing the input feature vector with the composite feature vector. Various implementation examples are feasible and fall within the scope of this disclosure.

[0114] In determination block 825, if the input feature vector is deemed to match at least one skillbot 116, in block 830, the master bot 114 may route the input utterance to the skillbot 116 that is deemed to match the input feature vector. However, in determination block 825, if the input feature vector is deemed not to match any skillbot 116, in block 835, the master bot 114 may indicate that the utterance cannot be processed by any skillbot 116. This instruction may be passed to a digital assistant, which may indicate that the user input 110 cannot be processed or addressed by the digital assistant. It is acceptable to provide power to the user.

[0115] Examples of cluster types available by classifier models As described above, an exemplary type of set representation available to the classifier model 324 is a cluster of training feature vectors 620. Generally, these can be dealt with by available skillbots 116, and for this reason, it can be assumed that input utterances related to available skillbots 116 have some semantic similarity to the training utterances 615 relating to those skillbots 116. Thus, input utterances 303 relating to available skillbots 116 can likely be represented as input feature vectors adjacent to one or more training utterances 615 in the feature space 630.

[0116] Considering the proximity of feature vectors of semantically similar utterances, a boundary can be defined to demarcate feature vectors plotted at points with a common intent, or to separate feature vectors with different intents. In two-dimensional space, the boundary can be a line or a circle, so that points on one side of the line belong to a first intent class (i.e., corresponding to utterances with the first intent), and points on the other side of the line belong to a second intent class. In three dimensions, the boundary can be represented as a plane or a sphere. More broadly, in various dimensions, the boundary can be a hyperplane, a hypersphere, or a hypervolume. The boundary can have a variety of shapes and does not need to be perfectly spherical or symmetrical.

[0117] Figure 9 shows an example of a feature space 630 containing points representing feature vectors of exemplary utterances, according to some embodiments described herein. In this example, some of the exemplary utterances belong to a balance class and represent a first intent related to requests for account balance information, while the rest of the exemplary utterances belong to a transaction class and represent a second intent related to requests for transaction information. In Figure 9, exemplary utterances in the balance class are labeled b, and exemplary utterances in the transaction class are labeled t. These intent classes may be defined, for example, with respect to a financial-related skillbot 116.

[0118] For example, the utterance examples in the table below may belong to the balance class in the first column and the transaction class in the second column.

[0119] [Table 2]

[0120] Figure 10 shows an example of the feature space 630 of Figure 9 having class boundaries 1010 between intent classes of exemplary utterance feature vectors, according to some embodiments described herein. Specifically, as shown in Figure 10, lines may be drawn as class boundaries 1010 to separate points in the balance class (i.e., feature vectors of exemplary utterances) from points in the transaction class. This class boundary 1010 is a rough approximation of the division between the two intent classes. To create a more precise boundary 1010 between intent classes, it may be necessary to provide circles or other geometric volumes that define clusters of feature vectors such that each cluster contains only feature vectors in a single corresponding intent class (i.e., has intents associated with that intent class).

[0121] Figure 11 shows the common intents related to several embodiments described herein. Figure 9 shows an example of a feature space 630 having class boundaries 1010 that separate, specifically isolate, the attached feature vectors into their respective clusters. In this example, as in some embodiments, not all feature vectors of a given intent class are in a single cluster, nor are there any clusters containing feature vectors from two or more intent classes. Specifically, in the illustrated example, the first cluster 1110a, defined by the first class boundary 1010, contains only feature vectors within the balance class, and the second cluster 1110b, defined by the second class boundary 1010, contains only feature vectors within the transaction class. As will be described in detail below, some embodiments described herein can form class boundaries 1010 to create clusters as shown in Figure 11.

[0122] Figure 12 shows another example of the feature space 630 in Figure 9 having class boundaries 1010 that separate, specifically isolate, feature vectors associated with common intents into their respective clusters, according to some embodiments described herein. In this example, as in some embodiments, not all feature vectors of an intent class are in a single cluster, and there are no clusters containing feature vectors from two or more intent classes. Specifically, in the illustrated example, the first cluster 1110c, defined by the first class boundary 1010, contains only feature vectors within the balance class, and the second cluster 1110d, defined by the second class boundary 1010, contains only feature vectors within the transaction class. However, in contrast to the example in Figure 11, the class boundaries 1010 and, for this reason, the clusters overlap. Some embodiments described herein support overlapping clusters as shown in Figure 12. As will be described in detail below, some embodiments described herein can form class boundaries 1010 to create clusters as shown in Figure 12.

[0123] Figure 13 shows another example of the feature space 630 in Figure 9 having class boundaries 1010 that separate feature vectors into clusters, according to some embodiments described herein. In this example, as in some embodiments, not all feature vectors of a certain intent class are in a single cluster, and furthermore, a certain cluster may represent various intents by including feature vectors with different intents. Specifically, in the illustrated example, the first cluster 1110e defined by the first class boundary 1010 contains only feature vectors in the balance class, and the second cluster 1110f defined by the second class boundary 1010 contains some feature vectors in the balance class (i.e., having balance-related intents) and some feature vectors in the transaction class (i.e., having transaction-related intents). Some embodiments described herein support clusters with various intents, as shown in Figure 13. As will be described in detail below, some embodiments described herein can form class boundaries 1010 to create clusters as shown in Figure 13.

[0124] Clustering to identify irrelevant input utterances Figure 14 shows a method 1400 for determining whether an input utterance is unrelated to or related to an available skillbot 116 by initializing a classifier model 324 of a masterbot 114 and utilizing clusters of training feature vectors, according to several embodiments described herein. This method 1400 or a similar method can be used in block 205 of the above-described method 200, which configures and uses the masterbot 114 to direct an input utterance 303 to a skillbot 116. Furthermore, method 1400 in Figure 14 is a more specific modification of method 500 in Figure 5. In some embodiments of this method, k-means clustering is performed to configure the classifier model 324, as described below. k-means clustering may allow for more accurate cluster formation compared to other clustering techniques such as k nearest neighbors, but various clustering techniques may be used instead of or in addition to k-means clustering.

[0125] Method 1400 shown in Figure 14 and other methods described herein may be implemented in software (e.g., as code, instructions, or programs) executed by one or more processing units (e.g., processors or processor cores), in hardware, or in a combination thereof. The software may be stored in a non-temporary storage medium such as a memory device. Method 1400 is intended to be illustrative and non-limiting. Figure 14 shows various operations occurring in a particular order or sequence, but this is not intended to be limiting. In particular embodiments, for example, these operations may be performed in a different order, or one or more operations of Method 1400 may be performed in parallel. In particular embodiments, Method 1400 may be performed by a training system 350, which may be part of DABP 102.

[0126] In block 1405 of Method 1400, the training system 350 accesses training utterances 615, also referred to as exemplary utterances, for various skillbots 116 associated with the master bot 114. For example, the training utterances 615 may be stored as skillbot data in training data 354 accessible to the training system 350. In some embodiments, each skillbot 116 available to the master bot 114 may be associated with a subset of training utterances 615, each such subset of training utterances 615 may contain a training utterance 615 for each intent in which the skillbot 116 is composed. Thus, a set of training utterances 615 may contain a training utterance 615 for each intent of each skillbot 116, so that every intent of any skillbot 116 is represented.

[0127] Figure 15 shows an example of the execution of a phase of Method 1400 of Figure 14, according to several embodiments described herein. As shown in Figure 15, the training system 350 may access training data 354. The training data 354 may include skillbot data relating to skillbots 116 available to the master bot 114 to which the classifier model 324 being trained belongs. In this example, the training data 354 accessed by the training system 350 may include training utterances 615 from first skillbot data 358d, second skillbot data 358e, and third skillbot data 358f, each of which may include a training utterance 615 representing each skillbot 116 available to the master bot 114. Furthermore, for a given set of skillbot data, each training utterance 615 may be associated with an intent that the associated skillbot 116 is configured to evaluate and address, as shown with respect to the first skillbot data 358d in Figure 15.

[0128] In block 1410 of Figure 14, the training system 350 generates a training feature vector 620 from the training utterances 615 accessed in block 1405. As described above, the training utterances 615 may include a subset associated with each skillbot 116, and further, a training utterance 615 may be included for each intent of each skillbot 116. Therefore, in some embodiments, the training feature vector 620 may include a subset for each skillbot 116, and further, a feature vector may be included for each intent of each skillbot. As described above, the training system 350 can generate the training feature vector 620 to describe and represent each training utterance 615. Various techniques are known for converting a set of words such as training utterances 615 into feature vectors such as the training feature vector 620, and one or more such techniques may be used. For example, the training system 350 may use one-hot coding or any other method. Using this encoding, each training utterance 615 can be encoded as a corresponding training feature vector 620, but encoding is not always necessary.

[0129] As shown in Figure 15, for example, each training utterance 615 from various skillbot data 358 can be converted into its respective training feature vector 620. Thus, in some embodiments, the resulting training feature vector 620 may represent all skillbots 116 to which the training utterance 615 was provided, and all intents to which the training utterance 615 was provided.

[0130] In block 1415 of Figure 14, the training system 350 may set (i.e., initialize) a count, which is the amount of clusters to be generated. In some embodiments, for example, the count may initially be set to a quantity n, where n is the total number of intents across the various skill bots 116 available to the master bot 114. However, various other quantities can be used as the initial value of the count.

[0131] In some embodiments, the training system 350 performs the step of iteratively determining a set of clusters of the training feature vector 620 one, two, or more times. This method 1400 utilizes two steps to determine the clusters, but in some embodiments, one or more steps may be used. In block 1420, the training system 350 begins the first step of determining the clusters. The first iteration utilizes an iterative loop. In this iterative loop, the training system 350 generates clusters of the training feature vector 620 and then determines whether there are enough such clusters. If the clusters are deemed sufficient, as described below, the training system 350 may terminate the first iteration.

[0132] In some embodiments, block 1425 is the beginning of the first iteration loop for determining clusters. Specifically, in block 1425, the training system 3509 may determine the centroid locations (i.e., the location for each centroid) for each of the various clusters to be generated in this iteration. The number of centroid locations is equal to the count determined in block 1415. In some embodiments, in the first iteration of this loop, the training system 350 may select a randomly chosen set of centroid locations in the feature space 630 containing the training utterance 615, having a quantity equal to the count. More specifically, for example, the training system 350 may determine a bounding box, such as a minimum bounding box, with respect to the points corresponding to the training feature vector 620. The training system 350 may then randomly select a set of centroid locations within that bounding box. In this case, the number of selected locations is equal to the count determined with respect to the clusters. During iterations other than the first, the count has increased from the previous iteration, and therefore, in some embodiments, only the centroid locations of newly added centroids are randomly determined. The centroids carried over from the previous iteration can retain their positions. Each centroid for a corresponding cluster can be positioned according to its respective centroid position.

[0133] In block 1430, the training system 350 can determine clusters by assigning each training feature vector 620 to the nearest centroid among the various centroids whose positions were determined in block 1425. For example, for each training feature vector 620, the training system 350 can calculate the distance to each centroid position and assign that training feature vector 620 to the centroid having the smallest such distance. A set of training feature vectors 620 assigned to a common centroid can together form a cluster associated with that centroid. Thus, there may be a number of clusters equal to the number of centroids, which is the count value.

[0134] In the example in Figure 15, only a portion of the feature space 630 is shown, within which the first centroid 1510a and the second centroid 1510b are visible. The training system 350 determined that of the three illustrated training feature vectors 620, two were closest to the first centroid 1510a and one was closest to the second centroid 1510b. In this example, the two training feature vectors 620 closest to the first centroid 1510a constitute the first cluster 1110g, and the one training feature vector 620 closest to the second centroid 1510b constitutes the second cluster 1110h.

[0135] In block 1435 of Figure 14, with respect to the clusters determined in block 1430, the training system 350 recalculates the position of the centroid of each cluster. For example, in some embodiments, the centroid of each cluster is calculated to be the average (e.g., arithmetic mean) of the training feature vectors 620 assigned to that centroid and therefore to that cluster.

[0136] In the decision block 1440, the training system 350 may determine whether a stopping condition is met. In some embodiments, the training system 350 may use this method 1400 or a similar method to repeatedly increase the number of centroids (i.e., increase the count) and assign the training feature vectors 620 to their nearest related centroids until convergence occurs to such an extent that significant improvement in clustering is no longer possible. In some embodiments, the stopping condition defines a sufficient level of convergence.

[0137] In some embodiments, the stopping condition may be met if one or both of the following conditions are true: (1) the average cluster cost satisfies a first threshold, for example, the average cluster cost is less than 1.5 or less than some other predetermined value; or (2) the outlier ratio satisfies a second threshold, for example, the outlier ratio is less than or equal to 0.25 or less than or equal to some other predetermined value. The cluster cost for a particular cluster may be defined as (a) the sum of the squared distances between the cluster centroid recalculated in block 1435 and each training feature vector 620 assigned to that centroid, divided by the amount of training feature vectors 620 assigned to that centroid; and (b) the average cluster cost may be the average of the various cluster costs of various centroids. The outlier ratio may be defined as the total number of outliers between clusters divided by the number of clusters (i.e., count). There are various techniques for defining outliers, and one or more such techniques may be used by the training system 350. In some embodiments, the stopping condition is met if (1) the average cluster cost satisfies a first threshold (e.g., less than 1.5) and (2) the outlier ratio satisfies a second threshold (e.g., less than or equal to 0.25), and only if such conditions are met.

[0138] Generally, the average cluster cost tends to decrease as the value of k (i.e., the count) increases, while the outlier ratio tends to increase as k increases. In some embodiments, when the above stopping conditions that consider both factors are applied, the maximum k value that satisfies the respective thresholds for both the cluster count and the outlier ratio becomes the final k value for classifier model 324.

[0139] If the termination condition is not met in the determination block 1440, method 1400 may proceed to block 1445. In block 1445, the training system increments the count for the next clustering iteration. In some embodiments, the count may be incrementally increased by an amount that increases the likelihood that the termination condition will be met after a certain number of loop iterations or less. For example, in some embodiments, the count is

[0140]

number

[0141] The step size can be increased by a value equal to n, where n is the total number of intents across all available skillbots 116, and u is the total number of training utterances 615 across all available skillbots 116. This step size will ensure that the stopping condition is eventually met. Specifically, for this step size, starting with a count equal to n, the 20th iteration will have a count that is greater than or equal to u, which is the number of utterances. If the count is greater than or equal to u, each utterance may have its own cluster, or if not, there is still a chance that the average cluster cost will be less than 1.5 and the outlier ratio will be less than or equal to 0.25, which satisfies the exemplary stopping condition. More schematically, the step size may be chosen to ensure that the above iterations do not waste computing resources by looping over an unreasonable amount of time. After the count is updated in block 1445, method 1400 returns to block 1425 and performs another loop iteration.

[0142] However, if the termination condition is met in the determination block 1440, the method 1400 may terminate the current iteration loop and proceed to block 1450. In block 1450, the training system 350 starts the second clustering. In this case, the clusters may be further defined based on the work performed in the first iteration.

[0143] In some embodiments, block 1455 is the beginning of the second iteration loop to determine the clusters. Specifically, in block 1455, the training system 3509 may determine the respective centroid positions (i.e., each position for each centroid) for the various clusters to be generated in this iteration. The quantity of centroid positions is equal to the current value of the count. In some embodiments, in the first iteration of this loop, the training system 350 uses the centroid positions as recalculated in block 1435 before the end of the first iteration. In iterations other than the first, the count has increased from the previous iteration. In this case, the centroids from the previous iteration may retain their centroid positions, and in some embodiments, the centroid positions of newly added centroids due to the incrementing count may be determined randomly. Each centroid for the corresponding cluster may be positioned according to each centroid position.

[0144] In block 1460, the training system 350 can determine clusters by assigning each training feature vector 620 to the nearest centroid among the various centroids whose positions were determined in block 1455. For example, for each training feature vector 620, the training system 350 can calculate the distance to each centroid position and assign that training feature vector 620 to the centroid having the smallest such distance. A set of training feature vectors 620 assigned to a common centroid can together form clusters associated with that centroid. Thus, there may be a number of clusters equal to the number of centroids, which is the count value.

[0145] In block 1465, with respect to the clusters determined in block 1460, the training system 350 recalculates the position of the centroid of each cluster. For example, in some embodiments, the centroid of each cluster is calculated to be the average (e.g., arithmetic mean) of the training feature vectors 620 assigned to that centroid and therefore to that cluster.

[0146] In judgment block 1470, the training system 350 is stopped when the stop condition is met. It is possible to determine whether or not this is the case. In some embodiments, the training system 350 may use this method 1400 or a similar method to repeatedly increase the number of centroids (i.e., increase the count) and assign the training feature vectors 620 to their nearest related centroids until convergence occurs to such an extent that significant improvement in clustering is no longer possible. In some embodiments, the stopping condition defines a sufficient level of convergence.

[0147] In some embodiments, the stopping condition may be met if one or both of the following conditions are true: (1) the average cluster cost satisfies a first threshold, for example, the average cluster cost is less than 1.5 or less than some other predetermined value; or (2) the outlier ratio satisfies a second threshold, for example, the outlier ratio is less than or equal to 0.25 or less than or equal to some other predetermined value. The cluster cost for a particular cluster may be defined as (a) the sum of the squared distances between the cluster centroid recalculated in block 1465 and each training feature vector 620 assigned to that centroid, divided by the amount of training feature vectors 620 assigned to that centroid; and (b) the average cluster cost may be the average of the various cluster costs of various centroids. The outlier ratio may be defined as the total number of outliers between clusters divided by the number of clusters (i.e., count). There are various techniques for defining outliers, and one or more such techniques may be used by the training system 350. In some embodiments, the stopping condition is met if (1) the average cluster cost satisfies a first threshold (e.g., less than 1.5) and (2) the outlier ratio satisfies a second threshold (e.g., less than or equal to 0.25), and only if such conditions are met.

[0148] If the termination condition is not met in determination block 1470, method 1400 may proceed to block 1475. In block 1475, the training system increments the count for the next clustering iteration. In some embodiments, the count may be incrementally increased by an amount that increases the likelihood that the termination condition will be met after a certain number of loop iterations or less. For example, in some embodiments, the count is

[0149]

number

[0150] The step size can be increased by a value equal to . This step size ensures that the termination condition is eventually met. Specifically, with this step size, the termination condition is likely to be met by the end of the fifth iteration. More broadly, the step size may be chosen to ensure that the above iteration does not waste computing resources by looping over an unreasonable amount of time. After the count is updated in block 1475, method 1400 returns to block 1455 and executes another loop iteration.

[0151] However, if the termination condition is met in decision block 1470, method 1400 may terminate the current iteration loop and proceed to block 1480. In block 1480, the training system 350 determines a boundary 1010 for each cluster determined above. In some embodiments, the boundary 1010 for a cluster is defined to be centered on the centroid of the cluster and to include all the training feature vectors 620 assigned to the cluster. In some embodiments, for example, the cluster boundary 1010 is a hypersphere (e.g., a circle or sphere) centered on the centroid. In some embodiments, the radius of the boundary 1010 is the margin value (i.e., padding amount) plus the larger of (1) the maximum distance from the center to the training feature vector 620 in the cluster furthest from the centroid, or (2) the training within the cluster. The radius can be the sum of the average distances from the feature vector 620 to the centroid, plus three times the standard deviation of these distances. That is, the radius can be set to radius = margin + max(max(distances), mean(distances) + 3σ(distances)). In this case, Distance is the set of distances from the cluster's training feature vector to the cluster's centroid, max(distances) is the maximum value of that set, and mean(distances) is the set of distances. The mean is the average of the set, and σ (distances) is the standard deviation of that set. Furthermore, the margin value may be the margin of error and can have a value of zero or greater.

[0152] In some embodiments, margins may be used to define a boundary 1010 that encompasses a wider scope than otherwise possible, reducing the likelihood that the relevant input utterance 303 lies outside the scope of all clusters and is therefore labeled as a member of a non-class 316. In other words, margins may fill the boundary 1010. For example, margins

[0153]

number

[0154] The value may be such that u is the total number of training utterances 615 used. The margin can take the following situations into consideration. Specifically, situations where the number of training utterances is potentially too low (e.g., 20-30) so that the training feature vector 620 does not cover a significant portion of the feature space 630, and as a result the clusters are too small to capture the relevant input utterances 303.

[0155] Returning to the example in Figure 15, the training system 350 determines its respective boundary for each cluster determined based on the assignment of the training feature vectors 620. Specifically, in this example, the first boundary 1010a is determined for the first cluster 1110g, which contains two training feature vectors 620, and the second boundary 1010b is determined for the second cluster 1110h, which contains one training feature vector 620. The number of training feature vectors can be small, as in this example, or large, such as the number of training feature vectors 620 per cluster. It will be understood that this simplified example is non-limiting and provided for illustrative purposes only.

[0156] As shown in Figure 14, in block 1485, the training system 350 may configure the master bot 114's classifier model 324 to utilize the boundary 1010, also called the cluster boundary, determined in block 1480. For example, the training system 350 may store the cluster boundary instructions in a memory device accessible to the classifier model (for example, on the digital assistant 106). The classifier model 324 may be configured to compare the input utterance with such a cluster boundary 1010, which serves as a set representation of the training feature vectors 620.

[0157] Various modifications can be made to the method 1400 described above, and these modifications are within the scope of this disclosure. For example, some examples of the training system 350 perform cluster improvement only once. In this case, the operations from block 1445 to block 1475 may be skipped so that method 1400 proceeds to block 1480 when the stop condition is met in decision block 1440. Some other examples of the training system 350 perform cluster improvement more than two times. In this case, the operations from block 1445 to block 1475 may be repeated each time an additional run is added after the second run. These and other implementations are within the scope of this disclosure.

[0158] In some embodiments, the k value (i.e., the count value and thus the number of clusters) determined in the method 1400 described above depends on the total number of training utterances 615 and the distribution of training feature vectors 620 across the feature space 630. Thus, the k value may vary for each master bot 114 depending on the available skillbots 116 and the training utterances 615 available to represent the intents of those skillbots 116. The optimal k value is a balanced value such that each cluster is large enough so that input utterances 303 related to each cluster fall within the boundaries 1010 of that cluster, while unrelated utterances fall outside the boundaries 1010. If the clusters are too large, the risk of false matches increases. If the clusters are too small (e.g., consisting of a single training utterance 615), the classifier model 324 may be overfitted, thus limiting the usefulness of such clusters.

[0159] In the above example of Method 1400, the training feature vectors 620 are not split or grouped based on intents, and therefore a given cluster may contain training feature vectors 620 representing different skillbots 116 or different intents of one or more skillbots 116. Additionally or alternatively, one embodiment of the training system 350 may ensure that each cluster contains only training feature vectors 620 representing a single skillbot 116, a single intent, or a single subbot. In this case, the subbot is associated with a subset of training utterances representing a single skillbot 116. Various techniques can be used to restrict clusters in this way. For example, the training utterances may be separated into groups based on intents, subbots, or skillbots 116, and each instance of Method 1400 above may be performed for each group. In this way, a cluster determined in one instance of method 1400 for a corresponding group (for example, a training utterance 615 representing a particular skillbot 116) may contain only the training feature vectors 620 from that corresponding group. This may be limited to a single intent, subbot, or training utterance of skillbot 116. Various other implementations are possible and are within the scope of this disclosure.

[0160] Figure 16 shows a method 1600 that uses a classifier model 324 of a master bot 114, according to some embodiments described herein, to determine whether an input utterance 303 provided as user input 110 is unrelated to any of the available skill bots 116 associated with the master bot 114. This method 1600 or a similar method can be used in block 215 of the above-described method 200, which configures and uses the master bot 114 to direct the input utterance 303 to a skill bot 116. The method 1600 in Figure 16 is a more specific modification of the method 700 in Figure 7, and, like the method 700 in Figure 7, the method 1600 in Figure 16 can be used in conjunction with the preliminary filtering operation described with respect to the method 800 in Figure 8. More specifically, as described below, the classifier model 324 of the master bot 114 may utilize clusters of training feature vectors 620 to determine whether the input utterance 303 belongs to a non-class 316 (i.e., unrelated to any of the available skill bots 116).

[0161] Method 1600 shown in Figure 16 and other methods described herein may be implemented in software (e.g., as code, instructions, or programs) executed by one or more processing units (e.g., processors or processor cores), in hardware, or in a combination thereof. The software may be stored in a non-temporary storage medium such as a memory device. Method 1600 is intended to be illustrative and non-limiting. Figure 16 shows various operations occurring in a particular order or sequence, but this is not intended to be limiting. In particular embodiments, for example, these operations may be performed in a different order, or one or more operations of Method 1600 may be performed in parallel. In certain embodiments, method 1600 may be performed by a master bot 114 associated with a set of available skill bots 116.

[0162] In block 1605 of method 1600, the master bot 114 accesses the input utterance 303 provided as user input 110. For example, in some embodiments, the user may provide user input 110 in the form of voice input, and the digital assistant 106 may convert the user input 110 into a text input utterance 303 for use by the master bot 114.

[0163] In block 1610, the masterbot 114 may generate an input feature vector from the input utterance 303 accessed in block 1605. More specifically, in some embodiments, the masterbot 114 causes its classifier model 324 to generate an input feature vector from the input utterance 303. The input feature vector can describe and represent the input utterance 303. Various techniques are known for converting a set of words, such as an input utterance, into a feature vector, and one or more such techniques may be used. For example, the training system 350 may, but does not necessarily, encode the input utterance 303 as the corresponding input feature vector using one-hot coding or some other coding method. However, one embodiment of the masterbot 114 uses the same technique used to generate the training feature vector 620 from the training utterance 615 when training the classifier model 324.

[0164] In the decision block 1615, the master bot 114 compares the input feature vectors generated in block 1610 with clusters. More specifically, in some embodiments, the master bot 114 causes the classifier model 324 to compare the input feature vectors generated in block 1610 with clusters determined during training, specifically the cluster boundaries. As described above, each cluster may include a set of training feature vectors 620 and a boundary 1010 based on the training feature vectors 620 of that cluster. For example, the boundary 1010 includes all the training feature vectors 620 assigned to the cluster and, optionally, some additional space outside those training feature vectors 620. Specifically, in some embodiments, the classifier model 324 determines whether the input feature vectors (i.e., points corresponding to the input feature vectors) are within any boundary 1010 of any cluster of the training feature vectors 620. Various techniques exist in the art for determining whether a point lies within a boundary, and one or more such techniques can be used to determine whether an input feature vector lies within any of the cluster boundaries 1010.

[0165] In decision block 1620, the classifier model 324 makes a decision based on comparing the input feature vector with the cluster boundaries. If the input feature vector is not within any cluster boundary and is therefore outside the range of all cluster boundaries 1010, method 1600 proceeds to block 1625.

[0166] Figure 17 shows an example of performing this method 1600 when the input feature vector 1710 is outside the range of all cluster boundaries, according to some embodiments described herein. In some embodiments, the master bot 114 provides the input utterance 303 to the classifier model 324, thereby causing the classifier model 324 to convert the input utterance 303 into the input feature vector 1710 and compare the input feature vector with the cluster boundaries 1010. In the example in Figure 17, five clusters 1110 are shown in the feature space 630. However, more or fewer clusters 1110 may be used. In this example, the input feature vector 1710 is outside the range of all cluster boundaries 1010, and therefore the classifier model 324 instructs the master bot that the input utterance 303 belongs to a non-class 316. Output to Bot 114.

[0167] Returning to Figure 16, in block 1625, the master bot 114 indicates that it cannot process (i.e., cannot process further) the input utterance 303, based on the classifier model 324 showing that the input feature vectors are outside the range of all cluster boundaries 1010. For example, the digital assistant 106 may respond to the user by requesting clarification or by reporting that the user input 110 is not relevant to the digital assistant's skills.

[0168] However, if the input feature vector is within the range of one or more cluster boundaries 1010, method 1600 skips to block 1630. In block 1630, the master bot 114 determines which skill bot 116 will deal with the input utterance 303 (i.e., to further process the input utterance 303 and determine the response 435 to the input utterance 303) by selecting one of the available skill bots 116. The determination of the skill bot 116 can be performed in various ways, and the technique used may depend on the configuration of the one or more cluster boundaries 1010 in which the input feature vector is contained.

[0169] To select a skillbot 116 for the input utterance 303, the masterbot 114 may consider various training utterances represented by training feature vectors 620 that share clusters with the input feature vector 1710 (i.e., training feature vectors that are members of clusters 1110 within the boundary 1010 in which the input feature vector 1710 resides). For example, if the training feature vector 620 resides in two or more overlapping clusters 1110, then training utterances 615 having the corresponding training feature vector 1710 in any of those two or more clusters 1110 may be considered. Similarly, if the training feature vector 620 resides in only one cluster 1110, then training utterances 615 having the corresponding training feature vector 620 in that cluster 1110 are considered. If all the training utterances 615 being considered represent a single skillbot 116, then, in some cases, for example, if the input feature vector 1710 falls within a cluster 1110 containing training feature vectors 620 associated with only a single skillbot 116, then the master bot 114 may select that skillbot 116 to address the input utterance 303.

[0170] In some embodiments, the classifier model 324 may be able to identify a specific intent for a particular skillbot 116 to address an input utterance. For example, if one or more clusters 1110 containing training feature vectors 620 contain only the training feature vectors 620 of training utterances 615 that represent a single intent for a single skillbot 116, the classifier model 324 may identify that the particular intent is applicable to the input utterance 303. If the classifier model 324 can identify a specific intent for a particular skillbot 116, the master bot 114 may route the input utterance 303 to that skillbot 116 and may also indicate that intent to the skillbot 116. As a result, the skillbot 116 may omit performing its own classification of the input utterance 303 to infer the intent, but rather may infer the intent indicated by the master bot 114.

[0171] However, if the training utterance 615 under consideration represents multiple skillbots 116, for example, if the input feature vector 1710 falls within a cluster 1110 composed of training feature vectors 620 representing multiple skillbots 116, or if the input feature vector 1710 falls within multiple overlapping clusters 1110, the master bot 114 may need to further classify the input utterance 303 to select a skillbot 116. Classification is also possible. In some embodiments, the classifier model 324 implements a machine learning mode to calculate a confidence score (for example, using a logistic regression model) for each skillbot 116 that has associated training feature vectors 620 within one or more clusters 1110 containing the input feature vectors 1710. For example, the machine learning model may be trained using synthetic feature vectors. The master bot 114 may then select the skillbot 116 with the highest confidence score to address the input utterance 303. In contrast to the conventional use of confidence scores to identify relevant skillbots 116, in some embodiments, it has already been determined that the input utterance 303 is related to a skillbot 116. Thus, the risk of routing the input utterance 303 to an irrelevant skillbot 116 is reduced or eliminated.

[0172] In additional or alternative embodiments, the classifier model 324 may utilize the k-nearest neighbor technique to select a skillbot 116 from among two or more skillbots 116 to which relevant training feature vectors 620 that share one or more clusters 1110 with the input feature vector 1710 belong. For example, the classifier model 324 may select a value for k and identify the k-nearest neighbor training feature vectors 620 for the input feature vector 1710 from among the training feature vectors 620 that fall within one or more clusters 1110 of the input feature vector 1710. The classifier model 324 may identify the skillbot 116 that has the largest number of relevant training feature vectors 620 in its set of k-nearest neighbor training feature vectors 620. The master bot 114 may select its skillbot 116 to address the input utterance. Various other implementations for selecting a skillbot 116 are possible and within the scope of this disclosure.

[0173] In block 1635, the master bot 114 may forward the input utterance 303 to the skill bot selected in block 1630. The skill bot 116 may then process the input utterance 303 and respond to the user input 110.

[0174] Using composite vectors to identify irrelevant input utterances As described above, generally, some embodiments described herein utilize a set representation of the training feature vector 620 to determine whether the input utterance 303 relates to a set of available skillbots 116. Also, as described above, the set representation may be a cluster 1110. However, additionally or alternatively, the set representation may be a higher-level feature vector, referred to herein as a composite feature vector.

[0175] Figure 18 shows a method 1800 that uses a composite feature vector to initialize a classifier model 324 of a master bot 114 to determine whether an input utterance 303 is unrelated to or related to an available skill bot 116, according to some embodiments described herein. This method 1800 or a similar method can be used in block 205 of the above-described method 200, which configures and uses the master bot 114 to direct the input utterance 303 to a skill bot 116. Furthermore, method 1800 in Figure 18 is a more specific modification of method 500 in Figure 5.

[0176] Method 1800 shown in Figure 18 and other methods described herein can be implemented in software (e.g., as code, instructions, or programs) executed by one or more processing units (e.g., a processor or processor core), in hardware, or in a combination thereof. The software may be stored in a non-temporary storage medium such as a memory device. Method 1800 is intended to be illustrative and non-limiting. Figure 18 shows various operations that occur in a particular order or sequence, but this is not intended to be limiting. In certain embodiments, for example, these operations may occur in a different order. Alternatively, one or more operations of method 1800 may be performed in parallel. In certain embodiments, method 1800 may be performed by a training system 350, which may be part of DABP 102.

[0177] In block 1805 of Method 1800, the training system 350 accesses training utterances 615 (also referred to as exemplary utterances) for various skillbots 116 associated with the masterbot 114. For example, the training utterances 615 may be stored as skillbot data in training data 354 accessible to the training system 350. In some embodiments, each skillbot 116 available to the masterbot 114 may be associated with a subset of training utterances 615, each such subset of training utterances 615 may contain a training utterance 615 for each intent in which the skillbot 116 is composed. Thus, a set of training utterances 615 may contain a training utterance 615 for each intent of each skillbot 116, so that every intent of any skillbot 116 is represented.

[0178] In block 1810, the training system 350 generates a training feature vector 620 from the training utterances 615 accessed in block 1805. As described above, the training utterances 615 may include a subset associated with each skillbot 116, and may also include a training utterance 615 for each intent of each skillbot 116. Therefore, in some embodiments, the training feature vector 620 may include a subset for each skillbot 116, and may also include a feature vector for each intent of each skillbot. As described above, the training system 350 may generate a training feature vector 620 to describe and represent each training utterance 615. Various techniques are known for converting a set of words, such as training utterances 615, into feature vectors, such as training feature vectors 620, and one or more such techniques may be used. For example, the training system 350 may, but does not necessarily, encode each training utterance 615 as the corresponding training feature vector 620 using one-hot coding or some other coding method.

[0179] In block 1815, the training system 350 divides the training feature vector 620 and, therefore, the corresponding training utterance 615 into conversation categories. Conversation categories can be defined, for example, based on intents, subbots, or skillbots 116. In some embodiments, when conversation categories are based on intents, each conversation category includes a training feature vector 620 representing a single corresponding intent on which a skillbot 116 is comprised. In this case, the number of conversation categories may be equal to the number of intents across the various skillbots 116 available to the master bot 114. In some embodiments, when conversation categories are based on skillbots 116, each conversation category includes a training feature vector 620 representing a single corresponding skillbot 116. In this case, the number of conversation categories may be equal to the number of skillbots 116 available to the master bot 114. In some embodiments, when conversation categories are based on subbots, each conversation category includes a subset of training feature vectors 620 representing a single corresponding skillbot 116. In that case, the number of conversation categories may be greater than or equal to the number of skill bots 116 available to the master bot 114.

[0180] In block 1820, the training system 350 generates a composite feature vector from the training feature vector 620, so that each training feature vector 620 within a conversation category represents that conversation category and also within that conversation category. The features are aggregated into a corresponding composite feature vector. In other words, for each conversation category to which a training feature vector 620 is assigned in block 1815, the training system 350 may combine each training feature vector 620 into a composite feature vector for that conversation category. Various techniques can be used for aggregation. For example, the composite feature vector may be the mean (e.g., arithmetic mean) of the training feature vectors 620 in each category. Based on the conversation category, the composite feature vector may be an intent vector and a sub-bot vector, or a bot vector, depending on whether the conversation category is defined based on an intent, sub-bot, or skillbot 116.

[0181] A composite feature vector can be generated as the arithmetic mean of the training feature vectors 620, but other mathematical functions, including other types of linear combinations, may be used additionally or alternatively to generate the composite feature vector. For example, in some embodiments, the composite feature vector may be a weighted average of the training feature vectors 620 within a conversation category. The weighting may be based on various factors, such as the priority of a particular keyword in the corresponding training utterance 615, so that training utterances 615 with a particular keyword are given a greater weight. When generating a subbot vector or bot vector, the training feature vectors 620 may be aggregated as a weighted average such that training feature vectors 620 corresponding to a particular intent are given a greater weight than other training feature vectors 620. For example, training feature vectors 620 corresponding to intents with a larger number of representative training utterances 615 may be given a greater weight in a set state compared to training feature vectors 620 corresponding to intents with a smaller number of representative training utterances 615. Various implementations are possible and are within the scope of this disclosure.

[0182] Figures 19 and 20 illustrate the concept of a composite feature vector. Specifically, Figure 19 shows the generation of a composite feature vector using intent-based conversation categories according to several embodiments described herein. As a result, the composite feature vector shown in Figure 19 is intent-level and therefore an intent vector. In the example of Figure 19, at least two skillbots 116 are available to the masterbot 114. The first skillbot 116 is associated with first skillbot data 358g, which includes a training utterance 615 representing intent A, which is a first intent, and another training utterance 615 representing intent B, which is a second intent. Thus, the first skillbot 116 is configured to handle input utterances associated with either intent A or intent B. The second skillbot 116 is associated with second skillbot data 358h, which includes a training utterance 615 representing intent C, which is a third intent. Therefore, the second skillbot 116 is configured to handle the input utterance associated with intent C.

[0183] In the example in Figure 19, the training system 350 converts all training utterances 615 related to the skillbot 116 into their respective feature vectors 620. The training system 350 groups the training feature vectors into intent-based conversation categories. Thus, the training feature vectors 620 corresponding to the training utterances 615 for intent A form the first conversation category, the training feature vectors 620 corresponding to the training utterances 615 for intent B form the second conversation category, and the training feature vectors 620 corresponding to the training utterances 615 for intent C form the third conversation category. In this example, as described above, the training feature vectors 620 of a given intent-based conversation category are aggregated (e.g., averaged) into a composite feature vector for the conversation category and therefore for the associated intents. Specifically, the training system 350 for intent A The training feature vector 620 corresponding to the training utterance 615 of intent B is aggregated into the first composite feature vector 1910a. The training system 350 then aggregates the training feature vector 620 corresponding to the training utterance 615 of intent B into the second composite feature vector 1910b. The training system 350 then aggregates the training feature vector 620 corresponding to the training utterance 615 of intent C into the third composite feature vector 1910c. Thus, in this example, each composite feature vector represents the respective intent of the skillbot 116.

[0184] Figure 20 illustrates the generation of a composite feature vector using skillbot-based conversation categories, also referred to as bot-based conversation categories, according to some embodiments described herein. As a result, the composite feature vector shown in Figure 20 is skillbot-level and therefore a bot vector. In the example of Figure 20, at least two skillbots 116 are available to the master bot 114. The first skillbot 116 is associated with first skillbot data 358g, which includes a training utterance 615 representing intent A, a first intent, and another training utterance 615 representing intent B, a second intent. Thus, the first skillbot 116 is configured to handle input utterances associated with either intent A or intent B. The second skillbot 116 is associated with second skillbot data 358h, which includes a training utterance 615 representing intent C, a third intent. Thus, the second skillbot 116 is configured to handle input utterances associated with intent C.

[0185] In the example in Figure 20, the training system 350 converts all training utterances 615 relating to the skillbot 116 into their respective feature vectors 620. The training system 350 groups the training feature vectors into skillbot-based conversation categories. Thus, the training feature vectors 620 corresponding to the training utterances 615 for intent A, together with the training feature vectors 620 corresponding to the training utterances 615 for intent B, form a first conversation category, and the training feature vectors 620 corresponding to the training utterances 615 for intent C form a second conversation category. In this example, as described above, the training feature vectors 620 of a given skillbot-based conversation category are aggregated (e.g., averaged) into a composite feature vector for the conversation category and therefore for the relevant skillbot 116. Specifically, the training system 350 combines the training feature vector 620 corresponding to the training utterance 615 for intent A and the training feature vector 620 corresponding to the training utterance 615 for intent B into a first composite feature vector 1910d, and the training system 350 combines the training feature vector 620 corresponding to the training utterance 615 for intent C into a second composite feature vector 1910e. Therefore, in this example, each composite feature vector represents a respective skillbot 116.

[0186] In some embodiments, the training system 350 is not limited to generating only one type of composite feature vector. For example, the training system 350 may generate intent vectors, subbot vectors, and skillbot vectors, or it may generate any other combination of these or other types of composite feature vectors.

[0187] Returning to Figure 18, in block 1825, the training system 350 may configure the master bot 114's classifier model 324 to utilize the composite feature vector determined in block 1820. For example, the training system 350 may use memory (e.g., on the digital assistant 106) accessible by the classifier model 324. The device can store instructions for a composite feature vector. The classifier model 324 may be configured to compare the input utterance with such a composite feature vector, which serves as a set representation of the training feature vectors 620.

[0188] Figure 21 shows a method 2100 that uses a classifier model 324 of the masterbot 114, according to some embodiments described herein, to determine whether an input utterance 303 provided as user input 110 is unrelated to any available skillbot 116 associated with the masterbot 114. This method 2100 or a similar method can be used in block 215 of the above-described method 200, which configures and uses the masterbot 114 to direct the input utterance 303 to a skillbot 116. The method 2100 in Figure 21 is a more specific modification of the method 700 in Figure 7, and, like the method 700 in Figure 7, the method 2100 in Figure 21 can be used in conjunction with the preliminary filtering operation described with respect to the method 800 in Figure 8. More specifically, as described below, the classifier model 324 of the masterbot 114 may utilize a composite feature vector to determine whether the input utterance 303 belongs to a non-class 316 (i.e., unrelated to any available skillbot 116).

[0189] Method 2100 shown in Figure 21 and other methods described herein may be implemented in software (e.g., as code, instructions, or programs) executed by one or more processing units (e.g., processors or processor cores), in hardware, or in a combination thereof. The software may be stored in a non-temporary storage medium such as a memory device. Method 2100 is intended to be illustrative and non-limiting. Figure 21 shows various operations occurring in a particular order or sequence, but this is not intended to be limiting. In particular embodiments, for example, these operations may be performed in a different order, or one or more operations of Method 2100 may be performed in parallel. In particular embodiments, Method 200 may be performed by a master bot 114 associated with a set of available skill bots 116.

[0190] In block 2105 of method 2100, the master bot 114 accesses the input utterance 303 provided as user input 110. For example, in some embodiments, the user may provide the user input 110 in the form of voice input, and the digital assistant 106 may convert the user input 110 into a text input utterance 303 for use by the master bot 114.

[0191] In block 2110, the masterbot 114 may generate an input feature vector 1710 from the input utterance 303 accessed in block 2105. More specifically, in some embodiments, the masterbot 114 causes its classifier model 324 to generate the input feature vector 1710 from the input utterance 303. The input feature vector 1710 can describe and represent the input utterance 303. Various techniques are known for converting a set of words, such as the input utterance 303, into a feature vector, and one or more such techniques may be used. For example, the training system 350 may, but does not necessarily, encode the input utterance 303 as the corresponding input feature vector 1710 using one-hot coding or some other coding method. However, one embodiment of the masterbot 114 uses the same technique used to generate the training feature vector 620 from the training utterance 615 when training the classifier model 324.

[0192] In block 2115, the masterbot 114 compares the input feature vector 1710 generated in block 2110 with the composite feature vector. More specifically, in some embodiments, the masterbot 114 compares the classifier model 324 in block 2110. The determined input feature vector 1710 is compared with a composite feature vector that can be determined as described above with respect to Figure 18. When comparing the input feature vector 1710 with the composite feature vector, the classifier model 324 may determine the similarity or distance between the input feature vector 1710 and each composite feature vector previously constructed for the classifier model 324.

[0193] The similarity or distance between an input feature vector 1710 and another feature vector, specifically a composite feature vector, can be calculated in various ways. For example, to determine similarity, the classifier model 324 may calculate the absolute value of the arithmetic difference (i.e., the Euclidean distance) between the composite feature vector and the input feature vector 1710 by subtracting one from the other and obtaining the absolute value. In another example, the classifier model 324 may multiply the input feature vector 1710 by the composite feature vector. For example, if one-hot coding is used for the input feature vector 1710 and the composite feature vector, each vector entry (i.e., each dimension) will have a value of 1 or 0. In this case, a value of 1 may represent the presence of a particular feature. If the input feature vector 1710 and the composite feature vector have almost identical features, the result of the vector-vector multiplication will be a vector with approximately the same number of single values ​​as either of the two vectors being multiplied. Otherwise, the resulting vector will be mostly 0. As another example, cosine similarity can be used. For example, the cosine similarity between the input feature vector 1710 and the composite feature vector can be calculated as the dot product of the two vectors divided by the product of the Euclidean norms of both vectors. Various other techniques for measuring similarity are feasible and are within the scope of this disclosure.

[0194] In the decision block 2120, the classifier model 324 of the master bot 114 determines whether the input feature vector 1710 is sufficiently similar to any of the composite feature vectors. The classifier model 324 may use a predetermined threshold such that the similarity is considered sufficient if it satisfies the threshold. As with the distance metric, if the similarity metric used provides a small value when the two vectors are similar, the threshold may be, for example, an upper threshold such that the input feature vector 1710 is sufficiently close to the composite feature vector when the similarity is below the threshold. However, if the similarity metric used provides a large value when the two vectors are similar, the threshold may be a lower threshold such that the input feature vector 1710 is sufficiently close to the composite feature vector when the similarity is above the threshold.

[0195] In some embodiments, the determination of sufficient similarity can be made using a hierarchy of composite feature vectors. For example, the input feature vector 1710 may be compared to bot vectors until sufficient similarity is found, or until it is determined that the input feature vector 1710 is not sufficiently similar to any such composite feature vector, then, if necessary, to subbot vectors, and then, if necessary, to intent vectors. Since there are fewer bot vectors than intent vectors, and fewer intent vectors than subbot vectors, the computing resources required to determine similarity to bot vectors may be less than the computing resources required to determine similarity to intent vectors, and less than the computing resources required to determine similarity to subbot vectors. Thus, by comparing the input feature vector 1710 to composite feature vectors at various levels, starting from the highest available level (e.g., skillbot level), the classifier model 324 can determine which skillbot 116 is best suited to address the input utterance 303, if any, while using less computationally intensive processing before employing more computationally intensive processing.

[0196] In some embodiments, an input utterance 303 may be considered irrelevant to any of the available skill bots 116 if the input feature vector 1710 is sufficiently far from all bot vectors (e.g., beyond a predetermined distance considered excessive). However, in additional or alternative embodiments, an input utterance 303 is not considered irrelevant solely on the basis of its differences from bot vectors. For example, if the input feature vector 1710 is not similar to all bot vectors (i.e., not sufficiently similar), this is not necessarily a clear indication that the input utterance 303 belongs to a non-class 316. There may be situations where the input feature vector 1710 is far from any bot vector (i.e., not similar) because the corresponding intent is not similar to any other intents that comprise the skill bot 116, but is still close to a certain intent vector. In some embodiments, comparing the input feature vector 1710 to both bot vectors and intent vectors is typically sufficient to identify an irrelevant input utterance 303. Therefore, an input utterance 303 may be considered irrelevant to any available skill bot 116 if the input feature vector 1710 is not similar to any bot vector and is not similar to any intent vector. In some embodiments, the input feature vector 1710 may additionally or alternatively be compared to subbot vectors or other composite feature vectors as part of determining whether the input utterance 303 is a non-class member. Various implementations are possible and are within the scope of this disclosure.

[0197] In decision block 2120, if, based on any comparison performed, the classifier model 324 determines that the input feature vector 1710 is not sufficiently similar to any of the composite feature vectors, then method 2100 proceeds to block 2125. In block 2125, the master bot 114 may determine that the input utterance is unrelated to any of the available skill bots 116, and therefore indicate that it cannot process the input utterance 303. As a result, for example, the digital assistant may ask the user to clarify the user input 110.

[0198] However, if the classifier model 324 determines in decision block 2120 that the input feature vector 1710 is sufficiently similar to one or more composite feature vectors, method 2100 proceeds to block 2130. In block 2130, based on comparison with the composite feature vectors, the master bot 114 determines (i.e., selects) a skill bot 116 to address the input utterance 303. For example, the master bot 114 may select a skill bot 116 associated with the composite feature vector considered to be most similar to the input feature vector 1710. For example, if the most similar composite feature vector is an intent vector, the master bot 114 may select a skill bot 116 configured to address the intent corresponding to the intent vector. Furthermore, if the most similar composite feature vector is a sub-bot vector or bot vector, the master bot 114 may select a skill bot 116 corresponding to the sub-bot vector or bot vector. In another example, the master bot 114 may use the k-nearest neighbor technique to select a skill bot 116, as further described below.

[0199] In block 2135, the master bot 114 may route the input utterance to the skill bot 116 selected in block 2130. In some embodiments, if the master bot 114 identifies a particular intent for the input utterance (for example, if the input feature vector 1710 is considered close enough to a single intent vector), the master bot 114 may present that intent to the skill bot 116, thereby allowing the skill bot 116 to skip the process of inferring the intent. The skill bot 116 can then process the input utterance 303 and respond to the user input 110.

[0200] Figure 22 shows a method 2200 for selecting a skillbot 116 to address an input utterance, according to some embodiments described herein. Method 2200 or a similar method may be used in block 2130 of method 2100 described above, after determining that the input utterance 303 relates to at least one available skillbot 116. Method 2200 provides a k-nearest neighbor method for selecting a skillbot 116, although other techniques may be used in addition to or instead of this method. Specifically, method 2200 utilizes k-nearest neighbor training feature vectors 620 for an input feature vector 1710 to determine which skillbot 116 should be selected.

[0201] Method 2200 shown in Figure 22 and other methods described herein may be implemented in software (e.g., as code, instructions, or programs) executed by one or more processing units (e.g., processors or processor cores), in hardware, or in a combination thereof. The software may be stored in a non-temporary storage medium such as a memory device. Method 2200 is intended to be illustrative and non-limiting. Figure 22 shows various operations occurring in a particular order or sequence, but this is not intended to be limiting. In particular embodiments, for example, these operations may be performed in a different order, or one or more operations of Method 2200 may be performed in parallel. In particular embodiments, Method 2200 may be performed by a master bot 114.

[0202] As shown in Figure 22, in block 2205, the master bot 114 determines a value for k, where k is the number of neighbors to be considered. In some embodiments, the value of k may be a coefficient of the total number of training feature vectors 620 (i.e., the total number of training utterances 615).

[0203] In block 2210, using the value of k determined in block 2205, the master bot 114 determines the set of k training feature vectors 620 that are closest to (i.e., most similar to) the input feature vector 1710. The master bot 114 may use the same or different similarity metrics as those used above when determining whether the input feature vector 1710 is sufficiently similar to any of the composite feature vectors. For example, the similarity metric used could be Euclidean distance, vector multiplication, or cosine similarity.

[0204] In block 2215, the master bot 114 selects the skill bot 116 having the most training feature vectors 620 in the set determined in block 2210. In some embodiments, the training feature vectors 620 of skill bot 116 do not need to constitute the majority of the set; rather, it is only necessary that there are no other skill bots 116 in the set that have a larger number of training feature vectors 620. As described above, after selecting this skill bot 116, the master bot 114 may route the input utterance 303 to the selected skill bot 116 for processing.

[0205] In addition to considering the training feature vector 620 closest to the input feature vector 1710, or alternatively, some embodiments of the master bot 114 may also consider the closest composite feature vector, such as the closest intent vector. Figure 23 shows another exemplary method 2300 for selecting a skill bot 116 to address an input utterance, according to some embodiments described herein. Method 2300 or a similar method may be used in block 2130 of method 2100 described above after determining that the input utterance 303 relates to at least one available skill bot 116. Method 2300 provides a k-nearest neighbor method for selecting a skill bot 116, but other techniques may be used in addition to or instead of this method. Specifically, Figure 2 In contrast to method 2200, method 2300 uses the k nearest neighbor intent vectors for the input feature vector 1710 to determine which skillbot 116 to select.

[0206] Method 2300 shown in Figure 23 and other methods described herein can be implemented in software (e.g., as code, instructions, or programs) executed by one or more processing units (e.g., processors or processor cores), in hardware, or in a combination thereof. The software may be stored in a non-temporary storage medium such as a memory device. Method 2300 is intended to be illustrative and non-limiting. Figure 23 shows various operations occurring in a particular order or sequence, but this is not intended to be limiting. In particular embodiments, for example, these operations may be performed in a different order, or one or more operations of Method 2300 may be performed in parallel. In particular embodiments, Method 2300 may be performed by a master bot 114.

[0207] As shown in Figure 23, in block 2305, the master bot 114 determines a value for k, where k is the number of neighborhoods to be considered. In some embodiments, the value of k may be a coefficient of the total number of intent vectors. The total number of intent vectors may be equal to the total number of intents represented in the training utterance 615, or equal to the total number of intents configured to be dealt with by the available skill bot 116, and the value of k may be chosen as a coefficient of that amount.

[0208] In block 2310, the masterbot 114 uses the value of k determined in block 2305 to determine the set of k intent vectors that are closest to (i.e., most similar to) the input feature vector 1710. The masterbot 114 may use the same or a different similarity metric as used above when determining whether the input feature vector 1710 is sufficiently similar to any of the composite feature vectors in method 2100. For example, the similarity metric used may be Euclidean distance, vector multiplication, or cosine similarity.

[0209] In block 2315, the master bot 114 selects the skill bot 116 that has the most intent vectors in the set determined in block 2310. In some embodiments, the intent vectors of skill bot 116 do not need to constitute the majority of the set; rather, it is only necessary that there are no other skill bots 116 in the set that have more intent vectors. As described above, after selecting this skill bot 116, the master bot 114 may route the input utterance 303 to the selected skill bot 116 for processing.

[0210] Examples of implementation Figure 24 shows a simplified diagram of a distributed system 2400 for realizing one embodiment. In the illustrated embodiment, the distributed system 2400 includes one or more client computing devices 2402, 2404, 2406, and 2408 connected to a server 2412 via one or more communication networks 2410. The client computing devices 2402, 2404, 2406, and 2408 may be configured to run one or more applications.

[0211] In various examples, server 2412 may be adapted to run one or more services or software applications that enable the processing described in this disclosure.

[0212] In certain embodiments, the server 2412 may also provide other services or software applications, which may include non-virtual and virtual environments. In some embodiments, These services are provided to users of client computing devices 2402, 2404, 2406 and / or 2408 as web-based services or under the Software as a Service (SaaS) model. These may be offered as loud services. Users operating client computing devices 2402, 2404, 2406, and / or 2408 may access the services provided by these components by interacting with server 2412 using one or more client applications.

[0213] In the configuration shown in Figure 24, server 2412 may include one or more components 2418, 2420, and 2422 that implement the functions performed by server 2412. These components may include one or more processors, hardware components, or software components that can be executed by a combination thereof. It should be recognized that a wide variety of system configurations different from the distributed system 2400 are possible. Therefore, the embodiment shown in Figure 24 is an example of a distributed system for implementing the system of the embodiment and is not intended to be limiting.

[0214] A user may interact with server 2412 in accordance with the teachings of this disclosure using client computing devices 2402, 2404, 2406, and / or 2408. A client device may provide an interface that enables a user of that client device to interact with that client device. A client device may also output information to the user through this interface. Although Figure 24 shows only four client computing devices, any number of client computing devices may be supported.

[0215] Client devices may include various types of computing systems, such as portable handheld devices, general-purpose computers like personal computers and laptops, workstation computers, wearable devices, gaming systems, thin clients, various messaging devices, sensors or other sensing devices. These computing devices may run various types and versions of software applications and operating systems (e.g., Microsoft Windows®, Apple Macintosh®, UNIX® or UNIX-like operating systems, Linux® or Linux-like operating systems, and various mobile operating systems, including Google Chrome® OS, such as Microsoft Windows Mobile®, iOS®, Windows Phone®, Android®, BlackBerry®, and Palm OS®). Portable handheld devices may include cellular phones, smartphones (e.g., iPhone®), tablets (e.g., iPad®), and personal digital assistants (PDAs). Wearable devices may include Google Glass® head-mounted displays and other devices. The gaming system may include various handheld gaming devices and internet-connected gaming devices (for example, Microsoft Xbox® game consoles with or without Kinect® gesture input devices, Sony PlayStation® systems, and various gaming systems provided by Nintendo®). Client devices may be capable of running a wide variety of applications, such as various internet-related applications and communication applications (for example, email applications and short message service (SMS) applications), and may use various communication protocols.

[0216] Network 2410 may be any type of network known to those skilled in the art that can support data communication using any of the various available protocols, including but not limited to TCP / IP (Transmission Control Protocol / Internet Protocol), SNA (System Network Architecture), IPX (Internet Packet Switching), AppleTalk®, etc. Just as an example, Network 2410 may include local area networks (LANs), Ethernet®-based networks, Token Ring, wide area networks (WANs), the Internet, virtual networks, virtual private networks (VPNs), intranets, extranets, public switched telephone networks (PSTNs), infrared networks, wireless networks (for example, wireless networks operating under any of the IEEE 802.11 protocol suites, Bluetooth®, and / or any other wireless protocols), and / or any combination of these and / or other networks.

[0217] Server 2412 may consist of one or more general-purpose computers, dedicated server computers (including, for example, PC (personal computer) servers, UNIX® servers, midrange servers, mainframe computers, rack-mount servers, etc.), server farms, server clusters, or other appropriate configurations and / or combinations. Server 2412 may include one or more virtual machines running a virtual operating system, or other computing architectures with virtualization. This could be, for example, one or more flexible pools of logical storage that can be virtualized to maintain virtual storage for the server. In various embodiments, Server 2412 may be adapted to run one or more services or software applications that provide the functions described above.

[0218] The computing system within Server 2412 may run one or more operating systems, including any of those described above, and commercially available server operating systems. Server 2412 may also run any of a variety of additional server applications and / or middle-tier applications, including HTTP (Hypertext Transfer Protocol) servers, FTP (File Transfer Protocol) servers, CGI (Common Gateway Interface) servers, Java® servers, and database servers. Illustrative database servers include, but are not limited to, those commercially available from Oracle®, Microsoft®, Sybase®, IBM® (International Business Machines), and others. It will not be done.

[0219] In some implementations, server 2412 may include one or more applications for analyzing and organizing data feeds and / or event updates received from users of client computing devices 2402, 2404, 2406, and 2408. For example, data feeds and / or event updates may include, but are not limited to, Twitter® feeds, Facebook® updates, or real-time updates received from one or more third-party sources and continuous data streams, including real-time events related to sensor data applications, financial stock market boards, network performance measurement tools (e.g., network monitoring and traffic management applications), clickstream analysis tools, and automotive traffic monitoring. Server 2412 may also include one or more applications for displaying data feeds and / or real-time events via one or more display devices on client computing devices 2402, 2404, 2406, and 2408.

[0220] The distributed system 2400 may also include one or more data repositories 2414, 2416. In certain embodiments, these data repositories may be used to store data and other information. For example, one or more of the data repositories 2414, 2416 may be used to store data or information generated by the processing described herein, and / or data or information used for the processing described herein. The data repositories 2414, 2416 may reside in various locations. For example, a data repository used by server 2412 may be located locally with server 2412, or it may be located remotely from server 2412 and communicates with server 2412 via a network-based connection or a dedicated connection. The data repositories 2414, 2416 may be of different types. In certain embodiments, a data repository used by server 2412 may be a database, for example, a relational database such as a database provided by Oracle Corporation® and other manufacturers. One or more of these databases may be adapted to allow data to be stored, updated, and retrieved from the database in response to commands in SQL format.

[0221] In certain embodiments, one or more of the data repositories 2414, 2416 may be used by the application to store application data. The data repositories used by the application may be of various types, such as a key-value store repository, an object store repository, or a general-purpose storage repository supported by the file system.

[0222] In certain embodiments, the functions described herein may be provided as services through a cloud environment. Figure 25 is a simplified block diagram of a cloud-based system environment that may provide the functions described herein as cloud services according to a particular embodiment. In the embodiment shown in Figure 25, the cloud infrastructure system 2502 may provide one or more cloud services that a user may request using one or more client computing devices 2504, 2506, and 2508. The cloud infrastructure system 2502 may include one or more computers and / or servers, which may include those described above with respect to server 2412. The computers within the cloud infrastructure system 2502 may be organized as general-purpose computers, dedicated server computers, server farms, server clusters, or any other appropriate arrangement and / or combination.

[0223] Network 2510 can facilitate data communication and exchange between clients 2504, 2506, and 2508 and the cloud infrastructure system 2502. Network 2510 may include one or more networks. The networks may be of the same type or different types. Network 2510 may support one or more communication protocols, including wired and / or wireless protocols, to facilitate communication.

[0224] The embodiment shown in Figure 25 is merely one example of a cloud infrastructure system and is not intended to be limiting. It should be understood that in several other embodiments, the cloud infrastructure system 2502 may have more or fewer components than those shown in Figure 25, may combine two or more components, or may have components in different configurations or arrangements. For example, while Figure 25 shows three client computing devices, any number of client computing devices may be supported in alternative embodiments.

[0225] The term "cloud service" is generally used to refer to services made available to users on demand via communication networks such as the Internet, through a service provider's system (e.g., cloud infrastructure system 2502). Typically, in a public cloud environment, the servers and systems that make up the cloud service provider's system are different from the customer's own on-premises servers and systems. The cloud service provider's system is managed by the cloud service provider. Therefore, customers can use the cloud services provided by the cloud service provider without purchasing separate licenses, support, or hardware and software resources for the service. For example, the cloud service provider's system can host applications, and users can order and use applications on demand via the Internet without purchasing infrastructure resources to run the applications. Cloud services are designed to provide easy and scalable access to applications, resources, and services. Several providers offer cloud services. For example, some cloud services, such as middleware services, database services, and Java® cloud services, are offered by Oracle Corporation® in Redwood Shores, California.

[0226] In certain embodiments, the cloud infrastructure system 2502 may provide one or more cloud services using various models, including a hybrid service model, such as a Software as a Service (SaaS) model, a Platform as a Service (PaaS) model, and an Infrastructure as a Service (IaaS) model. The cloud infrastructure system 2502 may include a set of applications, middleware, databases, and other resources that enable the provisioning of various cloud services.

[0227] The SaaS model enables the delivery of applications or software as a service to customers over a communication network such as the internet, without requiring customers to purchase the underlying hardware or software for the application. For example, the SaaS model can be used to allow customers to access on-demand applications hosted on a cloud infrastructure system 2502. Examples of SaaS services offered by Oracle Corporation® include, but are not limited to, various services for human resources / capital management, customer relationship management (CRM), enterprise resource planning (ERP), supply chain management (SCM), enterprise performance management (EPM), analytics services, and social applications.

[0228] The IaaS model is generally used to provide flexible computing and storage capabilities by offering infrastructure resources (such as servers, storage, hardware, and networking resources) to customers as cloud services. Various IaaS services are offered by Oracle Corporation®.

[0229] The PaaS model is generally used to provide a platform and environmental resources as a service, enabling customers to develop, run, and manage applications and services without having to procure, build, or manage those resources themselves. An example of a PaaS service offered by Oracle Corporation is Oracle This includes, but is not limited to, Java Cloud Service (JCS), Oracle Database Cloud Service (DBCS), data management cloud services, and various application development solution services.

[0230] Cloud services are generally on-demand, self-service based, and subscription-based. It is provided in a flexible, scalable, reliable, highly available, and secure manner on a subscription basis. For example, a customer may order one or more services provided by the cloud infrastructure system 2502 via a subscription order. The cloud infrastructure system 2502 then provides the services requested in the customer's subscription order by performing processing. For example, in certain embodiments, the chatbot-related functionality described herein may be provided as a cloud service provided by the user / subscriber. The cloud infrastructure system 2502 may be configured to provide one cloud service or more cloud services.

[0231] The cloud infrastructure system 2502 can provide cloud services through various deployment models. In a public cloud model, the cloud infrastructure system 2502 may be owned by a third-party cloud service provider, and the cloud services are provided to the general public. These customers may be individuals or companies. In other specific embodiments, under a private cloud model, the cloud infrastructure system 2502 may function within an organization (for example, within a corporate organization), and the services are provided to customers within this organization. For example, these customers may be various departments within the company, such as the human resources department or the payroll department, or individuals within the company. In yet another embodiment, under a community cloud model, the cloud infrastructure system 2502 and the services provided may be shared among various organizations within a relevant community. Various other models, such as hybrid models of the above models, may also be used.

[0232] The client computing devices 2504, 2506, and 2508 may be of different types (for example, devices 2402, 2404, 2406, and 2408 shown in Figure 24) and may be capable of operating one or more client applications. Users may interact with the cloud infrastructure system 2502, such as requesting services provided by the cloud infrastructure system 2502, by using the client devices. For example, a user may use a client device to request chatbot-related services described in this disclosure.

[0233] In some embodiments, the processing performed by the cloud infrastructure system 2502 may include big data analysis. This analysis may include detecting and visualizing various trends, behaviors, relationships, etc., within a large dataset by using, analyzing, and processing it. This analysis may be performed by one or more processors, which may, in some cases, process the data in parallel and run simulations using the data. The data used for this analysis may be structured data (e.g., data stored in a database or data structured according to a structured model) and / or unstructured data (e.g., data blobs (binary large objects)). It may include the object.

[0234] As shown in the embodiment of Figure 25, the cloud infrastructure system 2502 may include infrastructure resources 2530 used to facilitate the provision of various cloud services provided by the cloud infrastructure system 2502. The infrastructure resources 2530 may include, for example, processing resources, storage or memory resources, networking resources, etc.

[0235] In a particular embodiment, these resources are efficiently used to support various cloud services that the cloud infrastructure system 2502 provides to different customers. To facilitate provisioning, resources may be grouped into sets of resources or resource modules (also referred to as "pods"). Each resource module or pod may contain a pre-integrated and optimized combination of one or more types of resources. In certain embodiments, different pods may be pre-provisioned for different types of cloud services. For example, a first set of pods may be provisioned for a database service, and a second set of pods, which may contain different resource combinations than the pods in the first set, may be provisioned for a Java service, etc. For several services, resources allocated to provisioning these services may be shared among the services.

[0236] The cloud infrastructure system 2502 itself may internally use services 2532 that are shared by various components of the cloud infrastructure system 2502 and facilitate the provisioning of services by the cloud infrastructure system 2502. These internally shared services may include, but are not limited to, security and identity services, integration services, enterprise repository services, enterprise manager services, virus scanning and whitelisting services, high availability, backup and recovery services, services that enable cloud support, email services, notification services, and file transfer services.

[0237] The cloud infrastructure system 2502 may include multiple subsystems. These subsystems may be implemented in software, hardware, or a combination thereof. As shown in Figure 25, the subsystems may include a user interface subsystem 2512 that enables users or customers of the cloud infrastructure system 2502 to interact with the cloud infrastructure system 2502. The user interface subsystem 2512 may include various different interfaces, such as a web interface 2514, an online store interface 2516 where cloud services offered by the cloud infrastructure system 2502 are advertised and available for purchase by consumers, and other interfaces 2518. For example, a customer may use a client device to request (service request 2534) one or more services offered by the cloud infrastructure system 2502 using one or more of interfaces 2514, 2516, and 2518. For example, a customer may access the online store, browse the cloud services offered by the cloud infrastructure system 2502, and place a subscription order for one or more services offered by the cloud infrastructure system 2502 that the customer wishes to subscribe to. This service request may include information that identifies the customer and one or more services that the customer wishes to request.

[0238] In certain embodiments, such as the embodiment shown in Figure 25, the cloud infrastructure system 2502 may include an order management subsystem (OMS) 2520 configured to process new orders. As part of this process, the OMS 2520 may be configured to prepare the order for provisioning by creating a customer account if one does not already exist, receiving billing and / or account information from the customer to be used to charge the customer in order to provide the requested services to the customer, verifying the customer information, reserving the order for the customer after verification, and coordinating various workflows.

[0239] If properly validated, OMS2520 may invoke the Order Provisioning Subsystem (OPS)2524, which is configured to provision resources for this order, including processing, memory, and networking resources. Lobbying can involve allocating resources for an order and configuring those resources to facilitate the services requested by the customer order. The way resources are provisioned for an order and the types of resources provisioned may depend on the type of cloud service ordered by the customer. For example, following a certain workflow, OPS2524 may be configured to determine the specific cloud service being requested and identify the number of pods that would have been pre-configured for that particular cloud service. The number of pods allocated for an order may depend on the size / volume / level / scope of the requested service. For example, the number of pods to allocate may be determined based on the number of users the service should support, the duration for which the service is requested, etc. The allocated pods may then be customized to suit the specific customer making the request in order to provide the requested service.

[0240] The cloud infrastructure system 2502 may send a response or notification 1044 to the requesting customer to indicate when the requested service will be available. In some examples, the customer may be sent information (e.g., a link) that will enable the customer to begin using and utilizing the benefits of the requested service.

[0241] The cloud infrastructure system 2502 may provide services to multiple customers. For each customer, the cloud infrastructure system 2502 manages information related to one or more subscription orders received from the customer, maintains customer data related to the orders, and provides the requested services to the customer. The cloud infrastructure system 2502 may also collect usage statistics about the customer's use of the subscribed services. For example, statistics may be collected on the amount of storage used, the amount of data transferred, the number of users, and the amount of system uptime and system downtime. This usage information may be used to charge customers. Billing may be done, for example, on a monthly basis.

[0242] The cloud infrastructure system 2502 may provide services to multiple customers in parallel. The cloud infrastructure system 2502 may store information about these customers, which may include copyright information. In certain embodiments, the cloud infrastructure system 2502 includes an Identity Management Subsystem (IMS) 2528 configured to manage customer information and separate the managed information so that information about one customer cannot be accessed by another customer. The IMS 2528 may be configured to provide various security-related services, such as identity services including information access management, authentication and authorization services, and services for managing customer identities and roles and related capabilities.

[0243] Figure 26 shows an exemplary computer system 2600 that may be used to implement a particular embodiment. For example, in some embodiments, the computer system 2600 may be used to implement either the system or subsystems of a chatbot system, as well as the various servers and computer systems described above. As shown in Figure 26, the computer system 2600 includes various subsystems, including a processing subsystem 2604 that communicates with several other subsystems via a bus subsystem 2602. These other subsystems may include a processing acceleration unit 2606, an I / O subsystem 2608, a storage subsystem 2618, and a communication subsystem 2624. The storage subsystem 2618 may include a non-temporary computer-readable storage medium including a storage medium 2622 and system memory 2610.

[0244] The bus subsystem 2602 is a bus subsystem for various components of the computer system 2600. It provides a mechanism for the terminal and subsystems to communicate with each other as intended. While the bus subsystem 2602 is schematically shown as a single bus, alternative embodiments of the bus subsystem may utilize multiple buses. The bus subsystem 2602 may be one of several types of bus structures, including a memory bus or memory controller, peripheral bus, local bus, etc., using any of the various bus architectures. For example, such architectures include the Industry Standard Architecture (ISA) bus, Micro Channel Architecture (MCA) bus, Enhanced ISA (EISA) bus, Video Electronics Standards Association (VESA) local bus, and IEEE P1386. This may include Peripheral Component Interconnect (PCI) buses, which can be implemented as mezzanine buses manufactured according to a specific standard.

[0245] The processing subsystem 2604 controls the operation of the computer system 2600 and may include one or more processors, application-specific integrated circuits (ASICs), or field-programmable gate arrays (FPGAs). The processors may include single-core or multi-core processors. The processing resources of the computer system 2600 can be organized into one or more processing units 2632, 2634, etc. A processing unit may include one or more processors, one or more cores from the same or different processors, a combination of cores and processors, or other combinations of cores and processors. In some embodiments, the processing subsystem 2604 may include one or more dedicated coprocessors, such as graphics processors or digital signal processors (DSPs). In some embodiments, some or all of the processing units of the processing subsystem 2604 may be implemented using customized circuitry, such as application-specific integrated circuits (ASICs) or field-programmable gate arrays (FPGAs).

[0246] In some embodiments, processing units within the processing subsystem 2604 may execute instructions stored in system memory 2610 or computer-readable storage medium 2622. In various embodiments, processing units may execute various program or code instructions and maintain multiple programs or processes running concurrently. At any given time, some or all of the program code to be executed may reside in system memory 2610 and / or computer-readable storage medium 2622, which may potentially include one or more storage devices. Through appropriate programming, the processing subsystem 2604 may provide the various functions described above. In an example where the computer system 2600 is running one or more virtual machines, one or more processing units may be assigned to each virtual machine.

[0247] In certain embodiments, a processing acceleration unit 2606 may optionally be provided to perform customized processing so as to accelerate the overall processing executed by the computer system 2600, or to offload part of the processing executed by the processing subsystem 2604.

[0248] The I / O subsystem 2608 may include devices and mechanisms for inputting information to the computer system 2600 and / or outputting information from or via the computer system 2600. In general, use of the term "input device" is intended to include all possible types of devices and mechanisms for inputting information to the computer system 2600. User interface input devices include, for example, a keyboard, a pointing device such as a mouse or a trackball, a touchpad or touch screen incorporated into a display, a scroll wheel, a click wheel, a dial, a button, a switch, a keypad code, a voice input device with a voice command recognition system, a microphone, and other types of input devices. The user interface input devices may also include motion sensing and / or gesture recognition devices, such as Microsoft Kinect® motion sensors, Microsoft Xbox® 360 game controllers, and devices equipped with interfaces for receiving inputs using gestures and voice commands, which allow users to control and interact with the input devices. The user interface input devices may also include eye gesture recognition devices such as the Google Glass® blink detector, which detects eye movements from the user (e.g., "blinking" while taking a photo and / or making a menu selection) and converts the eye gestures as inputs to an input device (e.g., Google Glass®). In addition, the user interface input devices may include voice recognition sensing devices that allow users to interact with voice recognition systems (e.g., Siri® navigator) via voice commands.

[0249] Other examples of user interface input devices include, but are not limited to, three-dimensional (3D) mice, joysticks or pointing sticks, gamepads and graphics tablets, as well as audio / visual devices such as speakers, digital cameras, digital camcorders, portable media players, webcams, image scanners, fingerprint scanners, barcode readers, 3D scanners, 3D printers, laser rangefinders, and gaze tracking devices. In addition, the user interface input devices may include medical imaging input devices, such as computed tomography, magnetic resonance imaging, positron emission tomography, and medical ultrasound examination devices. The user interface input devices may also include audio input devices such as MIDI keyboards and digital musical instruments, for example.

[0250] Generally, the use of the term "output device" is intended to include all conceivable types of devices and mechanisms for outputting information from the computer system 2600 to a user or another computer. User interface output devices may include non-visual displays such as display subsystems, indicator lights, or audio output devices. Display subsystems may include flat panel devices such as those using cathode ray tubes (CRTs), liquid crystal displays (LCDs), or plasma displays, projection devices, touchscreens, etc. For example, user interface output devices may include, but are not limited to, a variety of display devices that visually convey text, graphics, and audio / video information, such as monitors, printers, speakers, headphones, car navigation systems, plotters, audio output devices, and modems.

[0251] The storage subsystem 2618 provides a repository or datastore for storing information and data used by the computer system 2600. The storage subsystem 2618 provides a tangible, non-temporary, computer-readable storage medium for storing basic programming and data configurations that provide functionality in some embodiments. Software (e.g., programs, code modules, instructions) that, when executed by the processing subsystem 2604, provides the functionality described above may be stored in the storage subsystem 2618. The software may be executed by one or more processing units of the processing subsystem 2604. The storage subsystem 2618 may also provide a repository for storing data used in accordance with the teachings of this disclosure.

[0252] The storage subsystem 2618 may include one or more non-temporary memory devices, including volatile and non-volatile memory devices. As shown in Figure 26, the storage subsystem 2618 includes system memory 2610 and computer-readable storage medium 2622. The system memory 2610 may include several memories, including volatile primary random access memory (RAM) for storing instructions and data during program execution, and non-volatile read-only memory (ROM) or flash memory for storing fixed instructions. In some implementations, a basic input / output system (BIOS), which includes basic routines that assist in the transfer of information between elements within the computer system 2600 during startup, etc., may typically be stored in ROM. Typically, RAM contains data and / or program modules currently being operated and executed by the processing subsystem 2604. In some implementations, system memory 2610 may include several different types of memory, such as static random access memory (SRAM) and dynamic random access memory (DRAM).

[0253] As an example, and without limitation, as shown in Figure 26, the system memory 2610 may load running application programs 2612, program data 2614, and operating systems 2616, which may include various applications such as web browsers, middle-tier applications, and relational database management systems (RDBMS). As an example, the operating system 2616 may include Microsoft Windows®, Apple Macintosh®, and / or Linux operating systems, various commercially available UNIX® or UNIX-like operating systems (including, but not limited to, various GNU / Linux operating systems, Google Chrome® OS, etc.), and / or various versions of mobile operating systems such as iOS®, Windows Phone, Android® OS, BlackBerry® OS, and Palm® OS.

[0254] The computer-readable storage medium 2622 may store programming and data structures that provide functionality in several embodiments. The computer-readable storage medium 2622 may provide storage for computer-readable instructions, data structures, program modules, and other data for the computer system 2600. Software (programs, code modules, instructions) that, when executed by the processing subsystem 2604, provides the above functionality may be stored in the storage subsystem 2618. As an example, the computer-readable storage medium 2622 may include non-volatile memory such as a hard disk drive, magnetic disk drive, optical disk drive such as a CD-ROM, DVD, or Blu-ray® disc, or other optical media. The computer-readable storage medium 2622 may include, but is not limited to, a Zip® drive, flash memory card, Universal Serial Bus (USB) flash drive, Secure Digital (SD) card, DVD disc, digital videotape, and the like. Computer-readable storage media 2622 may also include solid-state drives (SSDs) based on non-volatile memory such as flash memory-based SSDs, enterprise flash drives, and solid-state ROMs; SSDs based on volatile memory such as solid-state RAM, dynamic RAM, and static RAM; DRAM-based SSDs; magnetoresistive RAM (MRAM) SSDs; and hybrid SSDs that use a combination of DRAM and flash memory-based SSDs.

[0255] In certain embodiments, the storage subsystem 2618 may also include a computer-readable storage medium reader 2620 that can be further connected to the computer-readable storage medium 2622. The reader 2620 may be configured to receive and read data from memory devices such as disks, flash drives, etc.

[0256] In certain embodiments, the computer system 2600 may support virtualization techniques, including but not limited to virtualization of processing and memory resources. For example, the computer system 2600 may provide support for running one or more virtual machines. In certain embodiments, the computer system 2600 may run programs such as hypervisors that facilitate the configuration and management of virtual machines. Each virtual machine may be allocated memory, computing resources (e.g., processors, cores), I / O, and networking resources. Each virtual machine typically runs independently of other virtual machines. A virtual machine typically runs its own operating system, which may be the same as or different from the operating system run by other virtual machines run by the computer system 2600. Thus, potentially multiple operating systems can run simultaneously by the computer system 2600.

[0257] The communication subsystem 2624 provides interfaces to other computer systems and networks. It functions as an interface for sending and receiving data between other systems and the computer system 2600. For example, the communication subsystem 2624 may enable the computer system 2600 to establish communication channels to one or more client devices via the Internet in order to send and receive information with one or more client devices.

[0258] The communication subsystem 2624 may support both wired and / or wireless communication protocols. For example, in certain embodiments, the communication subsystem 2624 may include radio frequency (RF) transceiver components for accessing wireless voice and / or data networks (e.g., using cellular telephone technology, 3G, 4G, or EDGE (High Speed ​​Data Rate for Global Evolution)), a Global Positioning System (GPS) receiver component, and / or other components. In some embodiments, the communication subsystem 2624 may provide wired network connectivity (e.g., Ethernet®) in addition to or instead of a wireless interface.

[0259] The communication subsystem 2624 can receive and transmit data in various formats. For example, in some embodiments, the communication subsystem 2624 may receive input communications in formats such as structured data feeds and / or unstructured data feeds 2626, event streams 2628, and event updates 2630, in addition to other formats. For example, the communication subsystem 2624 may be configured to receive (or transmit) data feeds 2626 in real time from users of social media networks and / or other communication services such as web feeds like Twitter® feeds, Facebook® updates, Rich Site Summary (RSS) feeds, and / or real-time updates from one or more third-party sources.

[0260] In certain embodiments, the communication subsystem 2624 may be configured to receive data in the form of a continuous data stream, which may include an event stream 2628 and / or event update 2630 of real-time events that are inherently continuous or infinite and do not have a clear end. Examples of applications that generate continuous data may include, for example, sensor data applications, financial stock market boards, network performance measurement tools (e.g., network monitoring and traffic management applications), clickstream analysis tools, and automotive traffic monitoring.

[0261] The communication subsystem 2624 may be configured to transmit data from the computer system 2600 to other computer systems or networks. This data may be transmitted in various different formats, such as structured and / or unstructured data feeds 2626, event streams 2628, and event updates 2630, to one or more databases that can communicate with one or more streaming data source computers connected to the computer system 2600.

[0262] The computer system 2600 may be one of many types, including handheld portable devices (e.g., iPhone® cellular phone, iPad® computing tablet, PDA), wearable devices (e.g., Google Glass® head-mounted display), personal computers, workstations, mainframes, kiosks, server racks, or other data processing systems. Because the nature of computers and networks is constantly changing, the description of the computer system 2600 shown in Figure 26 is intended only as a specific example. Many other configurations with more or fewer components than the system shown in Figure 26 are possible. Those skilled in the art will recognize other embodiments and / or methods for realizing various embodiments based on the disclosures and teachings herein.

[0263] While specific embodiments have been described, various modifications, changes, alternative configurations, and equivalents are possible. The embodiments are not limited to operation within a specific data processing environment, but can freely operate within multiple data processing environments. Furthermore, while specific embodiments have been described using a specific set of transactions and steps, it should be apparent to those skilled in the art that this is not intended as limitation. Some flowcharts describe operations as sequential processes, but many of these operations can be performed in parallel or concurrently. Additionally, the order of operations may be re-specified. Processes may have additional steps not shown in the diagrams. The various features and aspects of the embodiments described above may be used individually or together.

[0264] Furthermore, while specific embodiments have been described using particular combinations of hardware and software, it should be understood that other combinations of hardware and software are also possible. A particular embodiment may be implemented using hardware alone, software alone, or a combination thereof. The various processes described herein may be implemented on the same processor or on separate processors of any combination thereof.

[0265] Where it is stated that a device, system, component, or module is configured to perform a particular operation or function, such configuration can be achieved, for example, by designing electronic circuits to perform the operation; by programming programmable electronic circuits (such as a microprocessor) to perform the operation; by executing computer instructions or code, or any combination thereof, stored in a non-temporary memory medium, or by executing a processor or core. Processes can communicate using a variety of techniques, including but not limited to conventional techniques for inter-process communication, different pairs of processes may use different techniques, and the same pair of processes may use different techniques at different times.

[0266] This disclosure provides specific details to ensure that the embodiments are fully understood. However, the embodiments can be implemented without these specific details. For example, well-known circuits, processes, algorithms, structures, and techniques may have ambiguous embodiments. To avoid this, unnecessary details are omitted. This specification provides only exemplary embodiments and is not intended to limit the scope, applicability, or configuration of other embodiments. Rather, the above description of embodiments provides a description that enables various embodiments to be realized by those skilled in the art. Various modifications are possible within the scope of the function and configuration of the elements.

[0267] Therefore, the specification and accompanying drawings should be considered illustrative rather than restrictive. However, it will be apparent that additions, reductions, deletions, and other modifications and alterations may be made thereto without departing from the broader spirit and scope set forth in the claims. Thus, specific embodiments have been described, but these are not intended to be limiting. Various variations and equivalents, as well as any relevant combination of the disclosed features, are within the scope of the accompanying claims.

Claims

1. It is a method, Steps to access the input utterance, The steps include generating an input feature vector from the aforementioned input utterance, The process includes the step of determining that the input utterance is unrelated to the plurality of skillbots, based on a classifier model configured to determine whether the utterance is related to at least one of the plurality of skillbots, each of the plurality of skillbots is configured to contribute to the dialogue. The aforementioned determination step is, The steps include comparing the input feature vector with multiple set representations of pre-generated training feature vectors, The method further includes the step of determining, based on the results of the comparison, that the input utterance is unrelated to the plurality of skillbots, and the method further includes A method comprising the step of outputting an instruction indicating that the input utterance cannot be addressed by the plurality of skillbots, in response to determining that the input utterance is irrelevant to the plurality of skillbots.

2. The plurality of set representations are clusters of the training feature vectors, or composite vectors of the training feature vectors, The aforementioned comparison step is, If the set representation is a cluster of the training feature vectors, the step of determining whether the input feature vectors are within the range of the clusters, or If the set representation is a composite feature vector of the training feature vectors, the step includes comparing the input vector with the composite feature vector, The step of determining, based on the results of the comparison, that the input utterance is unrelated to the multiple skillbots is: The method according to claim 1, further comprising the step of determining that the input utterance is unrelated to the plurality of skillbots if the input feature vector is not within the range of the cluster, or if the input feature vector is not similar to any of the composite feature vectors.

3. The method further includes the step of accessing a plurality of training utterances associated with the plurality of skillbots, the plurality of training utterances including the respective training utterances associated with each of the plurality of skillbots, and the method further includes The method includes the step of generating a plurality of training feature vectors from the plurality of training utterances, wherein the plurality of training feature vectors include the respective training feature vector associated with each of the plurality of skillbots, and the method further includes The method includes the step of generating multiple set representations of the multiple training feature vectors, wherein each set representation corresponds to a subset of the multiple training feature vectors, and the method further includes the step of generating multiple set representations of the multiple training feature vectors, The method according to claim 1 or 2, comprising the step of constructing the classifier model based on the plurality of set representations.

4. The step of generating the plurality of set representations of the plurality of training feature vectors includes the step of generating a plurality of clusters to which the plurality of training feature vectors are assigned, The method according to claim 3, wherein the step of configuring the classifier model to determine whether an utterance is related to or unrelated to the plurality of skillbots includes the steps of comparing an input feature vector for the input utterance with the plurality of set representations and determining that the input feature vector does not fall within the boundaries of the clusters.

5. A step of accessing a second input utterance as a second user input, The steps include generating a second input feature vector from the second input utterance, The steps include determining that the second input feature vector is within the range of the boundary of one of the clusters, The method according to claim 4, further comprising the step of transferring the second input utterance to a skillbot associated with the cluster for processing, based on the fact that the second input feature vector lies within the boundaries of the cluster.

6. The step of generating the multiple set representations of the multiple training feature vectors is: The steps include dividing the aforementioned multiple training utterances into multiple conversation categories, The step includes generating a plurality of composite feature vectors corresponding to the plurality of conversation categories, The method according to claim 3, wherein the step of generating the plurality of composite feature vectors includes the step of generating each composite feature vector as a set of training feature vectors of the plurality of training utterances in each of the plurality of conversation categories.

7. A step of accessing a second input utterance as a second user input, The steps include generating a second input feature vector from the second input utterance, The steps include determining that the second input feature vector is sufficiently similar to one of the multiple composite feature vectors, The method according to claim 6, further comprising the step of transferring the second input utterance to a skillbot associated with the composite feature vector for processing, based on the fact that the second input feature vector is sufficiently similar to the composite feature vector.

8. The method according to any one of claims 1 to 7, further comprising the step of processing the input utterance to perform an action in response to the input utterance.

9. It is a system, Includes a master bot configured to perform multiple actions, the multiple actions being: Accessing the input utterance, The process involves generating an input feature vector from the aforementioned input utterance, This includes determining that the input utterance is unrelated to any of the multiple skillbots based on a classifier model configured to determine whether the utterance is related to at least one of the multiple skillbots, and each of the multiple skillbots is configured to contribute to the dialogue. The aforementioned determination is, This involves comparing the aforementioned input feature vector with multiple set representations of pre-generated training feature vectors, The process includes determining, based on the results of the comparison, that the input utterance is unrelated to the plurality of skillbots, and further, A system that, in response to determining that the input utterance is unrelated to the plurality of skillbots, includes the operation of outputting an instruction indicating that the input utterance cannot be handled by the plurality of skillbots.

10. The plurality of set representations are clusters of the training feature vectors, or composite vectors of the training feature vectors, The above comparison is, When the set representation is a cluster of the training feature vectors, determine whether the input feature vectors are within the range of the clusters, or If the set representation is a composite feature vector of the training feature vectors, then the set representation includes comparing the input vector with the composite feature vector, In accordance with the results of the above comparison, determining that the input utterance is unrelated to the multiple skill bots means that The system according to claim 9, further comprising determining that the input utterance is unrelated to the plurality of skillbots if the input feature vector is not within the range of the cluster, or if the input feature vector is not similar to any of the composite feature vectors.

11. The system further includes a training system configured to perform multiple actions, wherein the multiple actions are The operation includes accessing a plurality of training utterances associated with the plurality of skillbots, wherein the plurality of training utterances include the respective training utterances associated with each of the plurality of skillbots, and the plurality of operations further include The process includes generating a plurality of training feature vectors from the plurality of training utterances, the plurality of training feature vectors including the respective training feature vector associated with each of the plurality of skillbots, and the plurality of actions further include The process includes generating multiple set representations of the multiple training feature vectors, each set representation of which corresponds to a subset of the multiple training feature vectors, and further includes generating multiple set representations of the multiple training feature vectors, each set representation of which corresponds to a subset of the multiple training feature vectors, and the process further includes The system according to claim 9 or 10, comprising configuring the classifier model based on the plurality of set representations.

12. Generating the multiple set representations of the multiple training feature vectors includes generating multiple clusters to which the multiple training feature vectors are assigned, The system according to claim 11, wherein configuring the classifier model to determine whether an utterance is related to or unrelated to the plurality of skillbots includes comparing an input feature vector for the input utterance with the plurality of set representations and determining that the input feature vector does not fall within the boundaries of the plurality of clusters.

13. The aforementioned multiple operations of the master bot further, Accessing a second input utterance as a second user input, The process involves generating a second input feature vector from the second input utterance, Determining that the second input feature vector is within the boundary range of one of the clusters, The system according to claim 12, further comprising, based on the fact that the second input feature vector lies within the boundaries of the cluster, transferring the second input utterance to a skillbot associated with the cluster for processing.

14. Generating the multiple set representations of the multiple training feature vectors is, Dividing the aforementioned multiple training utterances into multiple conversation categories, This includes generating multiple composite feature vectors corresponding to the multiple conversation categories, The system according to claim 11, wherein generating the plurality of composite feature vectors includes generating a composite feature vector for each of the plurality of conversation categories as a set of training feature vectors of the plurality of training utterances in the plurality of conversation categories.

15. The aforementioned multiple operations of the master bot further, Accessing a second input utterance as a second user input, The process involves generating a second input feature vector from the second input utterance, Determining that the second input feature vector is sufficiently similar to one of the multiple composite feature vectors, The system according to claim 14, further comprising, based on the fact that the second input feature vector is sufficiently similar to the composite feature vector, transferring the second input utterance to a skillbot associated with the composite feature vector for processing.

16. The system according to any one of claims 9 to 15, further comprising the skillbot, wherein the skillbot is configured to process the input utterance to perform an action in response to the input utterance.

17. A computer program for causing a processor to perform the method described in any one of claims 1 to 8.

Images