Keyword data augmentation tool for natural language processing

Abstract

A technique for keyword data augmentation for training a chatbot system in natural language processing is provided. In one particular aspect, a method is provided that includes receiving a training set of utterances for training a machine learning model to identify one or more intents for one or more utterances and augmenting the training set of utterances with out-of-domain (OOD) examples. The augmenting step includes identifying keywords within utterances in the training set of utterances, generating a set of OOD examples with the identified keywords, filtering out OOD examples from the set of OOD examples that have contexts substantially similar to those of the utterances in the training set of utterances, and incorporating the filtered set of OOD examples that do not include the OOD examples into the training set of utterances to generate an augmented training set of utterances. The machine learning model is then trained using the augmented training set of utterances.

Description

[Technical Field]

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS This application claims the benefit of and priority to U.S. Provisional Application No. 63 / 119,540, filed November 30, 2020, which claims the benefit of and priority to U.S. Non-Provisional Application No. 17 / 452,742, filed October 28, 2021, the entire contents of which are incorporated herein by reference for all purposes.

[0002] Field The present disclosure relates generally to chatbot systems, and more particularly to techniques for keyword data augmentation for training chatbot systems in natural language processing. [Background technology]

[0003] background Many users around the world reside on instant messaging or chat platforms to get immediate responses. Organizations often use these instant messaging or chat platforms to participate in live conversations with customers (or end users). However, hiring service personnel to participate in live communications with customers or end users can be very costly for organizations. Chatbots, or bots, have begun to be developed to simulate conversations with end users, especially over the Internet. End users can communicate with bots through messaging apps they already have installed and use. Intelligent bots, generally driven by artificial intelligence (AI), can communicate more intelligently and contextually in live conversations and thus enable more natural conversations between bots and end users for an improved conversational experience. Instead of end users learning a fixed set of keywords or commands that the bot knows how to respond to, intelligent bots may be able to understand the end user's intent based on the end user's utterances in natural language and respond accordingly.

[0004] However, building chatbots is difficult because these automated solutions require specific knowledge in a particular domain and the application of specific techniques that may only be within the capabilities of an expert developer. As part of building such a chatbot, a developer may first understand the needs of the enterprise and end users. The developer may then analyze and make decisions related to, for example, selecting a dataset to be used for analysis, preparing the input dataset for analysis (e.g., cleansing the data, extracting, formatting, and / or transforming data before analysis, performing data feature engineering, etc.), identifying an appropriate machine learning (ML) technique or model to perform the analysis, and improving the technique or model to improve results / outcomes based on feedback. The task of identifying an appropriate model may involve developing multiple models, possibly in parallel, iteratively testing and experimenting with these models, and then identifying a specific model for use. Furthermore, supervised learning-based solutions typically include a training phase followed by an application (i.e., inference) phase, and an iterative loop between the training and application phases. The developer may be responsible for carefully implementing and monitoring these phases to achieve an optimal solution. For example, to train an ML technique or model, accurate training data is needed to enable the algorithm to understand and learn the specific patterns or features (e.g., in the case of a chatbot, intent extraction and careful syntactic parsing, rather than simply raw language processing) that the ML technique or model will use to predict a desired outcome (e.g., inferring intent from an utterance). To ensure that the ML technique or model properly learns these patterns and features, a developer may be responsible for selecting, enriching, and optimizing the set of training data for the ML technique or model. Summary of the Invention

[0005] overview The presently disclosed technology generally relates to chatbots. More specifically, but not by way of limitation, the presently disclosed technology relates to techniques for keyword data augmentation of a set of training data for training a chatbot system in natural language processing. A chatbot can classify a user's utterances into various classes, such as the user's predefined intents. The chatbot's classifier may include a trained ML model that generates output (e.g., intents) based on input (e.g., user utterances). The user utterances may take the form of speech. In this case, the trained ML model can be understood as improving speech recognition, where speech recognition enables more accurate identification of user intents. If the training data used to train the trained ML model is inappropriate, the chatbot may more frequently determine the wrong intent. The presently disclosed technology can provide a keyword-augmented dataset for training an ML model so that the ML model is more robust to unrelated contexts and more accurately learns intent patterns or boundaries.

[0006] Various embodiments provide a computer-implemented method that includes receiving a training set of utterances for training a machine learning model to identify one or more intents for one or more utterances, and augmenting the training set of utterances with out-of-domain (OOD) examples. The augmenting includes identifying keywords within utterances in the training set of utterances, generating a set of OOD examples with the identified keywords, filtering out OOD examples from the set of OOD examples that have contexts substantially similar to contexts of utterances in the training set of utterances, and incorporating the set of OOD examples that do not include the filtered OOD examples into the training set of utterances to generate an augmented training set of utterances. The machine learning model is then trained using the augmented training set of utterances.

[0007] In some embodiments, the method further includes normalizing the training set of utterances and / or the set of OOD examples, where the normalizing includes (i) filtering out stop words identified as the identified keywords, (ii) lemmatizing all words in the training set of utterances, (iii) lemmatizing all words in the set of OOD examples with the identified keywords, or (iv) any combination thereof.

[0008] In some embodiments, the keywords are identified using term frequency-inverse document frequency (TF-IDF), word frequency, label names, explainability tools, or any combination thereof.

[0009] In some embodiments, the set of OOD examples is generated using a corpus, a vocabulary database, a text generation model, an adversarial attack model, or any combination thereof.

[0010] In some embodiments, the substantial similarity between the context of an OOD example and the context of an utterance in the training set of utterances is determined based on a distance measure to avoid collisions between classes.

[0011] In some embodiments, the method further includes deploying the trained machine learning model in a chatbot system.

[0012] In some embodiments, the keywords are words that, through training of machine learning models, have the potential to be associated with a particular ground truth intent.

[0013] In various embodiments, a computer-implemented method is provided, the computer-implemented method including: a chatbot system receiving utterances generated by a user interacting with the chatbot system; and classifying the utterances into intent categories corresponding to intents using a machine learning model deployed within the chatbot system, the machine learning model including a plurality of model parameters identified using training data, the training data including a training set of augmented utterances for training an intent classifier to identify one or more intents for one or more utterances, the training set of augmented utterances being artificially generated to include augmented utterances from the training set of utterances, keywords being identified from the training set of utterances and incorporated into out-of-domain (OOD) utterances having contexts substantially different from those of the utterances in the training set of utterances to generate the augmented utterances, the plurality of model parameters being identified using the training data based on maximizing or minimizing a cost function, the computer-implemented method further including using the machine learning model to output an intent based on the classification.

[0014] In various embodiments, a system is provided that includes one or more data processors and a non-transitory computer-readable storage medium that includes instructions that, when executed on the one or more data processors, cause the one or more data processors to perform some or all of one or more methods disclosed herein.

[0015] In various embodiments, a computer program product tangibly embodied in a non-transitory machine-readable storage medium is provided, the computer program product including instructions configured to cause one or more data processors to perform some or all of one or more of the methods disclosed herein.

[0016] The techniques described above and below can be implemented in several ways and in several contexts. Some example implementations and contexts are provided as described in more detail below and with reference to the accompanying figures. However, the following implementations and contexts are only a few of many. [Brief explanation of the drawings]

[0017] [Figure 1] FIG. 1 is a simplified block diagram illustrating a distributed environment incorporating an illustrative embodiment. [Figure 2] FIG. 1 is a simplified block diagram illustrating a computing system implementing a Masterbot, according to certain embodiments. [Figure 3] FIG. 1 is a simplified block diagram illustrating a computing system for implementing a skillbot, according to certain embodiments. [Figure 4] FIG. 1 is a simplified block diagram illustrating a chatbot training and deployment system, according to various embodiments. [Figure 5] FIG. 1 illustrates a process flow for augmenting a training dataset with keywords, according to various embodiments. [Figure 6] 1 is a simplified diagram illustrating a distributed system for implementing various embodiments. [Figure 7] FIG. 1 is a simplified block diagram illustrating one or more components of a system environment in which services provided by one or more components of a system of one embodiment may be offered as cloud services, according to various embodiments. [Figure 8] FIG. 1 illustrates an exemplary computer system that may be used to implement various embodiments. DETAILED DESCRIPTION OF THE INVENTION

[0018] Detailed Description In the following description, for purposes of explanation, specific details are set forth in order to provide a thorough understanding of particular embodiments. It will be apparent, however, that various embodiments may be practiced without these specific details. The figures and description are not intended to be limiting. The word "exemplary" is used herein to mean "serving as an example, instance, or illustration." Any embodiment or design described herein as "exemplary" is not necessarily to be construed as preferred or advantageous over other embodiments or designs.

[0019] introduction A digital assistant is an artificial intelligence-driven interface that helps users accomplish various tasks in natural language conversation. For each digital assistant, customers can assemble one or more skills. Skills (also referred to herein as chatbots, bots, or skillbots) are individual bots that focus on specific types of tasks, such as tracking inventory, submitting time cards, and creating expense reports. When an end user engages with a digital assistant, the digital assistant evaluates the end-user input and routes the conversation to or from the appropriate chatbot. Digital assistants can be made available to end users through various channels, such as Facebook Messenger, SKYPE MOBILE Messenger, or short message service (SMS). Channels carry chats back and forth from the end user to the digital assistant and its various chatbots over various messaging platforms. Channels may also support user agent escalation, event-driven conversations, and testing.

[0020] Intents enable a chatbot to understand what a user wants it to do. Intents can be thought of as instructions given by a user to a computer (e.g., in spoken form), with the accuracy with which the user's utterance (e.g., a command or instruction, possibly delivered as speech) is understood, corresponding to the accuracy with which the user's intent is captured. This directly relates to the appropriateness of the computer's response to the command or instruction, because the better the command or instruction is understood, the better the response from the computer. As used herein, an utterance or message may refer to a set of words (e.g., one or more sentences) exchanged during a conversation with a chatbot. An intent may be created by providing a name that indicates some user action (e.g., ordering a pizza) and compiling a set of actual user statements or utterances commonly associated with triggering that action. Because a chatbot's cognition is derived from these intents, each intent can be created and modified from a robust dataset (one to a dozen utterances) so that the chatbot can interpret ambiguous user input. A rich set of utterances enables a chatbot to understand what a user wants when it receives messages that mean the same thing but are expressed differently, such as "Ignore this order!" or "Cancel delivery!" Collectively, the intents and their associated utterances constitute the training corpus for a chatbot. By training a model with the corpus, customers can essentially transform the model into a reference tool for decomposing end-user input into a single intent. Customers can improve the cognitive acuity of their chatbot through cycles of intent testing and intent training.

[0021] However, building a chatbot that can determine an end user's intent based on user utterances is a challenging task, in part due to the subtleties and ambiguities of natural language, as well as the dimensionality of the input space (e.g., possible user utterances) and the size of the output space (number of intents). Specific examples of this difficulty arise from characteristics of natural language, such as employing euphemisms, synonyms, or ungrammatical language to express intents. For example, an utterance may express the intent to order a pizza without explicitly mentioning pizza, ordering, or delivery. For example, in the local dialect of a particular region, "pizza" is called "pie." These tendencies, such as imprecision or variability in natural language, create uncertainty and introduce confidence as a parameter for intent prediction, as opposed to explicitly indicating the intent by, for example, including keywords. Therefore, chatbots may need to be trained, monitored, debugged, and retrained to improve their performance and the user experience with them. Conventional systems provide training systems for training and retraining machine learning models for digital assistants or chatbots in spoken language understanding (SLU) and natural language processing (NLP). Traditionally, models used for chatbot systems are trained with "manufactured" utterances for arbitrary intents in NLP. For example, the utterance "Do you do price changes?" may be used to train a classifier model for the chatbot system to classify this type of utterance as the intent "Do you offer a price match." Training the model with manufactured utterances initially helps train the chatbot system to provide a service, and then the chatbot system may be retrained once it is deployed and begins to receive real utterances from users.

[0022] Traditional training of models for text classification begins with a training dataset of utterances labeled with ground truth intents. However, the training dataset of utterances may contain anchor words (referred to herein as "keywords") that, through training of a machine learning model, become associated with specific ground truth intents. Specifically, machine learning models may have a tendency to over- or over-confidence in individual keywords. These keywords can cause false positives even when used in a context significantly different from the training data. For example, some out-of-domain (OOD) test utterances have been observed to be misclassified under in-domain intents because the presence of certain keywords induces a strong learning attraction toward in-domain intents, as shown in Table 1.

[0023] [Table 1]

[0024] Therefore, a different approach is needed to address these issues. To enable a machine learning model to learn that keywords are important signals only when used in the correct context, the approach described herein attempts to teach the machine learning model this by augmenting the training data with OOD examples (negative data augmentation) in which the keywords exist in contexts different from the training dataset of utterances. The labels for these OOD examples would be for ground truth, such as the exact intent or an unresolved intent. As a result, the machine learning model will better generalize and learn that a keyword should only matter if it is used in a context similar to the training dataset of utterances. In various embodiments, a method is provided that includes receiving a training set of utterances for training an intent classifier (i.e., a machine learning model) to identify one or more intents for one or more utterances, and augmenting the training set of utterances with OOD examples. The expanding step includes identifying keywords within utterances from the training set of utterances, generating a set of OOD examples with the identified keywords, filtering out OOD examples from the set of OOD examples that have contexts substantially similar to contexts of the utterances from the training set of utterances, and incorporating the filtered set of OOD examples that do not include the OOD examples into the training set of utterances to generate an expanded training set of utterances. An intent classifier is then trained using the expanded training set of utterances.

[0025] Bots and Analytics Systems A bot (also referred to as a skill, chatbot, chatterbot, or talkbot) is a computer program that can conduct a conversation with an end user. Bots can generally respond to natural language messages (e.g., questions or comments) through a messaging application using natural language messages. Businesses can use one or more bot systems to communicate with end users through messaging applications. The messaging application, sometimes called a channel, can be the end user's preferred messaging application that the end user already has installed and is familiar with. Thus, end users do not need to download and install a new application to chat with a bot system. Messaging applications may include, for example, over-the-top (OTT) messaging channels (e.g., Facebook Messenger, Facebook WhatsApp, WeChat, Line, Kik, Telegram, Talk, Skype, Slack, or SMS), virtual private assistants (e.g., Amazon Dot, Echo, or Show, Google® Home, Apple HomePod, etc.), mobile and web app extensions that extend native or hybrid / responsive mobile apps or web applications with chat capabilities, or voice-based input (e.g., devices or apps with interfaces that use Siri, Cortana, Google Voice, or other voice input for interaction).

[0026] In some examples, a bot system may be associated with a uniform resource identifier (URI). The URI may identify the bot system using a string of characters. The URI may be used as a webhook for one or more messaging application systems. The URI may include, for example, a uniform resource locator (URL) or a uniform resource name (URN). The bot system may be designed to receive a message (e.g., a HyperText Transfer Protocol (HTTP) post call message) from the messaging application system. The HTTP post call message may be directed to the URI from the messaging application system. In some embodiments, the message may differ from the HTTP post call message. For example, the bot system may receive a message via short message service (SMS). While the description herein may refer to a communication received by the bot system as a message, it should be understood that the message may be an HTTP post call message, an SMS message, or any other type of communication between two systems.

[0027] End users may interact with bot systems through conversational interactions (sometimes called conversational user interfaces (UIs)), much like interactions between people. In some cases, an interaction may involve the end user saying "Hello" to the bot, and the bot responding with "Hi" and asking the end user how it can assist the end user. In some cases, an interaction may also be a transactional interaction with a banking bot, such as transferring money from one account to another, an informational interaction with an HR bot, such as reconciling a vacation balance, or an interaction with a retail bot, such as describing a purchase return or a technical support request.

[0028] In some embodiments, the bot system may intelligently handle end-user interactions without interaction with a bot system administrator or developer. For example, an end user may send one or more messages to the bot system to achieve a desired goal. The messages may include specific content, such as text, emojis, audio, images, video, or other methods of conveying a message. In some embodiments, the bot system may convert the content into a standardized format (e.g., a representational state transfer (REST) ​​call to an enterprise service with appropriate parameters) and generate a natural language response. The bot system may also prompt the end user for additional input parameters or request other additional information. In some embodiments, the bot system may also initiate communication with the end user rather than passively responding to end-user utterances. Various techniques are described herein for identifying explicit invocations of the bot system and determining input for the invoked bot system. In certain embodiments, explicit invocation analysis is performed by a master bot based on detecting a call name in the utterance. In response to detecting the call name, the utterance may be refined for input to a skill bot associated with the call name.

[0029] A conversation with a bot may follow a particular conversational flow that includes multiple states. The flow may define what happens next based on input. In some embodiments, a bot system may be implemented using a state machine that includes user-defined states (e.g., end-user intents) and actions to be taken in or depending on the states. A conversation may take different paths based on end-user input, which may affect the decisions the bot makes about the flow. For example, in each state, based on the end-user input or utterances, the bot may determine the end-user's intent and determine the appropriate action to take next. As used herein, and in the context of utterances, the term "intent" refers to the intent of the user who gave the utterance. For example, a user may intend to engage a bot in a conversation to order a pizza; thus, the user's intent may be expressed by the utterance "order a pizza." A user's intent may be directed to a specific task the user wants the chatbot to perform on their behalf. Thus, an utterance may be expressed as a question, command, request, etc. that reflects the user's intent. An intent may include a goal that an end user may wish to achieve.

[0030] In the context of configuring a chatbot, the term “intent” is used herein to refer to configuration information for mapping a user’s utterance to a specific task / action or category of task / action that the chatbot can perform. To distinguish between an utterance intent (i.e., a user’s intent) and a chatbot’s intent, the latter may be referred to herein as a “bot intent.” A bot intent may include a set of one or more utterances associated with the intent. For example, an intent related to ordering a pizza may have various permutations of utterances expressing a desire to place a pizza order. These associated utterances can be used to train the chatbot’s intent classifier, which can then determine whether an input utterance from a user matches the pizza-ordering intent. A bot intent may be associated with one or more dialog flows for initiating a conversation with a user in a particular state. For example, the first message for a pizza-ordering intent may be the question, “What kind of pizza would you like?” In addition to the associated utterance, a bot intent may further include a named entity related to the intent. For example, a pizza ordering intent may include variables or parameters used to perform the task of ordering a pizza, such as topping 1, topping 2, pizza type, pizza size, pizza quantity, etc. The values ​​of the entities are typically obtained through conversation with the user.

[0031] FIG. 1 is a simplified block diagram of an environment 100 incorporating a chatbot system according to certain embodiments. The environment 100 includes a digital assistant builder platform (DABP) 102, which enables users of the DABP 102 to create and deploy digital assistant or chatbot systems. The DABP 102 can be used to create one or more digital assistant (or DA) or chatbot systems. For example, as shown in FIG. 1, a user 104 representing a particular business can use the DABP 102 to create and deploy a digital assistant 106 for users of the particular business. For example, a bank can use the DABP 102 to create one or more digital assistants for use by the bank's customers. Multiple businesses can use the same DABP 102 platform to create digital assistants. As another example, the owner of a restaurant (e.g., a pizza shop) may use the DABP 102 to create and deploy a digital assistant that enables customers of the restaurant to order food (e.g., order pizza).

[0032] For purposes of this disclosure, a "digital assistant" is an entity that helps a user of the digital assistant accomplish various tasks through natural language conversation. A digital assistant may be implemented solely using software (e.g., a digital assistant is a digital entity implemented using programs, codes, or instructions executable by one or more processors), using hardware, or using a combination of hardware and software. A digital assistant may be embodied or implemented in various physical systems or devices, such as a computer, a mobile phone, a watch, an appliance, a vehicle, etc. A digital assistant may also be referred to as a chatbot system. Thus, for purposes of this disclosure, the terms digital assistant and chatbot system are interchangeable.

[0033] A digital assistant, such as digital assistant 106 built using DABP 102, can be used to perform various tasks through natural language-based conversations between the digital assistant and its user 108. As part of the conversation, the user may provide one or more user inputs 110 to the digital assistant 106 and obtain responses 112 from the digital assistant 106. A conversation may include one or more of the inputs 110 and the responses 112. Through these conversations, the user can request one or more tasks to be performed by the digital assistant 106, and in response, the digital assistant 106 is configured to perform the user-requested task and respond to the user with an appropriate response.

[0034] User input 110 is generally in the form of natural language and is referred to as an utterance. User utterance 110 can be in text form, such as when a user types a sentence, a question, a piece of text, or even a single word and provides it as input to the digital assistant 106. In some embodiments, user utterance 110 can be in the form of voice input or speech, such as when a user says or speaks something that is provided as input to the digital assistant 106. The utterance is typically in the language spoken by the user 108. For example, the utterance may be in English or some other language. If the utterance is in voice form, the voice input is converted into textual speech in that particular language, and the textual speech is then processed by the digital assistant 106. Various speech-to-text processing techniques may be used to convert the voice or auditory input into textual speech, which is then processed by the digital assistant 106. In some embodiments, the speech-to-text conversion may be performed by the digital assistant 106 itself.

[0035] The utterance, which may be a text utterance or a voice utterance, may be a fragment, a sentence, multiple sentences, one or more words, one or more questions, a combination of the aforementioned types, etc. The digital assistant 106 is configured to apply natural language understanding (NLU) techniques to the utterance to understand the meaning of the user input. As part of the NLU processing on the utterance, the digital assistant 106 is configured to perform processing to understand the meaning of the utterance, which involves identifying one or more intents and one or more entities corresponding to the utterance. Upon understanding the meaning of the utterance, the digital assistant 106 may perform one or more actions or operations according to the understood meaning or intent. For purposes of this disclosure, it is assumed that the utterance is a text utterance provided directly by a user 108 of the digital assistant 106 or is the result of converting an input voice utterance into text form. However, this is not intended to be limiting or restrictive in any way.

[0036] For example, user 108's input may request that a pizza be ordered by providing an utterance such as, "I would like to order a pizza." Upon receiving such an utterance, digital assistant 106 is configured to understand the meaning of the utterance and take appropriate action. The appropriate action may include responding to the user with a question requesting user input regarding, for example, the type of pizza the user wants to order, the size of the pizza, any toppings on the pizza, etc. The responses provided by digital assistant 106 may also be in natural language format, typically in the same language as the input utterance. As part of generating these responses, digital assistant 106 may perform natural language generation (NLG). Through a conversation between the user and digital assistant 106, for the user to order a pizza, the digital assistant may guide the user to provide all necessary information to order the pizza and then, at the end of the conversation, have the user order the pizza. Digital assistant 106 may end the conversation by outputting information to the user indicating that the pizza has been ordered.

[0037] At a conceptual level, digital assistant 106 performs various processes in response to utterances received from a user. In some embodiments, this processing involves a series of processing steps or a pipeline of processing steps, including, for example, understanding the meaning of the input utterance (sometimes referred to as natural language understanding (NLU)), determining an action to be performed in response to the utterance, executing the action as appropriate, generating a response to be output to the user in response to the user utterance, and outputting the response to the user. NLU processing may include parsing the received input utterance to understand the structure and meaning of the utterance and refining and restructuring the utterance to develop a more understandable form (e.g., a logical form) or structure for the utterance. Generating a response may include using NLG technology.

[0038] NLU processing performed by a digital assistant, such as digital assistant 106, may include various NLP-related processes, such as sentence analysis (e.g., tokenization, lemmatization, identifying part-of-speech tags for sentences, identifying named entities in sentences, generating dependency trees to represent sentence structure, dividing sentences into clauses, analyzing individual clauses, resolving anaphora, performing chunking, etc.). In particular embodiments, the NLU processing, or portions thereof, is performed by digital assistant 106 itself. In some other embodiments, digital assistant 106 may use other resources to perform portions of the NLU processing. For example, the syntax and structure of an input spoken sentence may be identified by processing the sentence with a parser, a part-of-speech tagger, and / or a named entity recognizer. In one implementation, for English, a parser, a part-of-speech tagger, and a named entity recognizer, such as those provided by the Stanford Natural Language Processing (NLP) Group, are used to analyze sentence structure and syntax. These are provided as part of the Stanford CoreNLP toolkit.

[0039] Although various examples provided in this disclosure show utterances in English, this is meant to be exemplary only. In particular embodiments, the digital assistant 106 can also process utterances in languages ​​other than English. The digital assistant 106 may provide subsystems (e.g., components that implement NLU functionality) configured to perform processing for various languages. These subsystems may be implemented as pluggable units that can be invoked using service calls from the NLU core server. This makes NLU processing flexible and extensible for each language, including allowing for various orders of processing. Language packs may be provided for individual languages, and the language packs can register a list of subsystems that can be served from the NLU core server.

[0040] 1 can be available or accessible to its user 108 through a variety of different channels, such as, but not limited to, through a particular application, through social media platforms, through various messaging services and applications, and other applications or channels. A single digital assistant may have several channels configured for it, so that it can run on and be accessed by different services simultaneously.

[0041] A digital assistant or chatbot system typically includes or is associated with one or more skills. In particular embodiments, these skills are individual chatbots (referred to as skillbots) that are configured to interact with a user and perform specific types of tasks, such as tracking inventory, submitting a timecard, creating an expense report, ordering food, checking a bank account, making a reservation, purchasing a widget, etc. For example, in the embodiment shown in FIG. 1 , the digital assistant or chatbot system 106 includes skills 116-1, 116-2, etc. For purposes of this disclosure, the term "skill" is used synonymously with the term "skillbot."

[0042] Each skill associated with a digital assistant helps a user of the digital assistant complete a task through a conversation with the user, where the conversation may include a combination of text or audio input provided by the user and responses provided by the skill bot. These responses may take the form of text or audio messages to the user and / or simple user interface elements (e.g., selection lists) presented to the user for selection.

[0043] There are various ways in which skills or skillbots can be associated with or added to a digital assistant. In some cases, skillbots may be developed by a company and then added to a digital assistant using DABP 102. In other examples, skillbots may be developed and created using DABP 102 and then added to a digital assistant created using DABP 102. In yet other examples, DABP 102 provides an online digital store (referred to as a "skill store") that offers multiple skills aimed at a wide range of tasks. Skills offered through the skill store may also expose various cloud services. To add a skill to a digital assistant created using DABP 102, a user of DABP 102 may access the skill store via DABP 102, select the desired skill, and indicate that the selected skill be added to the digital assistant created using DABP 102. Skills from the skill store can be added to a digital assistant as is or in modified form (e.g., a user of DABP102 may select and clone a particular skillbot provided by the skill store, customize or modify the selected skillbot, and then add the modified skillbot to a digital assistant created with DABP102).

[0044] A variety of different architectures may be used to implement a digital assistant or chatbot system. For example, in certain embodiments, a digital assistant created and deployed using DABP 102 may be implemented using a masterbot / child (or sub)bot paradigm or architecture. According to this paradigm, a digital assistant is implemented as a masterbot that interacts with one or more child bots, which are skillbots. For example, in the embodiment shown in FIG. 1, digital assistant 106 includes masterbot 114 and skillbots 116-1, 116-2, etc., that are child bots of masterbot 114. In certain embodiments, digital assistant 106 itself may act as a masterbot.

[0045] A digital assistant implemented according to the master-child bot architecture allows users of the digital assistant to interact with multiple skills through a unified user interface, i.e., through a master bot. When a user engages with the digital assistant, user input is received by the master bot. The master bot then performs processing to determine the meaning of the user input utterance. The master bot then determines whether the task requested by the user in the utterance can be handled by the master bot itself. If not, the master bot selects an appropriate skill bot to handle the user request and routes the conversation to the selected skill bot. This allows users to converse with the digital assistant through a common, single interface while still providing the ability to use several skill bots configured to perform specific tasks. For example, in a digital assistant developed for an enterprise, the digital assistant's master bot may interface with skill bots with specific capabilities, such as a CRM bot for performing functions related to customer relationship management (CRM), an ERP bot for performing functions related to enterprise resource planning (ERP), and an HCM bot for performing functions related to human capital management (HCM). In this way, the end user or consumer of the digital assistant only needs to know how to access the digital assistant through a common master bot interface, and behind the scenes, multiple skill bots are provided to handle user requests.

[0046] In certain embodiments, in a masterbot / childbot infrastructure, the masterbot is configured to recognize an available list of skillbots. The masterbot may have access to metadata identifying various available skillbots and, for each skillbot, each skillbot's capabilities, including the tasks that can be performed by each skillbot. Upon receiving a user request in the form of an utterance, the masterbot is configured to identify or predict, from multiple available skillbots, a specific skillbot that can best accommodate or process the user request. The masterbot then routes the utterance (or a portion of the utterance) to that specific skillbot for further processing. Thus, control flows from the masterbot to the skillbot. A masterbot can support multiple input and output channels. In certain embodiments, routing can be performed with the help of processing performed by one or more available skillbots. For example, as described below, a skillbot can be trained to infer the intent of an utterance and determine whether the inferred intent matches the intent for which the skillbot is configured. Thus, the routing performed by the masterbot may involve the skillbot communicating to the masterbot an indication of whether the skillbot has been configured with an appropriate intent to process the utterance.

[0047] 1 illustrates a digital assistant 106 with a masterbot 114 and skillbots 116-1, 116-2, and 116-3, but this is not intended to be limiting. The digital assistant may include various other components (e.g., other systems and subsystems) that provide the functionality of the digital assistant. These systems and subsystems may be realized solely in software (e.g., code, instructions stored on a computer-readable medium and executable by one or more processors), solely in hardware, or in an implementation using a combination of software and hardware.

[0048] DABP 102 provides infrastructure and various services and features that enable users of DABP 102 to create digital assistants, including one or more skillbots associated with the digital assistant. In some cases, a skillbot can be created by cloning an existing skillbot, for example, by cloning a skillbot provided by a skill store. As described above, DABP 102 provides a skill store or skill catalog that offers multiple skillbots for performing various tasks. Users of DABP 102 can clone a skillbot from the skill store. They may modify or customize the cloned skillbot as needed. In some other cases, users of DABP 102 created a skillbot from scratch using tools and services provided by DABP 102. As described above, the skill store or skill catalog provided by DABP 102 may offer multiple skillbots for performing various tasks.

[0049] In certain embodiments, at one high level, creating or customizing a skillbot includes the following steps: (1) Steps to configure the settings for a new skill bot (2) Configuring one or more intents for the skill bot (3) configuring one or more entities with respect to one or more intents; (4) Steps to train the skill bot (5) Steps to create a dialogue flow for your skill bot (6) Add custom components to your skill bot as needed (7) Test and deploy your skill bot Each of the above steps will be briefly described below.

[0050] (1) Configuring settings for a new skillbot. Various settings may be configured for a skillbot. For example, a skillbot designer can specify one or more call names for the skillbot being created. These call names can then be used by a user of the digital assistant to explicitly call the skillbot. For example, a user can enter a call name in the user's utterance to explicitly call the corresponding skillbot.

[0051] (2) Setting one or more intents and associated example utterances for the skill bot. A skill bot designer specifies one or more intents (also called bot intents) for the skill bot being created. The skill bot is then trained based on these specified intents. These intents represent categories or classes that the skill bot is trained to infer about input utterances. Upon receiving an utterance, the trained skill bot infers an intent for the utterance, and the inferred intent is selected from a predefined set of intents used to train the skill bot. The skill bot then takes an appropriate action based on the inferred intent for the utterance. In some cases, the intents for a skill bot represent tasks that the skill bot can perform for a user of the digital assistant. Each intent is given an intent identifier or intent name. For example, for a skill bot trained for banking, the specified intents for the skill bot may include "Check Balance," "Transfer Money," "Deposit Check," etc.

[0052] For each intent defined for a skillbot, the skillbot designer may also provide one or more example utterances that represent and exemplify that intent. These example utterances are intended to represent utterances a user might input to the skillbot for that intent. For example, for a balance inquiry intent, example utterances might include, "What's my savings account balance?", "How much is in my checking account?", "How much money do I have in my account?", etc. Thus, various permutations of typical user utterances may be specified as example utterances for the intent.

[0053] The intents and their associated example utterances are used as training data to train the skill bot. A variety of different training techniques may be used. This training results in a predictive model that is configured to take an utterance as input and output an intent inferred for the utterance by the predictive model. In some cases, the input utterance is provided to an intent analysis engine that is configured to predict or infer an intent for the input utterance using the trained model. The skill bot may then take one or more actions based on the inferred intent.

[0054] (3) Configuring entities for one or more intents of the skill bot. In some cases, additional context may be required to enable the skill bot to respond appropriately to a user utterance. For example, there may be situations where a user input utterance results in the same intent in the skill bot. For example, in the above example, the utterances "What's my savings account balance?" and "How much is in my checking account?" both result in the same balance inquiry intent, but these utterances are different requests asking for different things. To disambiguate such requests, one or more entities are added to the intent. Using the banking skill bot example, an entity called AccountType that defines values ​​called "checking" and "savings" may enable the skill bot to parse the user request and respond appropriately. In the above example, the utterances result in the same intent, but the values ​​associated with the AccountType entity are different for the two utterances. This allows the skill bot to perform possibly different actions for two utterances, even though the two utterances result in the same intent. One or more entities can be specified for a particular intent configured for the skill bot. Thus, entities are used to add context to the intent itself. Entities help to more fully describe the intent, enabling the skill bot to complete the user request.

[0055] In certain embodiments, there are two types of entities: (a) built-in entities provided by DABP 102, and (2) custom entities that can be specified by a skill bot designer. Built-in entities are general-purpose entities that can be used with a wide variety of bots. Examples of built-in entities include, but are not limited to, entities related to time, date, address, number, email address, duration, circulation period, currency, phone number, URL, etc. Custom entities are used for more customized applications. For example, in a banking skill, an AccountType entity may be defined by a skill bot designer to enable various banking transactions by checking user input for keywords such as checking, saving, and credit card.

[0056] (4) Training the skillbot. The skillbot is configured to receive user input in the form of utterances, parse or process the received input, and identify or select an intent associated with the received user input. As described above, the skillbot must be trained for this. In certain embodiments, the skillbot is trained based on intents configured for the skillbot and example utterances associated with those intents (collectively, training data), thereby enabling the skillbot to transform user input utterances into one of the skillbot's configured intents. In certain embodiments, the skillbot uses a predictive model that is trained with the training data to enable the skillbot to identify what the user is saying (or, in some cases, what they are trying to say). DABP 102 provides a variety of different training techniques that can be used by the skillbot designer to train the skillbot, including various machine learning-based training techniques, rule-based training techniques, and / or combinations thereof. In certain embodiments, a portion (e.g., 80%) of the training data is used to train the skillbot model, and another portion (e.g., the remaining 20%) is used to test or validate the model. Once trained, the trained model (sometimes called a trained skill bot) can then be used to process and respond to user utterances. In certain cases, a user utterance may be a question that requires only a single answer and no further conversation. To address such situations, a Q&A (Question and Answer) intent may be defined for the skill bot. This allows the skill bot to output a response to a user request without having to update the dialog definition. A Q&A intent is generated similarly to a regular intent. The dialog flow for a Q&A intent may differ from the dialog flow for a regular intent.

[0057] (5) Creating a dialog flow for the skill bot. The dialog flow specified for the skill bot describes how the skill bot reacts as various intents for the skill bot are resolved in response to received user input. The dialog flow defines the behavior or actions the skill bot will take, such as how the skill bot responds to user utterances, how the skill bot prompts the user for input, and how the skill bot returns data. The dialog flow is like a flowchart that the skill bot follows. Skill bot designers specify the dialog flow using a language such as Markdown. In certain embodiments, a version of YAML called OBotML may be used to specify the dialog flow for the skill bot. The dialog flow definition for the skill bot serves as a model for the conversation itself, allowing skill bot designers to choreograph interactions between the skill bot and users that the skill bot serves.

[0058] In certain embodiments, the dialog flow definition for a skill bot includes three sections: (a) Context Section (b) Default transition section (c) State section.

[0059] Context Section. In the context section, the skill bot designer can define variables used in the conversation flow. Other variables that can be named in the context section include, but are not limited to, variables for error handling, variables for built-in or custom entities, user variables that allow the skill bot to recognize and persist user preferences, etc.

[0060] Default Transition Section. Transitions for a skill bot can be defined in the dialog flow state section or the default transition section. Transitions defined in the default transition section act as fallbacks and are triggered when there is no applicable transition defined in a state or when the conditions required to trigger a state transition cannot be met. The default transition section can be used to define routing that allows the skill bot to gracefully handle unexpected user actions.

[0061] State Section. Dialog flow and its associated behavior are defined as a series of temporary states that govern the logic within the dialog flow. Each state node in a dialog flow definition names a component that provides the functionality needed at that point in the dialog. In this way, you build states around components. States contain component-specific characteristics and define transitions to other states that are triggered after the component executes.

[0062] Special case scenarios can be handled using the state section. For example, you might want to give a user the option to temporarily leave a first skill they're engaged in and do something with a second skill within the digital assistant. For example, if a user is engaged in a conversation with a shopping skill (e.g., the user has made some selections for a purchase), the user might want to jump to a banking skill (e.g., the user might want to verify that they have enough money for the purchase) and then return to the shopping skill to complete the user's order. To address this, an action in a first skill can be configured to initiate an interaction with a second, different skill in the same digital assistant and then return to the original flow.

[0063] (6) Adding custom components to the skill bot. As described above, a state specified in a dialog flow for a skill bot names a component that provides the necessary functionality corresponding to that state. The component enables the skill bot to perform the function. In certain embodiments, DABP 102 provides a set of pre-configured components for performing a wide range of functions. A skill bot designer can select one or more of these pre-configured components and associate them with states in the dialog flow for the skill bot. A skill bot designer can also create custom or new components using tools provided by DABP 102 and associate the custom components with one or more states in the dialog flow for the skill bot.

[0064] (7) Testing and Deploying Skillbots. DABP 102 provides several features that allow skillbot designers to test the skillbots they are developing. The skillbots can then be deployed and included in the digital assistant.

[0065] While the above description describes how to create a skillbot, similar techniques may be used to create a digital assistant (or masterbot). At the masterbot or digital assistant level, built-in system intents may be configured for the digital assistant. These built-in system intents are used to identify common tasks that the digital assistant itself (i.e., the masterbot) can handle without invoking a skillbot associated with the digital assistant. Examples of system intents defined for a masterbot include: (1) Exit: Applicable when a user signals in the digital assistant that they want to end the current conversation or context; (2) Help: Applicable when a user asks for help or direction; and (3) UnresolvedIntent: Applicable to user input that does not adequately match the Exit and Help intents. The digital assistant also stores information about one or more skillbots associated with the digital assistant. This information allows the masterbot to select a specific skillbot to process an utterance.

[0066] At the masterbot or digital assistant level, when a user inputs a phrase or utterance into the digital assistant, the digital assistant is configured to process and determine how to route the utterance and associated conversation. The digital assistant makes this determination using a routing model, which can be rule-based, AI-based, or a combination thereof. The digital assistant uses the routing model to determine whether the conversation corresponding to the user input utterance should be routed to a specific skill for processing, handled by the digital assistant or masterbot itself according to built-in system intents, or handled as a different state in the current conversation flow.

[0067] In certain embodiments, as part of this processing, the digital assistant determines whether the user input utterance explicitly identifies a skill bot using its invocation name. If an invocation name is present in the user input, it is treated as an explicit invocation of the skill bot corresponding to the invocation name. In such a scenario, the digital assistant may route the user input to the explicitly invoked skill bot for further processing. In the absence of a specific invocation or explicit invocation, in certain embodiments, the digital assistant evaluates the received user input utterance and calculates confidence scores for the system intents and skill bots associated with the digital assistant. The calculated scores for the skill bots or system intents represent the likelihood that the user input represents a task that the skill bot is configured to perform or represents a system intent. System intents or skill bots whose associated calculated confidence scores exceed a threshold (e.g., a Confidence Threshold routing parameter) are selected as candidates for further evaluation. The digital assistant then selects a specific system intent or skill bot from the identified candidates for further processing of the user input utterance. In certain embodiments, after one or more skill bots are identified as candidates, the intents associated with those candidate skills are evaluated (according to the intent model for each skill), and a confidence score is determined for each intent. Generally, intents with a confidence score above a threshold (e.g., 70%) are treated as candidate intents. If a particular skill bot is selected, the user utterance is routed to that skill bot for further processing. If a system intent is selected, one or more actions are performed by the master bot itself according to the selected system intent.

[0068] FIG. 2 is a simplified block diagram of a Masterbot (MB) system 200, according to certain embodiments. The MB system 200 can be implemented solely in software, solely in hardware, or a combination of hardware and software. The MB system 200 includes a pre-processing subsystem 210, a multiple intent subsystem (MIS) 220, an explicit invocation subsystem (EIS) 230, a skillbot invoker 240, and a data store 250. The MB system 200 shown in FIG. 2 is merely an example of the configuration of components in a Masterbot. Those skilled in the art will recognize many possible variations, alternatives, and modifications. For example, in some implementations, the MB system 200 may have more or fewer systems or components than those shown in FIG. 2, may combine two or more subsystems, or may have subsystems in a different configuration or arrangement.

[0069] The pre-processing subsystem 210 receives an utterance "A" 202 from a user and processes the utterance through a language detector 212 and a language parser 214. As described above, the utterance may be provided in a variety of ways, including as audio or text. The utterance 202 may be a fragment, a complete sentence, multiple sentences, etc. The utterance 202 may include punctuation. For example, if the utterance 202 is provided as audio, the pre-processing subsystem 210 may convert the audio to text using a speech-to-text converter (not shown), which inserts punctuation, e.g., commas, semicolons, periods, etc., into the resulting text.

[0070] The language detection unit 212 detects the language of the utterance 202 based on the text of the utterance 202. Because each language has its own grammar and semantics, the way in which the utterance 202 is processed depends on the language. Language differences are taken into account when analyzing the syntax and structure of the utterance.

[0071] The language parser 214 parses the utterance 202 to extract part-of-speech (POS) tags for individual linguistic units (e.g., words) within the utterance 202. POS tags include, for example, nouns (NN), pronouns (PN), verbs (VB), etc. The language parser 214 may also tokenize the linguistic units of the utterance 202 (e.g., to convert each word into a separate token) and lemmatize the words. A lemma is the primary form of a set of words represented in a dictionary (e.g., "run" is the lemma for run, runs, ran, running, etc.). Other types of preprocessing the language parser 214 can perform include chunking complex expressions, e.g., combining "credit" and "card" into a single expression "credit_card." The language parser 214 may also identify relationships between words within the utterance 202. For example, in some embodiments, language parser 214 generates a dependency tree that indicates which parts of the utterance (e.g., particular nouns) are direct objects, which parts of the utterance are prepositions, etc. The results of the processing performed by language parser 214 form extracted information 205, which, along with utterance 202 itself, is provided as input to MIS 220.

[0072] As described above, utterance 202 may include multiple sentences. For purposes of detecting multiple intents and explicit invocations, utterance 202 may be treated as a single unit even if it includes multiple sentences. However, in particular embodiments, preprocessing may be performed, for example, by preprocessing subsystem 210, to identify single sentences within multiple sentences for combined intent analysis and explicit invocation analysis. Generally, the results produced by MIS 220 and EIS 230 are substantially the same whether utterance 202 is processed at the individual sentence level or as a single unit including multiple sentences.

[0073] The MIS 220 determines whether the utterance 202 expresses multiple intents. While the MIS 220 can detect the presence of multiple intents in the utterance 202, the processing performed by the MIS 220 does not involve determining whether the intent of the utterance 202 matches any intent configured for the bot. Instead, the processing for determining whether the intent of the utterance 202 matches a bot intent may be performed by the intent classifier 242 of the MB system 200 (e.g., as shown in the embodiment of FIG. 3) or by an intent classifier of a skill bot. The processing performed by the MIS 220 assumes that a bot (e.g., a particular skill bot or the master bot itself) exists that can process the utterance 202. Thus, the processing performed by the MIS 220 does not require knowledge of what bots are in the chatbot system (e.g., the identities of the skill bots registered with the master bot) or what intents are configured for a particular bot.

[0074] To determine that utterance 202 contains multiple intents, MIS 220 applies one or more rules from a set of rules 252 in data store 250. The rules applied to utterance 202 depend on the language of utterance 202 and may include sentence patterns that indicate the presence of multiple intents. For example, a sentence pattern may include a conjunction connecting two parts of a sentence (e.g., coordinates), both parts corresponding to separate intents. If utterance 202 matches a sentence pattern, it can be inferred that utterance 202 represents multiple intents. Note that an utterance with multiple intents does not necessarily have different intents (e.g., intents directed to different bots or different intents within the same bot). Instead, the utterance may have separate instances of the same intent, such as "order a pizza using payment account X, then order a pizza using payment account Y."

[0075] As part of determining that utterance 202 represents multiple intents, MIS 220 also determines what portions of utterance 202 are associated with each intent. For each intent expressed in the multiple-intent utterance, MIS 220 constructs a new utterance for separate processing in place of the original utterance, e.g., utterance “B” 206 and utterance “C” 208, as shown in FIG. 2 . Thus, original utterance 202 may be split into two or more separate utterances that are handled one at a time. MIS 220 determines which of the two or more utterances should be processed first using extracted information 205 and / or from an analysis of utterance 202 itself. For example, MIS 220 may determine that utterance 202 includes a marker word indicating that a particular intent should be handled first. The newly formed utterance corresponding to this particular intent (e.g., one of utterance 206 or utterance 208) will be sent first for further processing by EIS 230. After the conversation triggered by the first utterance has ended (or been temporarily interrupted), the next highest priority utterance (e.g., the other of utterance 206 or utterance 208) may then be sent to EIS 230 for processing.

[0076] The EIS 230 determines whether the utterance it receives (e.g., utterance 206 or utterance 208) includes a skillbot's invocation name. In particular embodiments, each skillbot in the chatbot system is assigned a unique invocation name that distinguishes the skillbot from other skillbots in the chatbot system. A list of invocation names can be maintained in the data store 250 as part of the skillbot information 254. When the utterance includes words that match the invocation name, the utterance is considered to be an explicit invocation. If the bot is not explicitly invoked, the utterance received by the EIS 230 is considered an implicit invocation utterance 234 and is input to the masterbot's intent classifier (e.g., intent classifier 242) to determine which bot to use to process the utterance. In some examples, the intent classifier 242 will determine that the masterbot should process the implicit invocation utterance. In other examples, the intent classifier 242 will determine which skillbot to route the utterance to for processing.

[0077] The explicit call functionality provided by EIS 230 has several advantages. It can reduce the amount of processing that a masterbot must perform. For example, if there is an explicit call, the masterbot may not have to perform any intent classification analysis (e.g., using intent classifier 242) or may have to perform reduced intent classification analysis to select a skillbot. Thus, explicit call analysis may enable the selection of a specific skillbot without relying on intent classification analysis.

[0078] There may also be situations where there is overlap in functionality among multiple skillbots. This can occur, for example, when the intents handled by two skillbots overlap or are very close to each other. In such situations, it may be difficult for the masterbot to identify which of the multiple skillbots to select based solely on intent classification analysis. In such scenarios, explicit invocation disambiguates the specific skillbot to be used.

[0079] In addition to determining that an utterance is an explicit invocation, EIS 230 is responsible for determining whether any portion of the utterance should be used as input to the explicitly invoked skillbot. In particular, EIS 230 may determine whether a portion of the utterance is not associated with an invocation. EIS 230 may make this determination through analysis of the utterance and / or analysis of extracted information 205. Instead of sending the entire utterance received by EIS 230, EIS 230 may send the portion of the utterance that is not associated with an invocation to the invoked skillbot. In some examples, the input to the invoked skillbot is formed simply by removing any portion of the utterance that is associated with an invocation. For example, "I would like to order a pizza using Pizza Bot" may be shortened to "I would like to order a pizza" because "using Pizza Bot" pertains to the invocation of the Pizzabot but not to any processing performed by the Pizzabot. In some examples, EIS 230 may reformat the portion to be sent to the invoked bot, for example, to form a complete sentence. Thus, the EIS 230 determines not only that there is an explicit call, but also what to send to the skill bot when there is an explicit call. In some instances, there may be no text to input to the called bot. For example, if the utterance was "Pizza Bot," the EIS 230 may determine that the Pizza Bot is being called, but that there is no text to be processed by the Pizza Bot. In such a scenario, the EIS 230 may indicate to the skill bot caller 240 that there is nothing to send.

[0080] The skillbot invoker 240 invokes a skillbot in various manners. For example, the skillbot invoker 240 can invoke a bot in response to receiving an indication 235 that a particular skillbot has been selected as a result of an explicit invoke. The indication 235 can be sent by the EIS 230 along with input for the explicitly invoked skillbot. In this scenario, the skillbot invoker 240 hands over control of the conversation to the explicitly invoked skillbot. The explicitly invoked skillbot will determine an appropriate response to the input from the EIS 230 by treating the input as an independent utterance. For example, the response can be to perform a particular action or to start a new conversation in a particular state, where the initial state of the new conversation depends on the input sent from the EIS 230.

[0081] Another manner in which the skillbot invoker 240 can invoke a skillbot is through implicit invocation using the intent classifier 242. The intent classifier 242 can be trained using machine learning and / or rule-based training techniques to determine the likelihood that an utterance represents a task that a particular skillbot is configured to perform. The intent classifier 242 is trained with various classes, one class for each skillbot. For example, each time a new skillbot is registered with the masterbot, the intent classifier 242 can be trained using a list of example utterances associated with the new skillbot to determine the likelihood that a particular utterance represents a task that the new skillbot can perform. Parameters generated as a result of this training (e.g., a set of values ​​for parameters of a machine learning model) can be stored as part of the skillbot information 254.

[0082] In particular embodiments, the intent classifier 242 is implemented using a machine learning model, as described in further detail herein. Training the machine learning model may include inputting at least a subset of utterances from example utterances associated with various skill bots to generate, as an output of the machine learning model, an inference about which bot is the correct bot to process any particular training utterance. For each training utterance, an indication of the correct bot to use for that training utterance may be provided as ground truth information. The behavior of the machine learning model may then be adapted (e.g., through backpropagation) to minimize the discrepancy between the generated inference and the ground truth information.

[0083] In particular embodiments, the intent classifier 242 determines a confidence score for each skill bot registered with the master bot, indicating the likelihood that the skill bot can process an utterance (e.g., an implicit invocation utterance 234 received from the EIS 230). The intent classifier 242 may also determine a confidence score for each configured system-level intent (e.g., help, exit). If a particular confidence score satisfies one or more conditions, the skill bot invoker 240 will invoke the bot associated with the particular confidence score. For example, a threshold confidence score value may need to be met. Thus, the output 245 of the intent classifier 242 is either the identification of a system intent or the identification of a particular skill bot. In some embodiments, in addition to meeting the threshold confidence score value, the confidence score must exceed the next-highest confidence score by a particular winning margin. Imposing such a condition would enable routing to a particular skill bot if the confidence scores of multiple skill bots each exceed the threshold confidence score value.

[0084] After identifying the bot based on the evaluation of the confidence score, the skillbot invoker 240 hands over processing to the identified bot. In the case of a system intent, the identified bot is a master bot. Otherwise, the identified bot is a skill bot. Furthermore, the skillbot invoker 240 determines what to provide as input 247 for the identified bot. As described above, in the case of an explicit invoke, the input 247 may be based on a portion of the utterance not associated with the invoke, or the input 247 may be nothing (e.g., an empty string). In the case of an implicit invoke, the input 247 may be the entire utterance.

[0085] The data store 250 comprises one or more computing devices that store data used by various subsystems of the masterbot system 200. As described above, the data store 250 includes rules 252 and skillbot information 254. The rules 252 include, for example, rules for the MIS 220 to determine when an utterance expresses multiple intents and how to split an utterance expressing multiple intents. The rules 252 further include rules for the EIS 230 to determine which portions of an utterance that explicitly invokes a skillbot should be sent to the skillbot. The skillbot information 254 includes the call names of the skillbots in the chatbot system, for example, a list of the call names of all skillbots registered to a particular masterbot. The skillbot information 254 may also include information, for example, machine learning model parameters, that the intent classifier 242 uses to determine a confidence score for each skillbot in the chatbot system.

[0086] 3 is a simplified block diagram of a Skillbot system 300 according to certain embodiments. Skillbot system 300 is a computing system that may be implemented solely in software, solely in hardware, or a combination of hardware and software. In certain embodiments, such as the embodiment shown in FIG. 1, Skillbot system 300 may be used to implement one or more Skillbots within a digital assistant.

[0087] Skillbot system 300 includes MIS 310, intent classifier 320, and conversation manager 330. MIS 310 is similar to MIS 220 of FIG. 2 and provides similar functionality, including being operable to determine (1) whether an utterance expresses multiple intents, and if so, (2) how to split the utterance into separate utterances for each of the multiple intents using rules 352 in data store 350. In particular embodiments, the rules applied by MIS 310 to detect multiple intents and split the utterance are the same as the rules applied by MIS 220. MIS 310 receives utterance 302 and extracted information 304. Extracted information 304 is similar to extracted information 205 of FIG. 1 and can be generated using language parser 214 or a language parser local to skillbot system 300.

[0088] The intent classifier 320 may be trained in a manner similar to the intent classifier 242 described above in connection with the embodiment of FIG. 2 and as described in more detail herein. For example, in particular embodiments, the intent classifier 320 is implemented using a machine learning model. The machine learning model of the intent classifier 320 is trained for a particular skill bot using at least a subset of example utterances associated with that particular skill bot as training utterances. The ground truth for each training utterance will be the particular bot intent associated with that training utterance.

[0089] The utterance 302 may be received directly from a user or may be provided via a masterbot. If the utterance 302 is provided through the masterbot, for example, as a result of processing through the MIS 220 and the EIS 230 in the embodiment shown in FIG. 2, the MIS 310 may be bypassed to avoid repeating processing already performed by the MIS 220. However, if the utterance 302 is received directly from a user, for example, during a conversation that occurs after routing to a skillbot, the MIS 310 may process the utterance 302 to determine whether the utterance 302 represents multiple intents. If the utterance 302 represents multiple intents, the MIS 310 applies one or more rules to split the utterance 302 into separate utterances for each intent, for example, utterance “D” 306 and utterance “E” 308. If the utterance 302 does not represent multiple intents, the MIS 310 forwards the utterance 302, without segmentation, to the intent classifier 320 for intent classification.

[0090] The intent classifier 320 is configured to match a received utterance (e.g., utterance 306 or 308) with an intent associated with the skillbot system 300. As described above, a skillbot can be configured with one or more intents, each of which includes at least one example utterance associated with the intent and used to train the classifier. In the embodiment of FIG. 2, the intent classifier 242 of the masterbot system 200 is trained to determine a confidence score for individual skillbots and a confidence score for system intents. Similarly, the intent classifier 320 can be trained to determine a confidence score for each intent associated with the skillbot system 300. While the classification performed by the intent classifier 242 is at the bot level, the classification performed by the intent classifier 320 is at the intent level and therefore has finer granularity. The intent classifier 320 has access to intent information 354. For each intent associated with the skillbot system 300, the intent information 354 includes a list of utterances that express and illustrate the meaning of the intent and are typically associated with tasks that can be performed by the intent. The intent information 354 may also include parameters generated as a result of training on this list of utterances.

[0091] The conversation manager 330 receives as output from the intent classifier 320 an indication 322 of the particular intent identified by the intent classifier 320 as the closest match to the utterance input to the intent classifier 320. In some cases, the intent classifier 320 is unable to determine any match. For example, the confidence score calculated by the intent classifier 320 may fall below a threshold confidence score value if the utterance is directed to a system intent or to the intent of a different skill bot. When this occurs, the skill bot system 300 may refer the utterance to the master bot for processing, e.g., routing to a different skill bot. However, if the intent classifier 320 successfully identifies the intent within the skill bot, the conversation manager 330 will initiate a conversation with the user.

[0092] The conversation initiated by the conversation manager 330 is specific to the intent identified by the intent classifier 320. For example, the conversation manager 330 may be implemented using a state machine configured to execute a dialog flow for the identified intent. The state machine may include a default starting state (e.g., for when the intent is invoked without any additional input) and one or more additional states, where each state is associated with an action to be performed by the skill bot (e.g., complete a purchase transaction) and / or a dialog to be presented to the user (e.g., question, response). Thus, the conversation manager 330 can determine an action / dialog 335 upon receiving an instruction 322 identifying an intent and can determine additional actions or dialog in response to subsequent utterances received during the conversation.

[0093] Data store 350 comprises one or more computing devices that store data used by various subsystems of skillbot system 300. As shown in Figure 3, data store 350 includes rules 352 and intent information 354. In particular embodiments, data store 350 may be integrated with a masterbot or digital assistant data store, such as data store 250 of Figure 2.

[0094] Keyword Data Extension It has been found that models used to classify utterances as intents can be overconfident and can produce poor results for text containing certain keywords. To overcome this problem, various embodiments are directed to techniques for augmenting training data with utterances (negative training utterances) that have keywords in contexts different from those of the training data utterances (positive training utterances) to enable the machine learning model to generalize and learn that a keyword should only be problematic when used in a context similar to the training data. By augmenting the training data with utterances that have keywords in various contexts, machine learning models become better at focusing on the most important parts of examples in the correct context that connects them to their respective classes. Machine learning models trained on the augmented text data can be implemented in chatbot systems, as described with respect to FIGS. 1, 2, and 3. Advantageously, these machine learning models and chatbots perform better on utterances with keywords in various contexts because the models can better focus on the important parts of the utterance in the correct context. Furthermore, because the augmentation is applied automatically in a synthesis-independent manner, customers or clients do not need to worry about adding utterances with keywords in various contexts to the training data.

[0095] Figure 4 is a block diagram illustrating aspects of a chatbot system 400 configured to train and utilize a classifier (e.g., the intent classifier 242 or 320 described with respect to Figures 2 and 3) based on text data 405. As shown in Figure 4, the text classification performed by the chatbot system 400 in this example includes various stages: a predictive model training stage 410; a skillbot invocation stage 415 for determining the likelihood that an utterance represents a task that a particular skillbot is configured to perform; and an intent prediction stage 420 for classifying the utterance as one or more intents. The predictive model training stage 410 builds and trains one or more predictive models 425a-425n (where "n" represents any natural number) (which may be individually referred to herein as predictive models 425 and collectively referred to as predictive models 425) that are used by the other stages. For example, predictive models 425 may include a model for determining the likelihood that an utterance represents a task that a particular skillbot is configured to perform, another model for predicting intents from utterances for a first type of skillbot, and another model for predicting intents from utterances for a second type of skillbot. Still other types of predictive models may be implemented in other examples consistent with this disclosure.

[0096] The predictive model 425 may be a machine-learning (ML) model, such as a convolutional neural network ("CNN"), e.g., an Inception neural network, a residual neural network ("Resnet"), or a recurrent neural network, e.g., a long short-term memory ("LSTM") model or a gated recurrent unit ("GRU") model, or other variants of a deep neural network ("DNN") (e.g., a multi-level n-binary DNN classifier or a multi-class DNN classifier for single-intent classification). The predictive model 425 may also be any other suitable ML model trained for natural language processing, such as a naive Bayes classifier, a linear classifier, a support vector machine, a bagging model such as a random forest model, a boosting model, a shallow neural network, or a combination of one or more of such techniques, e.g., a CNN-HMM or an MCNN (multiscale convolutional neural network). The chatbot system 400 may employ the same or different types of predictive models to determine the likelihood of a task that a particular skillbot is configured to perform, to predict intents from utterances for a first type of skillbot, and to predict intents from utterances for a second type of skillbot. Still other types of predictive models may be implemented in other examples consistent with this disclosure.

[0097] To train the various predictive models 425, the training stage 410 consists of three main components: dataset preparation 430, feature engineering 435, and model training 440. Dataset preparation 430 involves loading data assets 445, splitting the data assets 445 into training and validation sets 445a-445n, and performing basic preprocessing so that the system can train and test the predictive models 425. The data assets 445 may include at least a subset of utterances from example utterances associated with various skill bots. As described above, the utterances may be provided in various ways, including as audio or text. The utterances may be fragments, complete sentences, multiple sentences, etc. For example, if the utterances are provided as audio, data preparation 430 may convert the audio to text using a speech-to-text converter (not shown), which inserts punctuation marks, such as commas, semicolons, periods, etc., into the resulting text. In some cases, the example utterances are provided by a client or customer. In other examples, example utterances are automatically generated from a library of prior utterances (e.g., identifying utterances from the library specific to the skill the chatbot is to learn). Data assets 445 for the predictive model 425 may include input text or speech (or input features of text or speech frames) and labels 450 corresponding to the input text or speech (or input features) as a matrix or table of values. For example, for each training utterance, an indication of the exact bot to use for that training utterance may be provided as ground truth information for the labels 450. The behavior of the predictive model 425 can then be adapted (e.g., by backpropagation) to minimize the difference between the generated inferences and the ground truth information. Alternatively, the predictive model 425 may be trained for a particular skill bot using at least a subset of the example utterances associated with that skill bot as training utterances.The ground truth information for the label 450 for each training utterance will be the particular bot intent associated with that training utterance.

[0098] In various embodiments, data preparation 430 includes keyword expanding 455 data assets 445 to include utterances with keywords in various contexts to make predictive model 425 more robust to keywords in unrelated contexts. These keywords are words that have a probability of being associated with a particular ground truth intent through training of the predictive model. This probability is higher than all of the words considered in data assets 445. By expanding data assets 445 with utterances with keywords in various contexts, predictive model 425 can better focus on the most important parts of the examples and the context that connects these examples to their respective classes. Expansion 455 is achieved using a keyword expansion technique to incorporate utterances containing keywords in various contexts along with the original utterances in data assets 445. The keyword expansion technique includes the following four operations: The four operations generally include (i) optionally normalizing all the original utterance or portions thereof, (ii) identifying keywords in all the (optionally normalized) original utterance or portions thereof, (iii) generating one or more OOD examples containing each identified keyword (i.e., generating an utterance with the keyword in a context different from that of the original utterance), and (iv) filtering out OOD examples in contexts that are too similar to the context of the original utterance.

[0099] The normalizing step may include the following steps. (i) filtering out stop words identified as keywords in the second step of identifying keywords ("stop words" refer to the most common words in a given language, such as when, that, a, has, the, is, etc.); (ii) lemmatizing all words in the training data in the second step (of identifying keywords) so that various word forms are unified under a single lemma with term frequency (TF) and term frequency-inverse document frequency (TF-IDF) scores (lemmatization in linguistics is the process of grouping various forms of a word so that they can be analyzed as a single item identified by the lemma or dictionary form of the word); (iii) searching for examples containing the keyword lemmas (when found, these examples may be added to the training assets in their original form but may not be lemmatized); and (iv) or any combination thereof.

[0100] Identifying keywords may include (i) using TF-IDF, (ii) using TF, (iii) using label names, (iv) using explainability tools, or (v) any combination thereof.

[0101] 1.TF-IDF TF-IDF can be used to identify keywords within a set. There are many ways to calculate the TF-IDF score.

[0102] 1. Treat each training utterance as a document. 2. Treat each training class as documentation.

[0103] 3. Compute TF and IDF scores on separate datasets, e.g., compute TF scores on training data and IDF scores on a background corpus (e.g., Gigaword corpus).

[0104] 4.Other In some cases, method (2) is used because it identifies which words are most important or unique to a class and is easier to compute.

[0105] Examples of TF-IDF keywords identified in the original training data set are shown in Table 2.

[0106] [Table 2]

[0107] 2.TF Basic word frequency can also be used to find keywords. The advantage of using TF in addition to TF-IDF is that it can help identify words that are important or frequent at the training set level but may not be unique to a small number of classes.

[0108] In some cases, a TF score is calculated for words in each training class. Examples of TF keywords identified in the original training data set are shown in Table 3.

[0109] [Table 3]

[0110] 3. Label Name Label names often contain important words that can be added to the list of keywords, however, it is important to check that these words are also found in the training data, otherwise there is a risk of false negatives when the label words are mentioned in the augmented utterances.

[0111] 4. Explainability Tools Explainability tools may be implemented to identify the keywords (or "anchors") that are most important to a particular classification result (in other words, have the machine learning model explain which words it felt were important to the classification decision). Explainability tools or explainable AI (XAI) refer to the implementation of frameworks and techniques that help human experts understand solutions developed by AI. Explainability tools or XAI may be implemented in several ways, including, but not limited to, linear models, decision tree algorithms, generalized additive models, local interpretable model-agnostic explanations (LIME), partial dependence plots, individual condition expectation plots, leave one column out (LOCO), cumulative local effects, anchors, SHAP (SHapley Additive exPlanations), deep learning important features, layer-by-layer relevance propagation, contrastive explanations, permutational feature importance, AI Explainability 360, Skater, "Explain Like I'm Five," and InterpretML. Because explainability tools know how the model actually behaves, tools such as those mentioned above may be better at identifying keywords than other methods. However, because explainability tools are very computationally intensive and there is little drawback to over-identifying keywords, traditional methods have their own advantages.

[0112] Generating OOD examples may include (i) finding OOD examples from a corpus, (ii) finding OOD examples from a vocabulary database such as WordNet, (iii) generating OOD examples using a model, e.g., Generative Pre-trained Transformer 2 (GPT-2), (iv) generating OOD examples using an adversarial attack model, or (v) any combination thereof.

[0113] 1. Finding OOD examples from the corpus OOD examples can be identified as candidates by searching a corpus (e.g., NLTK's Brown corpus and the Reuters corpus) for sentences containing the keywords identified in the previous step. Examples that are too short (fewer than five words) may be filtered out, as it may not be beneficial to add examples containing the keyword that do not contain much other content. The retained examples are then sorted by length. The more influential the keyword is in a particular example, the more preferable a shorter sentence is. In some cases, if there is sufficient corpus data, it may be beneficial to select examples that are the same length as the original example. Various expansion ratios, such as 1, may be used, where an OOD example is added for each example in the training data that contains the keyword.

[0114] Examples of OOD utterances found in the corpus are shown in Table 4.

[0115] [Table 4]

[0116] 2. Finding OOD examples from the vocabulary database A lexical database such as WordNet can be used to find OOD examples that cover all possible word senses for the identified keywords.

[0117] 3. Generate OOD examples using a model, e.g., GPT-2 OOD examples can be generated using models such as GPT-2, which may provide more control over the types of examples generated.

[0118] 4. Generating OOD examples using adversarial attack models A variation of the adversarial attack model can be used to generate OOD sentences known to be classified as being in the domain by a machine learning model. This is a one-off method as it does not require a keyword identification step.

[0119] The filtering step may include filtering out OOD examples that are too similar (substantially similar) to examples in the training data using distance measures (e.g., Euclidean, Manhattan, correlation, and Eisen) to avoid inter-class mismatch. In some cases, the distance filtering is based on the Euclidean distance between the vector of candidate OOD examples and the examples in the training data.

[0120] In some embodiments, batch balancing is used to ensure that the correct amount of OOD examples are present in each training batch (too few examples means that the OOD examples will have no impact, while too many OOD examples can drown out the original examples in the domain examples). The characteristics of the above process for keyword expansion may be treated as hyperparameters of the machine learning model 425. Table 5 shows example settings for the hyperparameters.

[0121] [Table 5]

[0122] With the introduction of keyword expansion techniques, the predictive model 425 becomes more aware of the context in which keywords are used, so that the predictive model 425 performs better on utterances that contain keywords in unrelated contexts. This improvement is achieved with little or no cost to in-domain performance thanks to the applied filtering step. Furthermore, because keyword expansion is applied automatically in a synthesis-independent manner, customers or clients do not need to worry about adding utterances that contain keywords and unrelated contexts to their training data.

[0123] In some cases, additional augmentation may be applied to data assets 445 (along with keyword expansion). For example, Easy Data Augmentation (EDA) techniques may be used to improve the performance of text classification tasks. EDA includes four operations: synonym substitution, random insertion, random exchange, and random deletion, which prevent overfitting and help train more robust models. In contrast to keyword expansion, EDA operations generally (i) take words from the original text and (ii) incorporate those words into each data asset 445 relative to the original text. For example, a synonym substitution operation involves randomly selecting n words from the original sentence (e.g., utterance) that are not stop words and replacing each of these words with one of its randomly selected synonyms. A random insertion operation involves finding n random synonyms of a random word that is not a stop word in the original sentence and inserting the synonyms at random positions in the sentence. The random swap operation involves randomly selecting two words in a sentence and swapping their positions n times. The random deletion operation involves randomly deleting each word in a sentence with probability p.

[0124] In various embodiments, feature engineering 435 includes converting data assets 445 (with keyword expansion) into feature vectors and / or creating new features using data assets 445 (with keyword expansion). Feature vectors may include count vectors as features, TF-IDF vectors as features, such as word-level, n-gram-level, or character-level features, word embeddings as features, text / NLP as features, topic models as features, or combinations thereof. Count vectors are a matrix representation of data assets 445, where each row represents an utterance, each column represents a word from the utterance, and each cell represents a frequency count of a particular word within the utterance. TF-IDF scores represent the relative importance of a word in an utterance. Word embeddings are a form of representing words and utterances using dense vector representations. Word locations in the vector space are learned from text and are based on the words surrounding the word when it is used. Text / NLP-based features may include the number of words in the utterance, the number of characters in the utterance, the average word density, the number of punctuation marks, the number of capital letters, the number of lemmas, the frequency distribution of part-of-speech tags (e.g., nouns and verbs), or any combination thereof.Topic modeling is the technique of identifying, from a collection of utterances, groups of words (called topics) that contain the best information in the collection.

[0125] In various embodiments, model training 440 includes training a classifier using the feature vectors and / or new features created in feature engineering 435. In some cases, the training process includes iterative operations to find a set of parameters for the predictive model 425 that maximizes or minimizes a cost function, such as a loss function or error function, for the predictive model 425. Each iteration may involve finding a set of parameters for the predictive model 425 such that the value of the cost function using the set of parameters for the predictive model 425 is greater or less than the value of the cost function using another set of parameters in the previous iteration. The cost function may be constructed to measure the difference between the output predicted using the predictive model 425 and the labels 450 included in the data asset 445. Once the set of parameters is identified, the predictive model 425 is trained and can be utilized for prediction as designed.

[0126] In addition to data assets 445, labels 450, feature vectors, and / or new features, other techniques and information can be employed to refine the training process of the predictive model 425. For example, feature vectors and / or new features may be combined with each other to help improve the accuracy of the classifier or model. Additionally or alternatively, hyperparameters may be adjusted or optimized, e.g., several parameters such as tree length, leaf, and network parameters may be fine-tuned to obtain the best-fit model. However, the training mechanisms described herein primarily focus on training the predictive model 425. These training mechanisms can also be utilized to fine-tune an existing predictive model 425 trained from other data assets. For example, in some cases, the predictive model 425 may have been pre-trained using utterances specific to another skill bot. In such cases, the predictive model 425 can be retrained using the data assets 445 (along with keyword expansion) as described herein.

[0127] The prediction model training stage 410 outputs trained prediction models 425, including a task prediction model 460 and an intent prediction model 465. The task prediction model 460 may be used in the skillbot invocation stage 415 to determine the likelihood that an utterance represents a task that a particular skillbot is configured to perform (470), and the intent prediction model 465 may be used in the intent prediction stage 420 to classify the utterance as one or more intents (475). In some examples, the skillbot invocation stage 415 and the intent prediction stage 420 may proceed independently, using separate models. For example, the trained intent prediction model 465 may be used in the intent prediction stage 420 to predict an intent for a skillbot without first identifying the skillbot in the skillbot invocation stage 415. Similarly, the task prediction model 460 may be used in the skillbot invocation stage 415 to predict the task or skillbot to be used for an utterance without identifying the intent of the utterance in the intent prediction stage 420.

[0128] Alternatively, the skillbot invocation stage 415 and the intent prediction stage 420 may occur sequentially, with one stage using the output of the other stage as input, or with one stage being invoked in a specific manner for a particular skillbot based on the output of the other stage. For example, for given text data 405, the skillbot invocation unit can invoke a skillbot through implicit invocation using the skillbot invocation stage 415 and the task prediction model 460. The task prediction model 460 can be trained using predictive and / or rule-based training techniques to determine the likelihood that an utterance represents a task that a particular skillbot 470 is configured to perform. For an identified or invoked skillbot and given text data 405, the intent prediction stage 420 and the intent prediction model 465 can then be used to match the received utterance (e.g., an utterance in a given data asset 445) to an intent 475 associated with the skillbot. As described herein, a skillbot can be composed of one or more intents, each of which includes at least one example utterance associated with the intent and used to train a classifier. In some embodiments, the skillbot invocation stage 415 and task prediction model 460 used in the masterbot system are trained to determine confidence scores for individual skillbots and for system intents. Similarly, the intent prediction stage 420 and intent prediction model 465 can be trained to determine a confidence score for each intent associated with the skillbot system. While the classification performed by the skillbot invocation stage 415 and task prediction model 460 is at the bot level, the classification performed by the intent prediction stage 420 and intent prediction model 465 is at the intent level and therefore has finer granularity.

[0129] Techniques for Keyword Data Augmentation and Utterance Classification FIG. 5 is a flowchart illustrating a process 500 for expanding a training dataset with OOD examples, according to some embodiments. The process illustrated in FIG. 5 may be implemented in software (e.g., code, instructions, programs) executed by one or more processing units (e.g., processors, cores) of a respective system, hardware, or a combination thereof. The software may be stored on a non-transitory storage medium (e.g., on a memory device). The method presented in FIG. 5 and described below is intended to be exemplary and non-limiting. While FIG. 5 depicts various processing steps occurring in a particular sequence or order, this is not intended to be limiting. In certain alternative embodiments, the steps may be performed in some different order, or some steps may be performed in parallel. In certain embodiments, such as those illustrated in FIGS. 1-4, the process illustrated in FIG. 5 may be performed by a preprocessing subsystem (e.g., preprocessing subsystem 210 or predictive model training stage 410) to generate a keyword-expanded dataset for training by one or more predictive models (e.g., intent classifier 242 or 320 or predictive model 425).

[0130] At 505, a training set of utterances is received by a data processing system (e.g., the chatbot system 400 described with respect to FIG. 4). In some examples, the training set of utterances is used to train an intent classifier to identify one or more intents for one or more utterances.

[0131] At 510, the data processing system augments the training set of utterances with the OOD examples to obtain an augmented training set of utterances. In various embodiments, the augmenting step includes (i) identifying keywords within the utterances of the training set of utterances; (ii) generating a set of OOD examples that include the identified keywords; (iii) filtering out OOD examples from the set of OOD examples that have contexts substantially similar to those of the utterances of the training set of utterances; and (iv) incorporating the set of OOD examples that do not include the filtered OOD examples into the training set of utterances to generate an augmented training set of utterances. As used herein, when an action is "based on" something, this means that the action is at least partially based on at least a portion of something. As used herein, the terms "substantially," "approximately," and "about" are defined as being largely, but not necessarily completely, specified (and including being completely specified), as understood by those skilled in the art. In any of the disclosed embodiments, the words "substantially," "approximately," or "about" may be substituted with "within [a percentage]" of what is specified, where the percentage includes 0.1 percent, 1 percent, 5 percent, and 10 percent. The substantial similarity between the context of an OOD example and the context of an utterance in the training set of utterances may be determined based on a distance measure to avoid inter-class mismatches.

[0132] Optionally, the expanding step further comprises normalizing the training set of utterances and / or the set of OOD examples, where the normalizing step comprises (i) filtering out stop words identified as the identified keywords, (ii) lemmatizing all words in the training set of utterances, (iii) lemmatizing all words in the set of OOD examples with the identified keywords, or (iv) any combination thereof.

[0133] At 515, a machine learning model is trained using the augmented text data training set to determine the likelihood that an utterance or message represents a task that a skillbot is configured to perform or to match an utterance or message to an intent associated with a skillbot. The trained machine learning model may then be deployed within a chatbot system (e.g., as part of a masterbot or skillbot) at 520 to determine the likelihood that an utterance or message represents a task that a skillbot is configured to perform or to match an utterance or message to an intent associated with a skillbot. For example, an utterance may be received and analyzed to determine whether the utterance includes a call name for the skillbot. If a call name is not found, the utterance is considered to be an implicit call, and the process proceeds to an intent classifier, such as a trained model. If a call name is determined to be present, the utterance is considered to be an explicit call, and the process proceeds to determining which portion of the utterance is associated with the call name.

[0134] When a trained model is invoked, the entire received utterance is provided as input to an intent classifier. The intent classifier receiving the utterance may be the masterbot's intent classifier (e.g., intent classifier 242 of FIG. 2). The intent classifier may be a machine learning-based or rule-based classifier trained with keyword-expanded data to determine whether the intent of the utterance matches a system intent (e.g., exit, help) or a specific skillbot. As described herein, the intent analysis performed by the masterbot may be limited to matching a specific skillbot without determining which intent within the skillbot is the best match for the utterance. Thus, the intent classifier receiving the utterance may identify a specific skillbot to be invoked. Alternatively, if the utterance expresses a specific system intent (e.g., the utterance includes the words "exit" or "help"), the intent classifier receiving the utterance may identify that specific system intent to trigger a conversation between the masterbot and the user based on the dialog flow configured for that specific system intent.

[0135] If an invocation name is present, one or more explicit invocation rules are applied to determine which portions of the utterance are associated with the invocation name. This determination may be based on an analysis of the sentence structure of the utterance using POS tags, dependency information, and / or other extracted information received with the utterance. For example, a portion associated with the invocation name may be a noun phrase containing the invocation name or a prepositional object corresponding to the invocation name. Any portion associated with the invocation name, as determined based on the processing, is removed. Other portions of the utterance that are not required to convey the meaning of the utterance (e.g., prepositional words) may also be removed. The removal of certain portions of the utterance generates input for the skillbot associated with the invocation name. If any portions of the received utterance remain after the removal, the remaining portions form a new utterance for input to the skillbot, for example, as a text string. Alternatively, if the received utterance is completely removed, the input may be an empty string.

[0136] A skill bot associated with the invocation name is then invoked and provided with the generated input. Upon receiving the generated input, the invoked skill bot will process the input by, for example, performing intent analysis using the skill bot's intent classifier trained on keyword-expanded data to identify a bot intent that matches the user intent expressed in the input. As a result of identifying a matching bot intent, the skill bot may perform a specific action or begin a conversation with the user according to a dialog flow associated with the matching bot intent. For example, if the input is an empty string, the conversation may begin in a default state defined for the dialog flow, such as a welcome message. Alternatively, if the input is not an empty string, the conversation may begin in some intermediate state, for example, because the input includes a value for an entity or some other information that the skill bot received as part of the input and therefore no longer needs to prompt the user. As another example, the skill bot may determine that it cannot process the input (e.g., because the confidence scores of all bot intents configured for the skill bot are below a certain threshold). In this situation, the skillbot may pass the input back to the masterbot for processing (e.g., intent analysis using the masterbot's intent classifier), or the skillbot may prompt the user for clarification.

[0137] In various embodiments, deploying and using an intent classifier in a chatbot system includes: the chatbot system receiving utterances generated by a user interacting with the chatbot system; using the intent classifier deployed in the chatbot system to classify the utterances into intent categories corresponding to intents; and using the intent classifier to output an intent based on the classification. The intent classifier includes a plurality of model parameters identified using training data, the training data including a training set of augmented utterances for training the intent classifier to identify one or more intents for one or more utterances, the training set of augmented utterances being artificially generated from the training set of utterances to include augmented utterances, where keywords are identified from the training set of utterances and incorporated into OOD utterances having contexts substantially different from the contexts of the utterances in the training set of utterances to generate the augmented utterances. The plurality of model parameters are identified using the training data based on maximizing or minimizing a cost function. Deploying and using the intent classifier in a chatbot system further includes using the intent classifier to output an intent based on the classification.

[0138] The substantial difference between the context of the OOD example and the context of the utterances in the training set of utterances may be determined based on a distance measure to avoid inter-class mismatch.

[0139] Exemplary System 6 shows a simplified diagram of a distributed system 600. In the illustrated example, the distributed system 600 includes one or more client computing devices 602, 604, 606, and 608 coupled to a server 612 via one or more communication networks 610. The client computing devices 602, 604, 606, and 608 may be configured to run one or more applications.

[0140] In various examples, server 612 may be adapted to run one or more services or software applications that enable one or more embodiments described in this disclosure. In certain examples, server 612 may also provide other services or software applications, which may include non-virtualized and virtualized environments. In some examples, these services may be provided as web-based or cloud services, such as under a Software as a Service (SaaS) model, to users of client computing devices 602, 604, 606, and / or 608. Users operating client computing devices 602, 604, 606, and / or 608 may utilize the services provided by these components by interacting with server 612 utilizing one or more client applications.

[0141] 6, server 612 may include one or more components 618, 620, and 622 that implement the functionality performed by server 612. These components may include software components that may be executed by one or more processors, hardware components, or a combination thereof. It should be appreciated that a wide variety of system configurations are possible that may differ from distributed system 600. Thus, the example shown in FIG. 6 is one example of a distributed system for implementing an exemplary system and is not intended to be limiting.

[0142] A user may use client computing devices 602, 604, 606, and / or 608 to run one or more applications, models, or chatbots that may generate one or more events or models that may be implemented or processed according to the teachings of this disclosure. A client device may provide an interface that allows a user of the client device to interact with the client device. The client device may also output information to the user through this interface. Although FIG. 6 shows only four client computing devices, any number of client computing devices may be supported.

[0143] Client devices may include various types of computing systems, such as portable handheld devices, general-purpose computers such as personal computers and laptops, workstation computers, wearable devices, gaming systems, thin clients, various messaging devices, sensors or other sensing devices, etc. These computing devices may include various types and versions of software applications and operating systems (e.g., Microsoft Windows®, Apple Macintosh®, UNIX® or UNIX-like operating systems, Linux® or Linux-like operating systems, e.g., Google Chrome® OS), including various mobile operating systems (e.g., Microsoft Windows Mobile®, iOS®, Windows Phone®, Android®, BlackBerry®, Palm OS®). Portable handheld devices may include cellular phones, smartphones (e.g., iPhone®), tablets (e.g., iPad®), personal digital assistants (PDAs), etc. Wearable devices may include Google Glass® head-mounted displays and other devices. The gaming systems may include various handheld gaming devices, internet-enabled gaming devices (e.g., Microsoft Xbox® gaming consoles with or without Kinect® gesture input devices, Sony PlayStation® systems, various gaming systems offered by Nintendo®, etc.) The client devices may be capable of running a wide variety of applications, such as various internet-related applications, communication applications (e.g., email applications, short message service (SMS) applications), and may use a variety of communication protocols.

[0144] Network 610 may be any type of network known to those skilled in the art that can support data communications using any of a variety of available protocols, including, but not limited to, TCP / IP (Transmission Control Protocol / Internet Protocol), SNA (Systems Network Architecture), IPX (Internet Packet Exchange), AppleTalk®, etc. By way of example only, network 610 may be a local area network (LAN), an Ethernet-based network, a token ring, a wide area network (WAN), the Internet, a virtual network, a virtual private network (VPN), an intranet, an extranet, a public switched telephone network (PSTN), an infrared network, a wireless network (e.g., a network operating under any of the Institute of Electrical and Electronics Engineers (IEEE) 802.11 protocol suite, Bluetooth®, and / or any other wireless protocol), and / or any combination of these and / or other networks.

[0145] Servers 612 may be comprised of one or more general-purpose computers, dedicated server computers (including, by way of example, PC (personal computer) servers, UNIX servers, midrange servers, mainframe computers, rack-mounted servers, etc.), server farms, server clusters, or other suitable configurations and / or combinations. Servers 612 may include one or more virtual machines running a virtual operating system or other computing architectures involving virtualization, such as one or more flexible pools of logical storage that may be virtualized to maintain virtual storage for the servers. In various examples, servers 612 may be adapted to run one or more services or software applications that provide the functionality described in the above disclosure.

[0146] The computing systems in server 612 may run one or more operating systems, including any of the operating systems described above, as well as any commercially available server operating system. Server 612 may also run any of a variety of additional other server and / or middle-tier applications, including an HTTP (Hypertext Transfer Protocol) server, an FTP (File Transfer Protocol) server, a CGI (Common Gateway Interface) server, a JAVA server, a database server, etc. Exemplary database servers include, but are not limited to, those commercially available from Oracle®, Microsoft®, Sybase®, IBM® (International Business Machines), etc.

[0147] In some implementations, server 612 may include one or more applications for parsing and consolidating data feeds and / or event updates received from users of client computing devices 602, 604, 606, and 608. By way of example, the data feeds and / or event updates may include, but are not limited to, Twitter® feeds, Facebook® updates, or real-time updates received from one or more third-party sources and continuous data streams that may include real-time events related to sensor data applications, financial stock tickers, network performance measurement tools (e.g., network monitoring and traffic management applications), clickstream analysis tools, automobile traffic monitoring, etc. Server 612 may also include one or more applications for displaying the data feeds and / or real-time events via one or more display devices of client computing devices 602, 604, 606, and 608.

[0148] The distributed system 600 may also include one or more data repositories 614, 616. In certain examples, these data repositories may be used to store data and other information. For example, one or more of the data repositories 614, 616 may be used to store information, such as information related to chatbot performance or generated models for use by a chatbot used by the server 612 in performing various functions according to various embodiments. The data repositories 614, 616 may reside in a variety of locations. For example, the data repository used by the server 612 may be local to the server 612 or remote from the server 612 and may communicate with the server 612 via a network-based or dedicated connection. The data repositories 614, 616 may be of different types. In certain examples, the data repository used by the server 612 may be a database, for example, a relational database such as those provided by Oracle Corporation® and other manufacturers. One or more of these databases may be adapted to allow data to be stored, updated, and retrieved from the database in response to SQL-formatted commands.

[0149] In particular examples, one or more of the data repositories 614, 616 may be used by an application to store application data. The data repositories used by the application may be of various types, such as, for example, a key-value store repository, an object store repository, or a general-purpose storage repository supported by a file system.

[0150] In particular examples, the functionality described in this disclosure may be provided as services via a cloud environment. FIG. 7 is a simplified block diagram of a cloud-based system environment in which various services may be provided as cloud services, according to particular examples. In the example shown in FIG. 7, cloud infrastructure system 702 may provide one or more cloud services that users may request using one or more client computing devices 704, 706, and 708. Cloud infrastructure system 702 may include one or more computers and / or servers, which may include those described above with respect to server 612. The computers in cloud infrastructure system 702 may be organized as general-purpose computers, dedicated server computers, server farms, server clusters, or any other suitable arrangement and / or combination.

[0151] Network 710 may facilitate communication and exchange of data between clients 704, 706, and 708 and cloud infrastructure system 702. Network 710 may include one or more networks. The networks may be of the same or different types. Network 710 may support one or more communication protocols, including wired and / or wireless protocols, to facilitate communication.

[0152] The example shown in Figure 7 is merely one example of a cloud infrastructure system and is not intended to be limiting. It should be understood that in other examples, cloud infrastructure system 702 may have more or fewer components than those shown in Figure 7, may combine two or more components, or may have components in a different configuration or arrangement. For example, while Figure 7 shows three client computing devices, in alternative examples, any number of client computing devices may be supported.

[0153] The term cloud service generally refers to services made available to users on demand via a communications network, such as the Internet, by a service provider's system (e.g., cloud infrastructure system 702). Typically, in a public cloud environment, the servers and systems that comprise the cloud service provider's system are distinct from a customer's own on-premise servers and systems. The cloud service provider's systems are managed by the cloud service provider. Thus, customers can use cloud services offered by the cloud service provider without purchasing separate licenses, support, or hardware and software resources for the services. For example, the cloud service provider's system may host applications, and users may order and use the applications on demand via the Internet without purchasing infrastructure resources to run the applications. Cloud services are designed to provide easy and scalable access to applications, resources, and services. Several providers offer cloud services. For example, several cloud services, such as middleware services, database services, and Java cloud services, are offered by Oracle Corporation of Redwood Shores, California.

[0154] In particular examples, cloud infrastructure system 702 may provide one or more cloud services using various models, such as a Software as a Service (SaaS) model, a Platform as a Service (PaaS) model, an Infrastructure as a Service (IaaS) model, etc., including a hybrid service model. Cloud infrastructure system 702 may include a suite of applications, middleware, databases, and other resources that enable the provisioning of various cloud services.

[0155] The SaaS model allows applications or software to be delivered as a service to customers over a communications network such as the Internet, without the customer having to purchase the underlying application hardware or software. For example, the SaaS model may be used to provide customers with access to on-demand applications hosted by cloud infrastructure system 702. Examples of SaaS services offered by Oracle Corporation® include, but are not limited to, various services for human resource / capital management, customer relationship management (CRM), enterprise resource planning (ERP), supply chain management (SCM), enterprise performance management (EPM), analytics services, and social applications.

[0156] The IaaS model is commonly used to provide flexible computing and storage capabilities by providing infrastructure resources (e.g., servers, storage, hardware, and networking resources) to customers as cloud services. Various IaaS services are offered by Oracle Corporation (registered trademark).

[0157] The PaaS model is generally used to provide platform and environment resources as a service that enable customers to develop, run, and manage applications and services without having to procure, build, or maintain the environment resources. Examples of PaaS services provided by Oracle Corporation (registered trademark) include, but are not limited to, Oracle Java Cloud Service (JCS), Oracle Database Cloud Service (DBCS), data management cloud services, and various application development solution services.

[0158] Cloud services are generally provided on an on-demand, self-service basis, on a subscription basis, and in a flexible, scalable, reliable, highly available, and secure manner. For example, a customer may order one or more services offered by cloud infrastructure system 702 via a subscription order. Cloud infrastructure system 702 then performs processing to provide the services requested in the customer's subscription order. For example, a user may use utterances to request the cloud infrastructure system to take a particular action (e.g., intent), as described above, and / or provide a service for a chatbot system, as described herein. Cloud infrastructure system 702 may be configured to provide one cloud service or even multiple cloud services.

[0159] Cloud infrastructure system 702 may provide cloud services through a variety of deployment models. In a public cloud model, cloud infrastructure system 702 may be owned by a third-party cloud service provider, and cloud services are offered to general public customers. These customers may be individuals or businesses. In another example, under a private cloud model, cloud infrastructure system 702 may function within an organization (e.g., within a corporate organization), and services are offered to customers within the organization. For example, these customers may be various departments within a company, such as human resources, payroll, or individuals within the company. In another example, under a community cloud model, cloud infrastructure system 702 and the services it offers may be shared among various organizations within an associated community. Various other models, including hybrids of the above models, may also be used.

[0160] Client computing devices 704, 706, and 708 may be of different types (e.g., client computing devices 602, 604, 606, and 608 shown in FIG. 6 ) and may be capable of operating one or more client applications. Users may use the client devices to interact with cloud infrastructure system 702, such as to request services provided by cloud infrastructure system 702. For example, users may use client devices to request information or actions from a chatbot, as described in this disclosure.

[0161] In some examples, the processing performed by cloud infrastructure system 702 to provide services may include model training and deployment. This analysis may include using, analyzing, and processing data sets to train and deploy one or more models. This analysis may be performed by one or more processors, possibly processing the data in parallel, running simulations using the data, etc. For example, big data analysis may be performed by cloud infrastructure system 702 to generate and train one or more models for a chatbot system. The data used in this analysis may include structured data (e.g., data stored in a database or structured according to a structured model) and / or unstructured data (e.g., data blobs (binary large objects)).

[0162] 7, cloud infrastructure system 702 may include infrastructure resources 730 utilized to facilitate the provision of various cloud services offered by cloud infrastructure system 702. Infrastructure resources 730 may include, for example, processing resources, storage or memory resources, networking resources, etc. In particular examples, storage virtual machines available to handle storage requested by applications may be part of cloud infrastructure system 702. In other examples, the storage virtual machines may be part of a different system.

[0163] In certain examples, to facilitate efficient provisioning of these resources to support the various cloud services offered by cloud infrastructure system 702 to different customers, resources may be organized into resource sets or resource modules (also referred to as "pods"). Each resource module or pod may include a pre-integrated, optimized combination of one or more types of resources. In certain examples, different pods may be pre-provisioned for different types of cloud services. For example, a first set of pods may be provisioned for database services, while a second set of pods may be provisioned for Java services, etc., which may include a different combination of resources than the pods in the first set of pods. For some services, the resources allocated for provisioning those services may be shared between services.

[0164] Cloud infrastructure system 702 itself may use services 732 internally that are shared by various components of cloud infrastructure system 702 and that facilitate the provisioning of services by cloud infrastructure system 702. These internal shared services may include, but are not limited to, security and identity services, integration services, enterprise repository services, enterprise manager services, virus scanning and whitelist services, high availability, backup and recovery services, services enabling cloud support, email services, notification services, file transfer services, etc.

[0165] Cloud infrastructure system 702 may include multiple subsystems. These subsystems may be implemented in software, hardware, or a combination thereof. As shown in FIG. 7 , the subsystems may include a user interface subsystem 712 that allows users or customers of cloud infrastructure system 702 to interact with cloud infrastructure system 702. User interface subsystem 712 may include a variety of different interfaces, such as a web interface 714, an online store interface 716 through which cloud services offered by cloud infrastructure system 702 are advertised and available for consumer purchase, and other interfaces 718. For example, a customer may use a client device to request one or more services (service request 734) offered by cloud infrastructure system 702 using one or more of interfaces 714, 716, and 718. For example, a customer may access an online store, browse cloud services offered by cloud infrastructure system 702, and place a subscription order for one or more services offered by cloud infrastructure system 702 for which the customer wishes to subscribe. The service request may include information identifying the customer and the one or more services for which the customer wishes to subscribe. For example, a customer may submit an order to subscribe to services provided by cloud infrastructure system 702. As part of the order, the customer may provide information identifying the chatbot system for which the services will be provided and, optionally, one or more credentials for the chatbot system.

[0166] 7, cloud infrastructure system 702 may include an order management subsystem (OMS) 720 configured to process new orders. As part of this processing, OMS 720 may be configured to create an account for the customer if not already created, receive billing and / or account information from the customer to use for billing the customer for providing the requested services to the customer, verify the customer information, and, once verified, reserve the order for the customer and prepare the order for provisioning by coordinating various workflows.

[0167] Upon proper validation, OMS 720 may invoke an order provisioning subsystem (OPS) 724 configured to provision resources for the order, including processing, memory, and networking resources. Provisioning may include allocating resources for the order and configuring the resources to facilitate the service requested by the customer order. The manner in which resources are provisioned for the order and the type of resources provisioned may depend on the type of cloud service ordered by the customer. For example, following a workflow, OPS 724 may be configured to determine the specific cloud service being requested and identify the number of pods that will be pre-configured for this specific cloud service. The number of pods allocated for an order may depend on the size / amount / level / scope of the service requested. For example, the number of pods to allocate may be determined based on the number of users the service is to support, the duration for which the service is requested, etc. The allocated pods may then be customized to the specific requesting customer to provide the requested service.

[0168] In particular examples, the setup phase may be performed as part of the provisioning process, as described above, by cloud infrastructure system 702. Cloud infrastructure system 702 may generate an application ID and select a storage virtual machine for the application from among storage virtual machines provided by cloud infrastructure system 702 itself or from storage virtual machines provided by other systems other than cloud infrastructure system 702.

[0169] Cloud infrastructure system 702 may send a response or notification 744 to the requesting customer to indicate when the requested service will be available for use. In some examples, the customer may be sent information (e.g., a link) that enables the customer to begin using and utilizing the benefits of the requested service. In particular examples, for a customer requesting a service, the response may include a chatbot system ID generated by cloud infrastructure system 702 and information identifying the chatbot system selected by cloud infrastructure system 702 for the chatbot system corresponding to the chatbot system ID.

[0170] Cloud infrastructure system 702 may provide services to multiple customers. For each customer, cloud infrastructure system 702 manages information related to one or more subscription orders received from the customer, maintains customer data related to the orders, and is responsible for providing the requested services to the customer. Cloud infrastructure system 702 may also collect usage statistics regarding the customer's use of the subscribed services. For example, statistics may be collected about the amount of storage used, the amount of data transferred, the number of users, and the amount of system uptime and downtime. This usage information may be used to bill the customer. Billing may be on a monthly basis, for example.

[0171] Cloud infrastructure system 702 may provide services to multiple customers concurrently. Cloud infrastructure system 702 may store information about these customers, possibly including copyright information. In particular examples, cloud infrastructure system 702 includes an identity management subsystem (IMS) 728 configured to manage customer information and isolate the managed information so that information about one customer is not accessible to another customer. IMS 728 may be configured to provide various security-related services, such as identity services such as information access management, authentication and authorization services, services for managing customer identities and roles and associated capabilities, etc.

[0172] FIG. 8 illustrates an example computer system 800. In some examples, computer system 800 may be used to implement any of the digital assistant or chatbot systems in a distributed environment, as well as the various servers and computer systems described above. As shown in FIG. 8, computer system 800 includes various subsystems, including a processing subsystem 804 that communicates with several other subsystems via a bus subsystem 802. These other subsystems may include a processing acceleration unit 806, an I / O subsystem 808, a storage subsystem 818, and a communication subsystem 824. Storage subsystem 818 may include non-transitory computer-readable storage media, including a storage medium 822 and a system memory 810.

[0173] Bus subsystem 802 provides a mechanism for allowing the various components and subsystems of computer system 800 to communicate with each other as intended. While bus subsystem 802 is shown schematically as a single bus, alternative embodiments of the bus subsystem may utilize multiple buses. Bus subsystem 802 may be any of several types of bus structures including a memory bus or memory controller, a peripheral bus, a local bus, etc., using any of a variety of bus architectures. For example, such architectures may include an Industry Standard Architecture (ISA) bus, a Micro Channel Architecture (MCA) bus, an Enhanced ISA (EISA) bus, a Video Electronics Standards Association (VESA) local bus, and a Peripheral Component Interconnect (PCI) bus, which may be implemented as a mezzanine bus manufactured in accordance with the IEEE P1386.1 standard.

[0174] The processing subsystem 804 controls the operation of the computer system 800 and may include one or more processors, application specific integrated circuits (ASICs), or field programmable gate arrays (FPGAs). The processors may include single-core or multi-core processors. The processing resources of the computer system 800 may be organized into one or more processing units 832, 834, etc. The processing units may include one or more processors, one or more cores from the same or different processors, a combination of cores and processors, or other combinations of cores and processors. In some examples, the processing subsystem 804 may include one or more dedicated coprocessors, such as a graphics processor, a digital signal processor (DSP), etc. In some examples, some or all of the processing units of the processing subsystem 804 may be implemented using customized circuitry, such as an application specific integrated circuit (ASIC) or a field programmable gate array (FPGA).

[0175] In some examples, processing units within processing subsystem 804 may execute instructions stored in system memory 810 or computer-readable storage medium 822. In various examples, the processing units may execute various program or code instructions and maintain multiple programs or processes running simultaneously. At any given time, some or all of the program code to be executed may reside in system memory 810 and / or computer-readable storage medium 822, potentially including one or more storage devices. Through appropriate programming, processing subsystem 804 may provide the various functions described above. In examples in which computer system 800 is running one or more virtual machines, one or more processing units may be assigned to each virtual machine.

[0176] In certain examples, a processing acceleration unit 806 may optionally be provided to accelerate the overall processing performed by the computer system 800, to perform customized processing, or to offload portions of the processing performed by the processing subsystem 804.

[0177] I / O subsystem 808 may include devices and mechanisms for inputting information into computer system 800 and / or outputting information from or through computer system 800. In general, use of the term "input device" is intended to include all conceivable types of devices and mechanisms for inputting information into computer system 800. User interface input devices may include, for example, keyboards, pointing devices such as mice or trackballs, touchpads or touchscreens integrated into displays, scroll wheels, click wheels, dials, buttons, switches, keypads, voice input devices with voice command recognition systems, microphones, and other types of input devices. User interface input devices may also include motion-sensing and / or gesture-recognition devices, such as a Microsoft Kinect® motion sensor that allows a user to control and interact with the input device, a Microsoft Xbox® 360 game controller, or devices that provide an interface for receiving input using gestures and voice commands. The user interface input devices may also include eye gesture recognition devices, such as a Google Glass® blink detector, that detects eye movements from the user (e.g., "blinks" while taking a picture and / or making a menu selection) and translates the eye gestures as input to the input device (e.g., Google Glass®). Additionally, the user interface input devices may include a voice recognition sensing device that allows the user to interact with a voice recognition system (e.g., Siri® navigator) via voice commands.

[0178] Other examples of user interface input devices include, but are not limited to, three-dimensional (3D) mice, joysticks or pointing sticks, gamepads, and graphic tablets, as well as audio / visual devices such as speakers, digital cameras, digital camcorders, portable media players, webcams, image scanners, fingerprint scanners, barcode readers, 3D scanners, 3D printers, laser range finders, and eye-tracking devices. In addition, user interface input devices may include medical imaging input devices such as, for example, computed tomography, magnetic resonance imaging, position emission tomography, and medical ultrasound devices. User interface input devices may also include audio input devices such as, for example, MIDI keyboards, digital musical instruments, and the like.

[0179] In general, the use of the term output device(s) is intended to include all conceivable types of devices and mechanisms for outputting information from computer system 800 to a user or to another computer. User interface output devices may include display subsystems, indicator lights, or non-visual displays such as audio output devices. Display subsystems may be flat-panel devices such as displays using cathode ray tubes (CRTs), liquid crystal displays (LCDs), or plasma displays, projection devices, touch screens, etc. For example, user interface output devices may include, but are not limited to, various display devices that visually convey text, graphics, and audio / visual information, such as monitors, printers, speakers, headphones, automobile navigation systems, plotters, audio output devices, and modems.

[0180] Storage subsystem 818 provides a repository or data store for storing information and data used by computer system 800. Storage subsystem 818 provides a tangible, non-transitory computer-readable storage medium for storing the basic programming and data constructs that provide some example functionality. Software (e.g., programs, code modules, instructions) that, when executed by processing subsystem 804, provide the functionality described above may be stored in storage subsystem 818. The software may be executed by one or more processing units of processing subsystem 804. Storage subsystem 818 may also provide authentication in accordance with the teachings of the present disclosure.

[0181] The storage subsystem 818 may include one or more non-transitory memory devices, including volatile and non-volatile memory devices. As shown in FIG. 8, the storage subsystem 818 includes a system memory 810 and a computer-readable storage medium 822. The system memory 810 may include several memories, including volatile main random access memory (RAM) for storing instructions and data during program execution and non-volatile read-only memory (ROM) or flash memory in which fixed instructions are stored. In some implementations, a basic input / output system (BIOS), containing the basic routines that help transfer information between elements within the computer system 800, such as during start-up, may typically be stored in ROM. Typically, RAM contains data and / or program modules currently operated on and executed by the processing subsystem 804. In some implementations, the system memory 810 may include multiple different types of memory, such as static random access memory (SRAM), dynamic random access memory (DRAM), etc.

[0182] 8, system memory 810 may load running application programs 812, program data 814, and operating system 816, which may include various applications such as a web browser, a middle-tier application, a relational database management system (RDBMS), etc. By way of example, operating system 816 may include Microsoft Windows®, Apple Macintosh®, and / or Linux operating systems, various commercially available UNIX® or UNIX-like operating systems (including, but not limited to, various GNU / Linux operating systems, Google Chrome® OS, etc.), and / or various versions of mobile operating systems such as iOS®, Windows Phone, Android® OS, BlackBerry® OS, Palm® OS operating systems, etc.

[0183] The computer-readable storage medium 822 may store programming and data structures that provide some example functionality. The computer-readable storage medium 822 may provide storage of computer-readable instructions, data structures, program modules, and other data for the computer system 800. Software (programs, code modules, instructions) that, when executed by the processing subsystem 804, provide the above-described functionality may be stored in the storage subsystem 818. By way of example, the computer-readable storage medium 822 may include non-volatile memory such as a hard disk drive, a magnetic disk drive, a CD-ROM, a DVD, an optical disk drive such as a Blu-Ray® disk, or other optical media. The computer-readable storage medium 822 may include, but is not limited to, a Zip® drive, a flash memory card, a Universal Serial Bus (USB) flash drive, a Secure Digital (SD) card, a DVD disc, a digital video tape, etc. The computer-readable storage medium 822 may also include solid-state drives (SSDs) based on non-volatile memory such as flash memory-based SSDs, enterprise flash drives, solid-state ROM, etc., SSDs based on volatile memory such as solid-state RAM, dynamic RAM, static RAM, DRAM-based SSDs, magnetoresistive RAM (MRAM) SSDs, and hybrid SSDs that use a combination of DRAM and flash memory-based SSDs.

[0184] In particular examples, storage subsystem 818 may also include a computer-readable storage medium reader 820, which may be further connected to a computer-readable storage medium 822. Reader 820 may be configured to receive and read data from a memory device such as a disk, a flash drive, or the like.

[0185] In certain examples, computer system 800 may support virtualization techniques, including, but not limited to, virtualization of processing and memory resources. For example, computer system 800 may provide support for running one or more virtual machines. In certain examples, computer system 800 may execute a program such as a hypervisor that facilitates configuration and management of virtual machines. Each virtual machine may be assigned memory, computing (e.g., processors, cores), I / O, and networking resources. Each virtual machine typically executes independently from other virtual machines. A virtual machine typically executes its own operating system, which may be the same as or different from the operating systems executed by other virtual machines executed by computer system 800. Thus, potentially multiple operating systems may be executed simultaneously by computer system 800.

[0186] The communications subsystem 824 provides an interface to other computer systems and networks. The communications subsystem 824 serves as an interface for sending and receiving data between other systems and the computer system 800. For example, the communications subsystem 824 may enable the computer system 800 to establish a communications channel to one or more client devices over the Internet to send and receive information from the one or more client devices. For example, if the computer system 800 is used to implement the bot system 120 shown in FIG. 1, the communications subsystem may be used to communicate with a chatbot system selected for the application.

[0187] The communications subsystem 824 may support both wired and / or wireless communications protocols. In particular examples, the communications subsystem 824 may include radio frequency (RF) transceiver components for accessing wireless voice and / or data networks (e.g., using cellular telephone technology, advanced data network technologies such as 3G, 4G, or EDGE (High Data Rates for Global Evolution), WiFi (IEEE 802.XX family of standards, or other mobile communications technologies, or any combination thereof), global positioning system (GPS) receiver components, and / or other components. In some examples, the communications subsystem 824 may provide a wired network connection (e.g., Ethernet) in addition to or instead of a wireless interface.

[0188] The communications subsystem 824 may receive and transmit data in various formats. In some examples, the communications subsystem 824 may receive incoming communications in the form of structured and / or unstructured data feeds 826, event streams 828, event updates 830, etc., among other formats. For example, the communications subsystem 824 may be configured to receive (or transmit) data feeds 826 in real time from users of social media networks and / or other communications services, such as web feeds such as Twitter® feeds, Facebook® updates, Rich Site Summary (RSS) feeds, and / or real-time updates from one or more third-party sources.

[0189] In particular examples, the communications subsystem 824 may be configured to receive data in the form of a continuous data stream, which may include an event stream 828 of real-time events and / or event updates 830 that may be continuous or infinite in nature without a clear end. Examples of applications that generate continuous data may include, for example, sensor data applications, financial stock tickers, network performance measurement tools (e.g., network monitoring and traffic management applications), clickstream analysis tools, automobile traffic monitoring, etc.

[0190] Communications subsystem 824 may be configured to communicate data from computer system 800 to other computer systems or networks. This data may be communicated in a variety of different formats, such as structured and / or unstructured data feeds 826, event streams 828, event updates 830, etc., to one or more databases that may be in communication with one or more streaming data source computers coupled to computer system 800.

[0191] Computer system 800 can be one of a variety of types, including a handheld portable device (e.g., an iPhone® cellular phone, an iPad® computing tablet, a PDA), a wearable device (e.g., a Google Glass® head-mounted display), a personal computer, a workstation, a mainframe, a kiosk, a server rack, or other data processing system. Due to the ever-changing nature of computers and networks, the description of computer system 800 shown in FIG. 8 is intended only as a specific example. Many other configurations are possible, having more or fewer components than the system shown in FIG. 8. It should be recognized, based on the disclosure and teachings herein, that there are other aspects and / or methods for implementing the various examples.

[0192] While specific examples have been described, various variations, modifications, alternative configurations, and equivalents are possible. The examples are not limited to operation in a particular data processing environment, but may freely operate in multiple data processing environments. Furthermore, while some examples have been described using a particular sequence of transactions and steps, it should be apparent to those skilled in the art that this is not intended to be limiting. While some flowcharts describe operations as sequential processes, many of these operations may be performed in parallel or simultaneously. Additionally, the order of operations may be re-specified. A process may have additional steps not included in the figures. Various features and aspects of the above examples may be used individually or together.

[0193] Additionally, while particular examples have been described using particular combinations of hardware and software, it should be understood that other combinations of hardware and software are possible. Particular examples may be implemented exclusively in hardware, exclusively in software, or using a combination thereof. The various processes described herein may be implemented on the same processor or any combination of different processors.

[0194] Where a device, system, component, or module is described as being configured to perform particular operations or functions, such configuration may be achieved, for example, by designing electronic circuitry to perform the operations, by programming a programmable electronic circuit (such as a microprocessor) to perform the operations, by executing computer instructions or code, or a processor or core, programmed to execute code or instructions stored, for example, on a non-transitory memory medium, or any combination thereof. Processes may communicate using a variety of techniques, including, but not limited to, conventional techniques for inter-process communication, and different pairs of processes may use different techniques, or the same pair of processes may use different techniques at different times.

[0195] In this disclosure, specific details are provided to ensure a thorough understanding of the examples. However, the examples may be practiced without these specific details. For example, well-known circuits, processes, algorithms, structures, and techniques are shown without unnecessary detail so as not to obscure the examples. This specification provides illustrative examples only and is not intended to limit the scope, applicability, or configuration of other examples. Rather, the above description of the examples will provide one skilled in the art with an enabling description for implementing various examples. Various changes may be made within the function and configuration of elements.

[0196] Accordingly, the specification and drawings are to be regarded in an illustrative rather than a restrictive sense. It will be apparent, however, that additions, subtractions, deletions, and other modifications and alterations may be made thereto without departing from the broader spirit and scope as set forth in the claims. Thus, while specific examples have been described, they are not intended to be limiting. Various modifications and equivalents are within the scope of the appended claims.

[0197] While the foregoing specification describes aspects of the disclosure with reference to specific examples thereof, those skilled in the art will recognize that the disclosure is not limited thereto. Various features and aspects of the above disclosure may be used individually or together. Moreover, the examples can be utilized in a variety of environments and applications beyond those described herein without departing from the broader spirit and scope of the specification. Accordingly, the specification and drawings should be regarded as illustrative rather than restrictive.

[0198] In the above description, the methods are described in a particular order for purposes of illustration. It should be understood that in alternative examples, the methods may be performed in an order different from that described. It should also be understood that the methods described above may be performed by hardware components or embodied in a sequence of machine-executable instructions that, when used, cause a machine, such as a general-purpose or special-purpose processor or logic circuitry programmed with such instructions, to perform the method. These machine-executable instructions may be stored on one or more machine-readable media, such as a CD-ROM or other type of optical disk, floppy disk, ROM, RAM, EPROM, EEPROM, magnetic or optical card, flash memory, or other type of machine-readable medium suitable for storing electronic instructions. Alternatively, the methods may be performed by a combination of hardware and software.

[0199] Where a component is described as being configured to perform particular operations, such configuration may be achieved, for example, by designing electronic circuitry or other hardware to perform the particular operations, by programming a programmable electronic circuitry (e.g., a microprocessor or other suitable electronic circuitry) to perform the particular operations, or any combination thereof.

[0200] Although specific examples of the present application have been described in detail herein, it is to be understood that the concepts of the present invention may be variously embodied and employed in other forms, and that the claims are intended to be construed to include such variations except insofar as limited by the prior art.

Claims

1. 1. A method comprising: receiving, in a data processing system, a training set of utterances for training a machine learning model to identify one or more intents for one or more utterances; augmenting, by the data processing system, the training set of utterances with out-of-domain (OOD) examples, wherein the augmenting step comprises: identifying keywords within utterances from the training set of utterances; generating a set of OOD examples with the identified keywords; filtering out from the set of OOD examples those OOD examples that have a context substantially similar to a context of the utterance in the training set of utterances; and incorporating the set of OOD examples that does not include the filtered OOD examples into the training set of utterances to generate an extended training set of utterances, the method further comprising: training, by the data processing system, the machine learning model using the augmented training set of utterances; The method, wherein substantial similarity between the context of the OOD example and the context of the utterance in the training set of utterances is determined based on a distance measure to avoid inter-class mismatch.

2. 2. The method of claim 1, further comprising: normalizing the training set of utterances and / or the set of OOD examples, wherein the normalizing comprises: (i) filtering out stop words identified as the identified keywords; (ii) lemmatizing all words in the training set of utterances; (iii) lemmatizing all words in the set of OOD examples with the identified keywords; or (iv) any combination thereof.

3. 3. The method of claim 1, wherein the keywords are identified using term frequency-inverse document frequency (TF-IDF), word frequency, label names, explainability tools, or any combination thereof.

4. The method of claim 1 , wherein the set of OOD examples is generated using a corpus, a vocabulary database, a text generation model, an adversarial attack model, or any combination thereof.

5. The method of claim 1 , further comprising deploying the trained machine learning model in a chatbot system.

6. The method of claim 1 , wherein the keywords are words that, through training of the machine learning model, have the potential to be associated with a particular ground truth intent.

7. 1. A system comprising: one or more processors; a memory coupled to the one or more processors, the memory storing a plurality of instructions executable by the one or more processors, the plurality of instructions including instructions that, when executed by the one or more processors, cause the one or more processors to perform the following operations: receiving a training set of utterances for training a machine learning model to identify one or more intents for one or more utterances; and augmenting the training set of utterances with out-of-domain (OOD) examples, the augmenting comprising: identifying keywords within an utterance from the training set of utterances; generating a set of OOD examples with the identified keywords; filtering out OOD examples from the set of OOD examples that have a context substantially similar to a context of the utterance in the training set of utterances; and combining the set of OOD examples that does not include the filtered OOD examples into the training set of utterances to generate an extended training set of utterances, wherein the following operations further comprise: training the machine learning model using the augmented training set of utterances; The system, wherein substantial similarity between the context of the OOD example and the context of the utterance in the training set of utterances is determined based on a distance measure to avoid inter-class mismatch.

8. 8. The system of claim 7, wherein the following operations further include normalizing the training set of utterances and / or the set of OOD examples, the normalizing operations including (i) filtering out stop words identified as the identified keywords, (ii) lemmatizing all words in the training set of utterances, (iii) lemmatizing all words in the set of OOD examples with the identified keywords, or (iv) any combination thereof.

9. The system of claim 7 or 8, wherein the keywords are identified using word frequency-inverse document frequency (TF-IDF), word frequency, label names, explainability tools, or any combination thereof.

10. The system of claim 7 , wherein the set of OOD examples is generated using a corpus, a vocabulary database, a text generation model, an adversarial attack model, or any combination thereof.

11. The system of claim 7 , wherein the following operation further comprises an operation of deploying the trained machine learning model in a chatbot system.

12. The system of claim 7 , wherein the keywords are words that have a likelihood of being associated with a particular ground truth intent through training of the machine learning model.

13. 7. A program comprising a plurality of instructions executable by one or more processors, the plurality of instructions comprising instructions that, when executed by the one or more processors, cause the one or more processors to perform the method of any one of claims 1 to 6.

Images