Stopword data augmentation for natural language processing

Abstract

Stopword data augmentation techniques for training chatbot systems in natural language processing. In one particular aspect, a computer-implemented method includes receiving a training set of utterances for training an intent classifier to identify an intent of an utterance; augmenting the training set of utterances with stop words to generate an augmented out-of-domain training set of utterances for an unresolved intent category corresponding to an unresolved intent; and training the intent classifier using the training set of utterances and the augmented out-of-domain training set of utterances. The augmenting includes selecting utterances from the training set of utterances, and for each selected utterance, retaining existing stop words within the utterance and replacing at least one non-stop word within the utterance with a stop word or stop word phrase selected from a list of stop words to generate an out-of-domain utterance.

Description

[0001] CLAIM OF PRIORITY

[0002] This application is a non-provisional of U.S. Provisional Application No. 62 / 901,203, filed September 16, 2019, entitled “STOP WORD AUGMENTATION TOOL,” and claims the benefit of and priority to the U.S. Provisional Application under 35 U.S.C. 119(e). The entire contents of the foregoing application are incorporated herein by reference in their entirety for all purposes. TECHNICAL FIELD

[0003] The present disclosure relates generally to chatbot systems, and more particularly to stop word data augmentation techniques for training chatbot systems in natural language processing. BACKGROUND

[0004] To get instant reactions, many users around the world use instant messaging or chat platforms. Organizations often use these instant messaging or chat platforms to have real-time conversations with customers (or end users). However, it can be very expensive for an organization to employ service personnel to have real-time conversations with customers or end users. Chatbots or bots have been developed to simulate conversations with end users, especially over the Internet. End users can converse with the bots through messaging applications that the end users have already installed and use. Intelligent bots (often powered by artificial intelligence (AI)) can converse more intelligently and contextually in real-time conversations and thus can allow for a more natural conversation between the bot and the end user to improve the conversation experience. Rather than the end user going through a fixed set of keywords or commands that the bot knows how to respond to, intelligent bots can be able to understand the intent of the end user based on natural language user utterances and respond accordingly.

[0005] However, chatbots are difficult to build because these automated solutions require specific knowledge in certain domains and application of certain techniques that can be outside the capabilities of a professional developer. As part of building such chatbots, developers can first understand the needs of the enterprise and end users. Developers can then analyze and make decisions related to, for example, selecting a dataset to be used for analysis; preparing the input dataset for analysis (e.g., cleaning data, extracting, formatting, and / or transforming data, performing data feature engineering, etc. prior to analysis); identifying appropriate machine learning (ML) technique(s) or ML model(s) to perform the analysis; and improving the technique or model to improve results / effectiveness based on feedback. The task of identifying appropriate models can include developing multiple models (possibly in parallel), iteratively testing and experimenting with the models, and then identifying a particular model (or models) for use. Further, supervised learning-based solutions typically involve a training phase, followed by an application (i.e., inference) phase, and an iterative cycle in between the training and application phases. Developers can be responsible for carefully implementing and monitoring these phases to achieve the best solution. For example, to train ML technique(s) or model(s), precise training data is needed to enable the algorithm to understand and learn certain patterns or features (e.g., for chatbots - intent extraction and careful syntactic analysis are needed, not just raw language processing), which the ML technique(s) or model(s) will use to predict the desired outcome (e.g., infer intent from utterance). To ensure that the ML technique(s) or model(s) learn these patterns and features correctly, developers can be responsible for selecting, enriching, and optimizing the training dataset for the ML technique(s) or model(s). SUMMARY

[0006] The technology disclosed herein generally relates to chatbots. More specifically and non- limitingly, the technology disclosed herein relates to techniques for stopword data augmentation of training datasets for training chatbot systems in natural language processing. Chatbots can classify user utterances into different categories, such as predefined intents of the user. The classifier of a chatbot can include a trained ML model that generates an output (e.g., an intent) based on an input (e.g., a user utterance). When the training data used to train the trained ML model is insufficient, the chatbot can more frequently determine incorrect intents. The technology disclosed herein can provide a stopword-augmented dataset for training the ML model, enabling the ML model to stop picking up stopword patterns and more accurately learn patterns or boundaries of intents. Various embodiments are described herein, including methods, systems, non-transitory computer-readable storage media storing programs, code, or instructions executable by one or more processors, and the like.

[0007] In various embodiments, a computer-implemented method is provided, the method comprising: receiving, at a data processing system, a set of utterances for training an intent classifier to identify one or more intents of one or more utterances; augmenting, by the data processing system, the set of utterances with stop words to generate an augmented out-of-domain set of utterances for an unparseable intent category corresponding to unparseable intents, wherein the augmenting comprises: selecting one or more utterances from the set of utterances, and for each selected utterance, preserving existing stop words within the utterance and replacing at least one non-stop word within the utterance with a stop word or stop word phrase selected from a list of stop words to generate an out-of-domain utterance; and training, by the data processing system, the intent classifier using the set of utterances and the augmented out-of-domain set of utterances.

[0008] In some embodiments, for each selected utterance, the existing stop words within the utterance are preserved and all non-stop words within the utterance are replaced with a stop word or stop word phrase selected from the list of stop words.

[0009] In some embodiments, for each selected utterance, the existing stop words within the utterance are preserved and at least one of the non-stop words within the utterance is randomly replaced with a stop word or stop word phrase selected from the list of stop words.

[0010] In some embodiments, selecting the utterances comprises searching for consecutive stop words at the beginning of the utterances within the set of utterances and selecting the utterance having the longest consecutive stop words at the beginning of the utterances within the set of utterances.

[0011] In some embodiments, selecting the utterances comprises searching for n consecutive stop words at any position in the utterances within the set of utterances and selecting the utterance having n consecutive stop words at any position in the utterances.

[0012] In some embodiments, augmenting the set of utterances with the stop words further comprises: (i) iteratively selecting the one or more utterances from the set of utterances and processing each utterance a single time to generate a corresponding single out-of-domain utterance based on a predefined augmentation ratio, (ii) selecting the one or more utterances from the set of utterances and processing the one or more utterances multiple times to generate multiple out-of-domain utterances from each utterance based on the predefined augmentation ratio, or (iii) any combination thereof.

[0013] In some embodiments, the predefined augmentation ratio is between 1:0.05 and 1:1 (original utterance: augmented utterance).

[0014] In various embodiments, a computer-implemented method is provided that includes receiving, by a chatbot system, an utterance generated by a user interacting with the chatbot system; classifying, using an intent classifier deployed within the chatbot system, the utterance as a resolved intent category corresponding to a resolved intent or an unresolved intent category corresponding to an unresolved intent, wherein the intent classifier includes a plurality of model parameters identified using training data including a set of utterance training data used to train the intent classifier to identify one or more resolved intents of one or more utterances and an augmented out-of-domain utterance training data used to train the intent classifier to identify one or more unresolved intents of one or more utterances, wherein the augmented out-of-domain utterance training data is artificially generated to include utterances from the set of utterance training data in which existing stopword patterns within the utterances are preserved and at least one non-stopword within each utterance is randomly replaced with a stopword, and wherein the plurality of model parameters are identified based on minimizing a loss function using the training data; and outputting, using the intent classifier, the resolved intent or the unresolved intent based on the classifying.

[0015] In various embodiments, a system is provided that includes one or more data processors and a non-transitory computer-readable storage medium containing instructions that, when executed on the one or more data processors, cause the one or more data processors to perform part or all of one or more methods disclosed herein.

[0016] In various embodiments, a computer program product is provided that is tangibly embodied in a non-transitory machine-readable storage medium and includes instructions configured to cause one or more data processors to perform part or all of one or more methods disclosed herein.

[0017] Some embodiments of the disclosure include a system that includes one or more data processors. In some embodiments, the system includes a non-transitory computer-readable storage medium containing instructions that, when executed on the one or more data processors, cause the one or more data processors to perform part or all of one or more methods disclosed herein and / or part or all of one or more processes. Some embodiments of the disclosure include a computer program product tangibly embodied in a non-transitory machine- readable storage medium and including instructions configured to cause one or more data processors to perform part or all of one or more methods disclosed herein and / or part or all of one or more processes.

[0018] The techniques described above and below can be implemented in a number of ways and in a number of contexts. Several example implementations and contexts are provided below in greater detail, with reference to the following figures. However, the following implementations and contexts are merely some of the many. BRIEF DESCRIPTION OF DRAWINGS

[0019] Figure 1 is a simplified block diagram of a distributed environment incorporating example embodiments.

[0020] Figure 2 is a simplified block diagram of a computing system implementing a host robot in accordance with certain embodiments.

[0021] Figure 3 is a simplified block diagram of a computing system implementing a skill robot in accordance with certain embodiments.

[0022] Figure 4 is a simplified block diagram of a chatbot training and deployment system in accordance with various embodiments.

[0023] Figure 5 illustrates a process flow for augmenting a training dataset with stop words in accordance with various embodiments.

[0024] Figure 6 depicts a simplified diagram of a distributed system for implementing various embodiments.

[0025] Figure 7 is a simplified block diagram of one or more components of a system environment in accordance with various embodiments, through which services provided by one or more components of an embodiment system can be offered as cloud services.

[0026] Figure 8 illustrates an example computer system that can be used for implementing various embodiments. DETAILED DESCRIPTION

[0027] In the following description, for the purposes of explanation, specific details are set forth in order to provide a thorough understanding of certain embodiments. However, it will be apparent that various embodiments can be practiced without these specific details. The drawings and description are not intended to be restrictive. The word "exemplary" is used herein to mean "serving as an example, instance, or illustration." Any implementation or design described herein as "exemplary" is not necessarily to be construed as preferred or advantageous over other implementations or designs.

[0028] INTRODUCTION

[0029] A digital assistant is an artificial intelligence-driven interface that helps users accomplish various tasks in natural language conversations. For each digital assistant, a customer can assemble one or more skills. A skill (also described herein as a chatbot, a bot, or a skillbot) is a separate robot that focuses on a specific type of task such as tracking inventory, submitting time cards, and creating expense reports. When an end user engages with a digital assistant, the digital assistant evaluates end user input and routes the conversation to and from the appropriate chatbot. The digital assistant can be made available to end users through various channels such as Messenger (instant messaging), SKYPE Messenger, or short message service (SMS), among others. Channels enable chats to be communicated back and forth between end users and digital assistants and their respective chatbots through various messaging platforms. Channels can also support user agent escalation, event-initiated conversations, and testing.

[0030] An intent allows a chatbot to understand what a user wants the chatbot to do. An intent includes a typical user request and arrangement of statements, which can also be referred to as a utterance (e.g., get account balance, make a purchase, etc.). As used herein, a utterance or message can refer to a set of words (e.g., one or more sentences) exchanged during a conversation with a chatbot. An intent can be created by providing a name that describes some user action (e.g., order pizza) and compiling a set of real-life user statements or utterances that are typically associated with triggering the action. Because the cognition of a chatbot is derived from these intents, each intent can be created from a robust data set (one to two dozen utterances) and varied so that the chatbot can interpret ambiguous user input. A rich set of utterances enables a chatbot to understand what a user wants when it receives a message (a message that means the same thing but is expressed differently) such as “Forget this order!” or “Cancel delivery!” An intent and the utterances that belong to the intent collectively make up the training corpus of a chatbot. By training a model with the corpus, a customer can essentially turn the model into a reference tool for parsing end user input into a single intent. A customer can improve the acuity of the cognition of a chatbot through multiple rounds of intent testing and intent training.

[0031] However, building a chatbot that can determine the intent of an end user based on a user utterance is somewhat of a challenging task, in part due to the subtlety and ambiguity of natural language and the dimensionality of the input space (e.g., possible user utterances) and the size of the output space (number of intents). As such, to improve the performance of a chatbot and the user experience with a chatbot, it can be necessary to train, monitor, debug, and retrain the chatbot. In traditional systems, a training system is provided for training and retraining machine learning models of a digital assistant or chatbot in spoken language understanding (SLU) and natural language processing (NLP). Traditionally, models for chatbot systems are trained in NLP using utterances that are “manufactured” for any intent. For example, the utterance “Do you do price changes?” can be used to train a classifier model of a chatbot system to classify this type of utterance as the intent “Do you offer a price match?” Using manufactured utterances to train the models helps to initially train the chatbot system to provide a service, and then once the chatbot system is deployed and starts to get real utterances from users, the chatbot system can be retrained.

[0032] Traditionally, training of text classification models starts with pre-labeled data. One of the options to obtain labeled data is to manually annotate all data by manually reading through each text and categorizing the text (e.g., categorizing an utterance as an intent). Another option is to outsource this labeling to freelancers or companies dedicated to data labeling. Either option is expensive in terms of time and money. To overcome this problem in similar technical spaces such as image analysis, data augmentation has been used to increase the size of the image dataset for image classification tasks and to speed up the labeling process. While for images, natural augmentation primitives such as rotation, small translational shifts, and additive Gaussian noise have been developed well, for text data, similar primitives have not been developed as well. For example, in text-based data augmentation, the input data should be changed while keeping the classification label unchanged. If the augmentation method significantly changes the meaning of an utterance (e.g., a sentence or a stage), the original classification label can no longer be valid. Thus, traditional augmentation primitives developed for text focus on the words present in each sentence (e.g., synonym replacement, synonym random insertion, random swapping, random deletion, etc.) and the length of the sentence or phrase to keep the original classification label of the sentence or phrase. However, these methods can be difficult to implement for enterprise systems that use many chatbot systems trained for many different tasks in multiple languages and receive a wide variety of utterances for each task (many of which are very short utterances). In systems that use chatbots such as this, there is a need to apply augmentation automatically in a synthetic-agnostic manner in order to quickly and efficiently generate large training data corpora in multiple languages for many different chatbots.

[0033] Thus, different approaches are needed to address these problems. In various embodiments, a method is provided that includes receiving a set of utterances for training an intent classifier to identify one or more intents of one or more utterances and augmenting the set of utterances with stopword text to obtain an augmented training text dataset. The augmenting includes generating a set of utterances for an un-resolved intent category corresponding to an un-resolved intent, the set of utterances including utterances from the set of utterances for training, where existing stopword patterns are preserved and at least one non-stopword in the set of utterances is randomly replaced with a stopword. The computer-implemented method further includes training the intent classifier using the augmented training text dataset.

[0034] Robots and analysis systems

[0035] Robots (also referred to as skills, chatbots, conversational bots, or talkbots) are computer programs that can perform a conversation with an end user. A robot can typically respond to natural language messages (e.g., questions or comments) through a messaging application using natural language messages. An enterprise can use one or more robot systems to communicate with end users through a messaging application. The messaging application (which can be referred to as a channel) can be an end user preferred messaging application that the end user already has installed and is familiar with. Thus, to chat with a robot system, an end user does not need to download and install a new application. The messaging application can include, for example, over-the-top (OTT) messaging channels (such as Facebook Messenger, Facebook WhatsApp, WeChat, Line, Kik, Telegram, Talk, Skype, Slack, or SMS), virtual personal assistants (such as Amazon Dot, Echo, or Show, Google Home, Apple HomePod, etc.), native or hybrid extended mobile and web application extensions / responsive mobile or web applications with chat functionality, or voice-based inputs (such as devices or applications with Siri, Microsoft Cortana, Google Voice, or other voice inputs for interaction).

[0036] In some examples, a robot system can be associated with a uniform resource identifier (URI). The URI can identify the robot system using a string of characters. The URI can be used as a webhook for one or more messaging application systems. The URI can include, for example, a uniform resource locator (URL) or a uniform resource name (URN). The robot system can be designed to receive messages (e.g., hypertext transfer protocol (HTTP) post call messages) from a messaging application system. The HTTP post call messages can involve the URI from the messaging application system. In some embodiments, the messages can be different from the HTTP post call messages. For example, the robot system can receive messages from a short message service (SMS). While the discussion herein can refer to the communications received by the robot system as messages, it should be understood that the messages can be HTTP post call messages, SMS messages, or any other type of communication between the two systems.

[0037] End users can interact with a robotic system through conversational interactions (sometimes referred to as conversational user interfaces (UIs)), just as they would with another human. In some cases, the interaction can include an end user saying "Hello" to the robot and the robot responding with "Hi" and asking the end user how the robot can help. In some cases, the interaction can also be a transactional interaction with, for example, a banking robot, such as transferring money from one account to another; an informational interaction with, for example, an HR robot, such as checking vacation balances; or an interaction with, for example, a retail robot, such as discussing returning a purchased item or seeking technical support.

[0038] In some embodiments, a robotic system can intelligently handle end user interactions without interacting with an administrator or developer of the robotic system. For example, an end user can send one or more messages to a robotic system in order to achieve a desired goal. The messages can include some content, such as text, emoticons, audio, images, video, or other methods of conveying a message. In some embodiments, the robotic system can convert the content into a standardized form (e.g., a Representational State Transfer (REST) call with appropriate parameters for an enterprise service) and generate a natural language response. The robotic system can also prompt the end user for additional input parameters or request other additional information. In some embodiments, the robotic system can also initiate a conversation with the end user, rather than passively responding to end user utterances. Described herein are various techniques for recognizing explicit invocations of a robotic system and determining the input of the invoked robotic system. In certain embodiments, explicit invocation analysis is performed by a host robot based on detecting an invocation name in an utterance. In response to detecting the invocation name, the utterance can be refined for input to a skill robot associated with the invocation name.

[0039] A conversation with the robot can follow a particular conversation flow that includes a plurality of states. The flow can define what happens next based on input. In some embodiments, a robot system can be implemented using a state machine that includes user-defined states (e.g., end user intents) and actions to take in or between states. The conversation can take different paths based on end user input, which can influence the decisions made by the robot for the flow. For example, at each state, based on end user input or utterance, the robot can determine the intent of the end user in order to determine the next appropriate action to take. As used herein and in the context of an utterance, the term "intent" refers to the intent of the user providing the utterance. For example, a user can intend to engage the robot in a conversation for ordering a pizza, such that the intent of the user can be expressed by the utterance "Order pizza." The user intent can relate to a particular task that the user wants the chatbot to perform on behalf of the user. Thus, the utterance can be expressed as a question, command, request, etc. that reflects the intent of the user. The intent can include a goal that the end user wants to accomplish.

[0040] In the context of the configuration of the chatbot, the term "intent" as used herein refers to configuration information for mapping utterances of a user to a particular task / action or a particular kind of task / action that the chatbot can perform. To distinguish between the intent of an utterance (i.e., the user intent) and the intent of the chatbot, the latter is sometimes referred to herein as "robot intent." The robot intent can include a set of one or more utterances associated with the intent. For example, an intent to order a pizza can have various permutations of utterances that express the desire to place an order to purchase a pizza. These associated utterances can be used to train an intent classifier of the chatbot to enable the intent classifier to subsequently determine whether an input utterance from a user matches the intent to order a pizza. The robot intent can be associated with one or more dialog flows for initiating a conversation with the user in a certain state. For example, a first message for the intent to order a pizza can be the question "What kind of pizza would you like?" In addition to the associated utterances, the robot intent can further include named entities related to the intent. For example, the intent to order a pizza can include variables or parameters for performing the task of ordering a pizza, such as topping 1, topping 2, pizza type, pizza size, number of pizzas, etc. The values of the entities are typically obtained through a conversation with the user.

[0041] Figure 1is a simplified block diagram of an environment 100 incorporating a chatbot system according to certain embodiments. The environment 100 includes a digital assistant builder platform (DABP) 102 that enables users of the DABP 102 to create and deploy digital assistant or chatbot systems. The DABP 102 can be used to create one or more digital assistants (or DAs) or chatbot systems. For example, as shown in Figure 1 a user 104 on behalf of a particular enterprise can use the DABP 102 to create and deploy a digital assistant 106 for the user of the particular enterprise. For example, a bank can use the DABP 102 to create one or more digital assistants for use by customers of the bank. Multiple enterprises can use the same DABP 102 platform to create digital assistants. As another example, an owner of a restaurant (e.g., a pizza shop) can use the DABP 102 to create and deploy a digital assistant that enables customers of the restaurant to order food (e.g., order a pizza).

[0042] For purposes of this disclosure, a "digital assistant" is an entity that helps users of the digital assistant accomplish various tasks through natural language conversations. A digital assistant can be implemented using only software (e.g., the digital assistant is a digital entity implemented using programs, code, or instructions executable by one or more processors), using hardware, or using a combination of hardware and software. A digital assistant can be embodied or implemented in various physical systems or devices such as computers, mobile phones, watches, appliances, vehicles, etc. Digital assistants are sometimes also referred to as chatbot systems. Thus, for purposes of this disclosure, the terms digital assistant and chatbot system are interchangeable.

[0043] Digital assistants (such as the digital assistant 106 built using the DABP 102) can be used to perform various tasks via natural language based conversations between the digital assistant and its users 108. As part of a conversation, a user can provide one or more user inputs 110 to the digital assistant 106 and obtain returned responses 112 from the digital assistant 106. A conversation can include one or more of the inputs 110 and responses 112. Via these conversations, users can request that one or more tasks be performed by the digital assistant, and in response, the digital assistant is configured to perform the tasks requested by the user and respond to the user with appropriate responses.

[0044] User input 110 is typically in the form of natural language and is referred to as an utterance. User utterance 110 can be in textual form, such as when a user types a sentence, question, piece of text, or even a single word and provides the text as input to digital assistant 106. In some embodiments, user utterance 110 can be in the form of audio input or speech, such as when a user speaks or says something as input to digital assistant 106. The utterance is typically in the form of a language spoken by user 108. For example, the utterance can be in English or some other language. When the utterance is in the form of speech, the speech input is converted into a textual form of the utterance in that particular language, and then the textual utterance is processed by digital assistant 106. Various speech-to-text processing techniques can be used to convert speech or audio input into a textual utterance, which is then processed by digital assistant 106. In some embodiments, the conversion of speech to text can be done by digital assistant 106 itself.

[0045] The utterance, which can be a textual utterance or a speech utterance, can be a piece, a sentence, multiple sentences, one or more words, one or more questions, a combination of the above types, etc. Digital assistant 106 is configured to apply natural language understanding (NLU) techniques to the utterance to understand the meaning of the user input. As part of the NLU processing for the utterance, digital assistant 106 is configured to perform processing for understanding the meaning of the utterance, which involves identifying one or more intents and one or more entities corresponding to the utterance. Upon understanding the meaning of the utterance, digital assistant 106 can perform one or more actions or operations in response to the understood meaning or intent. For the purposes of this disclosure, it is assumed that these utterances are textual utterances that have been directly provided by user 108 of digital assistant 106, or are the result of converting input speech utterances into textual form. However, this is not intended to be limiting or constraining in any way.

[0046] For example, input by user 108 can request ordering a pizza by providing a utterance such as "I want to order a pizza." Upon receiving such a utterance, digital assistant 106 is configured to understand the meaning of the utterance and take appropriate action. The appropriate action can involve, for example, responding to the user with a question requesting user input regarding the type of pizza the user desires to order, the size of the pizza, any toppings for the pizza, etc. The response provided by digital assistant 106 can also be in natural language form and typically in the same language as the input utterance. As part of generating these responses, digital assistant 106 can perform natural language generation (NLG). In order for the user to be able to order a pizza via a conversation between the user and digital assistant 106, the digital assistant can guide the user to provide all necessary information for the pizza order and then cause the pizza to be ordered at the end of the conversation. Digital assistant 106 can end the conversation by outputting information to the user indicating that the pizza has been ordered.

[0047] At a conceptual level, digital assistant 106 performs various processing in response to utterances received from a user. In some embodiments, the processing involves a series of processing steps or a processing pipeline, including, for example, understanding the meaning of an input utterance (sometimes referred to as natural language understanding (NLU)), determining an action to perform in response to the utterance, causing the action to be performed if appropriate, generating a response to output to the user in response to the user utterance, outputting the response to the user, etc. The NLU processing can include parsing an input utterance to understand the structure and meaning of the utterance, refining and reformulating the utterance to develop a better understandable form (e.g., logical form) or structure of the utterance. Generating a response can include using NLG techniques.

[0048] The NLU processing performed by a digital assistant such as digital assistant 106 can include various NLP-related processing such as sentence syntax analysis (e.g., tokenization, part-of-speech tagging, identifying named entities in a sentence, generating dependency trees to represent sentence structure, dividing a sentence into clauses, analyzing individual clauses, resolving references, performing chunking, etc.). In certain embodiments, the NLU processing or portions thereof are performed by digital assistant 106 itself. In some other embodiments, digital assistant 106 can use other resources to perform portions of the NLU processing. For example, the syntax and structure of a sentence of an input utterance can be identified by processing the sentence using a syntax analyzer, a part-of-speech tagger, and / or a named entity recognizer. In one implementation, for the English language, a syntax analyzer, a part-of-speech tagger, and a named entity recognizer provided by the Stanford Natural Language Processing (NLP) Group are used to analyze sentence structure and syntax. These are provided as part of the Stanford CoreNLP toolkit.

[0049] While various examples provided in this disclosure illustrate utterances in the English language, this is meant only as an example. In certain embodiments, the digital assistant 106 is also capable of handling utterances in languages other than English. The digital assistant 106 can provide subsystems (e.g., components implementing NLU functionality) that are configured to perform processing for different languages. These subsystems can be implemented as pluggable units that can be invoked from an NLU core server using service calls. This makes the NLU processing flexible and scalable for each language, including allowing different processing orders. Language packs can be provided for individual languages, where the language packs can register a list of subsystems that can provide services from the NLU core server.

[0050] The digital assistant (such as the digital assistant 106 depicted in FIG. 1) can be made available or accessible to its users 108 through a variety of different channels, such as but not limited to, via certain applications, via social media platforms, via various messaging services and applications, and other applications or channels. Figure 1 A single digital assistant can configure itself with several channels, so that a single digital assistant can run on and be accessible through different services at the same time.

[0051] A digital assistant or chatbot system typically contains or is associated with one or more skills. In certain embodiments, these skills are individual chatbots (referred to as skill bots) that are configured to interact with users and complete specific types of tasks, such as tracking inventory, submitting time cards, creating expense reports, ordering food, querying bank accounts, making appointments, purchasing widgets, etc. For example, for the depicted embodiment, the digital assistant or chatbot system 106 includes skills 116-1, 116-2, etc. For the purposes of this disclosure, the terms "skill" and "skills" are used synonymously with the terms "skill bot" and "skill bots," respectively. Figure 1

[0052] Each skill associated with a digital assistant helps the user of the digital assistant complete a task through a conversation with the user, which can include a combination of textual or audio input provided by the user and responses provided by the skill bot. These responses can take the form of textual or audio messages to the user and / or using simple user interface elements (e.g., a list of choices) presented to the user for the user to select from.

[0053] ​There are various ways in which skills or skill bots can be associated with or added to a digital assistant. In some instances, a skill bot can be developed by an enterprise and then added to a digital assistant that uses DABP 102. In other instances, a skill bot can be developed and created using DABP 102 and then added to a digital assistant created using DABP 102. In yet other instances, DABP 102 provides an online digital store (referred to as a "skills store") that provides a plurality of skills that relate to a wide variety of tasks. The skills provided through the skills store can also expose various cloud services. To add a skill to a digital assistant generated using DABP 102, a user of DABP 102 can access the skills store via DABP 102, select a desired skill, and instruct that the selected skill be added to a digital assistant created using DABP 102. Skills from the skills store can be added to a digital assistant as-is or in modified form (e.g., a user of DABP 102 can select and copy a particular skill bot provided by the skills store, customize or modify the selected skill bot, and then add the modified skill bot to a digital assistant created using DABP 102).

[0054] A digital assistant or chatbot system can be implemented using a variety of different architectures. For example, in certain embodiments, a digital assistant created and deployed using DABP 102 can be implemented using a master bot / secondary (or child) bot paradigm or architecture. According to this paradigm, a digital assistant is implemented as a master bot that interacts with one or more secondary bots that are skill bots. For example, in the depicted embodiment, digital assistant 106 includes master bot 114 and skill bots 116-1, 116-2, etc. that are secondary bots to master bot 114. In certain embodiments, digital assistant 106 itself is considered to function as a master bot. Figure 1

[0055] ​Digital assistants implemented in accordance with the primary-secondary robot architecture enable users of the digital assistant to interact with multiple skills through a unified user interface (i.e., via the primary robot). When a user engages with the digital assistant, the primary robot receives the user input. The primary robot then performs processing to determine the meaning of the user input utterance. The primary robot then determines whether the task requested in the user's utterance can be handled by the primary robot itself, otherwise, the primary robot selects an appropriate skill robot for handling the user request and routes the conversation to the selected skill robot. This enables users to have conversations with the digital assistant through a common single interface and still have the ability to use several skill robots that are configured to perform specific tasks. For example, for a digital assistant developed for an enterprise, the primary robot of the digital assistant can interface with skill robots having specific functionality, such as a CRM robot for performing functions related to customer relationship management (CRM), an ERP robot for performing functions related to enterprise resource planning (ERP), an HCM robot for performing functions related to human capital management (HCM), etc. In this way, the end user or consumer of the digital assistant only needs to know how to access the digital assistant through the common primary robot interface, while multiple skill robots are provided in the background to handle user requests.

[0056] In certain embodiments, in a primary robot / secondary robot infrastructure, the primary robot is configured to understand a list of available skill robots. The primary robot can access metadata that identifies various available skill robots, and for each skill robot, the skill robot's capabilities include tasks that can be performed by the skill robot. After receiving a user request in the form of an utterance, the primary robot is configured to identify or predict a particular skill robot from the multiple available skill robots that can best serve or handle the user request. The primary robot then routes the utterance (or a portion of the utterance) to that particular skill robot for further handling. Thus, control flows from the primary robot to the skill robot. The primary robot can support multiple input channels and output channels.

[0057] While Figure 1 Embodiments of the digital assistant 106 show that the digital assistant includes a primary robot 114 and skill robots 116-1, 116-2, and 116-3, but this is not intended to be limiting. The digital assistant can include various other components (e.g., other systems and subsystems) that provide the functionality of the digital assistant. These systems and subsystems can be implemented in software only (e.g., code, instructions stored on a computer-readable medium and executable by one or more processors), in hardware only, or in embodiments that use a combination of software and hardware.

[0058] DABP 102 provides infrastructure and various services and features that enable users of DABP 102 to create digital assistants (including one or more skill bots associated with a digital assistant). In some instances, a skill bot can be created by cloning an existing skill bot, e.g., cloning a skill bot provided by a skill store. As previously mentioned, DABP 102 provides a skill store or skill catalog that provides a plurality of skill bots for performing various tasks. A user of DABP 102 can clone a skill bot from the skill store. The cloned skill bot can be modified or customized as needed. In some other instances, a user of DABP 102 creates a skill bot from scratch using tools and services provided by DABP 102. As previously mentioned, the skill store or skill catalog provided by DABP 102 provides a plurality of skill bots for performing various tasks.

[0059] In certain embodiments, at a high level, creating or customizing a skill bot involves the following steps:

[0060] (1) configuring settings for the new skill bot

[0061] (2) configuring one or more intents for the skill bot

[0062] (3) configuring one or more entities for the one or more intents

[0063] (4) training the skill bot

[0064] (5) creating a dialog flow for the skill bot

[0065] (6) adding custom components to the skill bot as needed

[0066] (7) testing and deploying the skill bot

[0067] Each of the above steps is briefly described below.

[0068] (1) configuring settings for the new skill bot - various settings can be configured for a skill bot. For example, a skill bot designer can specify one or more invocation names for the skill bot being created. These invocation names can then be used by users of the digital assistant to explicitly invoke the skill bot. For example, a user can enter an invocation name in a user utterance to explicitly invoke the corresponding skill bot.

[0069] (2) configuring one or more intents and associated example utterances for the skill robot - the skill robot designer specifies one or more intents (also referred to as robot intents) for the skill robot being created. The skill robot is then trained based on these specified intents. These intents represent the categories or classifications that the skill robot is trained to infer for an input utterance. After receiving an utterance, the trained skill robot infers the intent of the utterance, where the inferred intent is selected from a predefined set of intents used to train the skill robot. The skill robot then takes appropriate action to respond to the utterance based on the inferred intent for the utterance. In some instances, the intents of the skill robot represent tasks that the skill robot can perform for a user of the digital assistant. Each intent is given an intent identifier or intent name. For example, for a skill robot trained for a bank, the intents specified for the skill robot can include "CheckBalance", "TransferMoney", "DepositCheck", etc.

[0070] For each intent defined for the skill robot, the skill robot designer can also provide one or more example utterances that represent and illustrate the intent. These example utterances are intended to represent utterances that a user can input to the skill robot for the intent. For example, for the CheckBalance intent, example utterances can include "What's my savings account balance?", "How much is in my checking account?", "How much money do I have in my account", etc. Thus, various permutations of typical user utterances can be specified as example utterances for an intent.

[0071] These intents and their associated example utterances are used as training data for training the skill robot. Various different training techniques can be used. As a result of this training, a predictive model is generated that is configured to take an utterance as input and output the intent that the predictive model infers for the utterance. In some instances, the input utterance is provided to an intent analysis engine that is configured to use the trained model to predict or infer the intent of the input utterance. The skill robot can then take one or more actions based on the inferred intent.

[0072] (3) One or more entities are configured for one or more intents of the skill robot - in some instances, additional context can be needed to enable the skill robot to properly respond to user utterances. For example, there can be instances in which user input utterances resolve to the same intent in the skill robot. For example, in the above example, the utterances "What's my savings account balance?" and "How much is in my checking account?" both resolve to the same CheckBalance intent, but these utterances are different requests that request different things. To clarify such requests, one or more entities are added to the intent. Using the example of a banking skill robot, an entity called AccountType (which defines values called "checking" and "savings") can enable the skill robot to parse the user request and respond appropriately. In the above example, while the utterances resolve to the same intent, the values associated with the AccountType entity for the two utterances are different. This enables the skill robot to perform potentially different actions for the two utterances, despite the fact that the two utterances resolve to the same intent. One or more entities can be specified for certain intents that are configured for a skill robot. Thus, entities are used to add context to the intent itself. Entities help to more fully describe the intent and enable the skill robot to fulfill the user request.

[0073] In certain embodiments, there are two types of entities: (a) built-in entities provided by the DABP 102; and (2) custom entities that can be specified by the skill robot designer. The built-in entities are general-purpose entities that can be used with a variety of robots. Examples of built-in entities include, but are not limited to, entities related to time, date, address, number, email address, duration, recurring time period, currency, phone number, URL, etc. The custom entities are used for more customized applications. For example, for a banking skill, an AccountType entity can be defined by the skill robot designer to enable various banking transactions by checking the keywords (e.g., checking, savings, credit card, etc.) in the user input.

[0074] (4) Training the skill bot - The skill bot is configured to receive user input in the form of utterances, parse or otherwise process the received input and identify or select an intent that is relevant to the received user input. As indicated above, to do this, the skill bot must be trained. In certain embodiments, the skill bot is trained based on intents that are configured for the skill bot and example utterances associated with the intents (collectively, training data) so that the skill bot can parse a user input utterance into one of the intents that it is configured for. In certain embodiments, the skill bot uses a prediction model that is trained using the training data and that allows the skill bot to discern what a user is saying (or in some cases, is trying to say). The DABP 102 provides various different training techniques that can be used by a skill bot designer to train a skill bot, including various machine learning based training techniques, rule based training techniques, and / or combinations thereof. In certain embodiments, a portion of the training data (e.g., 80%) is used to train the skill bot model and another portion (e.g., the remaining 20%) is used to test or validate the model. Once trained, the trained model (sometimes also referred to as a trained skill bot) can be used to handle and respond to user utterances. In certain cases, a user utterance can be a question that only requires a single answer and no further conversation. To handle such cases, a Q&A (question and answer) intent can be defined for the skill bot. This enables the skill bot to output a reply to the user request without having to update the dialog box definition. The Q&A intent is created in a similar manner as a regular intent. The dialog flow for a Q&A intent can be different from the dialog flow for a regular intent.

[0075] (5) Creating a dialog flow for the skill bot - The dialog flow specified for the skill bot describes how the skill bot reacts when resolving different intents of the skill bot in response to received user input. The dialog flow defines the operations or actions that the skill bot will take, e.g., how the skill bot responds to user utterances, how the skill bot prompts the user for input, how the skill bot returns data. The dialog flow is like a script that the skill bot follows Figure 1

[0076] In certain embodiments, the dialog flow for the skill bot defines contains the following three parts:

[0077] ​(a) Context section

[0078] (b) Default transitions section

[0079] (c) State section

[0080] Context section - The skill bot designer can define variables used in the conversation flow in the context section. Other variables that can be named in the context section include, but are not limited to: variables for error handling, variables for built-in entities or custom entities, user variables that enable the skill bot to recognize and save user preferences, etc.

[0081] Default transitions section - Transitions of the skill bot can be defined in the dialog flow state section or in the default transitions section. Transitions defined in the default transitions section act as a fallback and are triggered when no applicable transition is defined within the state or the conditions required to trigger a state transition are not met. The default transitions section can be used to define routes that allow the skill bot to gracefully handle unexpected user actions.

[0082] State section - Dialog flows and their related operations are defined as a sequence of temporary states that manage the logic within the dialog flow. Each state node within the dialog flow definition names a component that provides the functionality required at that point in the dialog. Thus, states are built around components. The state contains properties specific to the component and defines transitions to other states that are triggered after the component executes.

[0083] Special case scenarios can be handled using the state section. For example, you might sometimes want to provide the user with the option to do something in a second skill within the digital assistant that temporarily lets the user engage with a first skill. For example, if the user is engaged in a conversation with a shopping skill (e.g., the user has made some purchase selections), the user might want to jump to a banking skill (e.g., the user might want to make sure he / she has enough money for the purchase) and then return to the shopping skill to complete the user's order. To address this, an action in the first skill can be configured to initiate an interaction with a second, different skill in the same digital assistant and then return to the original flow.

[0084] (6) Adding custom components to the skill bot - As described above, the states specified in the dialog flow of a skill bot are named to correspond to components that provide the functionality required by the state. Components enable the skill bot to perform functionality. In certain embodiments, the DABP 102 provides a set of preconfigured components for performing a wide variety of functionality. A skill bot designer can select one or more of these preconfigured components and associate it with a state in the dialog flow of the skill bot. The skill bot designer can also create custom components or new components using the tools provided by the DABP 102 and associate the custom components with one or more states in the dialog flow of the skill bot.

[0085] (7) Testing and deploying the skill bot - The DABP 102 provides several features that enable a skill bot designer to test the skill bot being developed. The skill bot can then be deployed and included in a digital assistant.

[0086] While the above description describes how to create a skill bot, similar techniques can also be used to create a digital assistant (or host bot). At the host bot or digital assistant level, the digital assistant can be configured with built-in system intents. These built-in system intents are used to identify general tasks that the digital assistant itself (i.e., the host bot) can handle without invoking a skill bot associated with the digital assistant. Examples of system intents defined for the host bot include: (1) Exit: applicable when a user signals that they want to exit the current session or context in the digital assistant; (2) Help: applicable when a user requests help or direction; and (3) Unresolved Intent: applicable to user input that does not closely match the exit intent and the help intent. The digital assistant also stores information about one or more skill bots associated with the digital assistant. This information enables the host bot to select a particular skill bot for handling an utterance.

[0087] At the host bot or digital assistant level, when a user inputs a phrase or utterance to the digital assistant, the digital assistant is configured to perform a process for determining how to route the utterance and the related session. The digital assistant uses a routing model to determine this, which can be rule-based, AI-based, or a combination thereof. The digital assistant uses the routing model to determine whether a session corresponding to the user input utterance is to be routed to a particular skill for handling, to be handled by the digital assistant or host bot itself in accordance with a built-in system intent, or to be handled as a different state in the current session flow.

[0088] In certain embodiments, as part of this process, the digital assistant determines whether the user input utterance explicitly identifies a skill bot using its invocation name. If the invocation name is present in the user input, the invocation name is considered to be an explicit invocation of the skill bot corresponding to the invocation name. In this scenario, the digital assistant can route the user input to the explicitly invoked skill bot for further disposition. In certain embodiments, if there is no specific invocation or explicit invocation, the digital assistant evaluates the received user input utterance and computes confidence scores for system intents and skill bots associated with the digital assistant. The scores computed for skill bots or system intents represent how likely the user input is representative of a task that the skill bot is configured to perform or of a system intent. Any system intent or skill bot whose associated computed confidence score exceeds a threshold (e.g., a confidence threshold routing parameter) is selected as a candidate for further evaluation. The digital assistant then selects a particular system intent or skill bot from the identified candidates for further disposition of the user input utterance. In certain embodiments, after one or more skill bots are identified as candidates, the intents associated with those candidate skills are evaluated (according to each skill’s intent model) and a confidence score is determined for each intent. Any intent whose confidence score exceeds a threshold (e.g., 70%) is generally considered a candidate intent. If a particular skill bot is selected, the user utterance is routed to that skill bot for further processing. If a system intent is selected, one or more actions are performed by the host bot itself according to the selected system intent.

[0089] Figure 2 is a simplified block diagram of a host bot (MB) system 200 according to certain embodiments. The MB system 200 can be implemented in software only, hardware only, or a combination of hardware and software. The MB system 200 includes a pre-processing subsystem 210, a plurality of intent subsystems (MIS) 220, an explicit invocation subsystem (EIS) 230, a skill bot invoker 240, and a data store 250. Figure 2 The depicted MB system 200 is merely an example of an arrangement of components in a host bot. Those of ordinary skill in the art will recognize many possible variations, alternatives, and modifications. For example, in some implementations, the MB system 200 can have more or fewer systems or components than those shown, can combine two or more subsystems, or can have a different configuration or arrangement of subsystems. Figure 2 The MB system 200 can include more or fewer systems or components than those shown, can combine two or more subsystems, or can have a different configuration or arrangement of subsystems.

[0090] The pre-processing subsystem 210 receives the utterance "A" 202 from a user and processes the utterance through a language detector 212 and a language syntax analyzer 214. As noted above, the utterance can be provided in various ways, including audio or text. The utterance 202 can be a sentence fragment, a complete sentence, multiple sentences, etc. The utterance 202 can include punctuation. For example, if the utterance 202 is provided as audio, the pre-processing subsystem 210 can use a speech-to-text converter (not shown) that inserts punctuation (e.g., commas, semicolons, periods, etc.) into the resulting text to convert the audio to text.

[0091] The language detector 212 detects the language of the utterance 202 based on the text of the utterance 202. The way the utterance 202 is handled depends on the language, as each language has its own syntax and semantics. Differences between languages are taken into account when analyzing the syntax and structure of the utterance.

[0092] The language syntax analyzer 214 performs syntax analysis on the utterance 202 to extract part-of-speech (POS) tags for individual linguistic units (e.g., words) in the utterance 202. POS tags include, for example, noun (NN), pronoun (PN), verb (VB), etc. The language syntax analyzer 214 can also tokenize (e.g., convert each word to an individual token) and lemmatize the linguistic units of the utterance 202. A lemma is the principal form of a set of words as represented in a dictionary (e.g., "run" is the lemma for run, runs, ran, running, etc.). Other types of pre-processing that the language syntax analyzer 214 can perform include chunking of compound expressions, e.g., combining "credit" and "card" into a single expression "credit card." The language syntax analyzer 214 can also identify relationships between words in the utterance 202. For example, in some embodiments, the language syntax analyzer 214 generates a dependency tree that indicates which part of the utterance (e.g., a particular noun) is a direct object, which part of the utterance is a preposition, etc. The result of the processing performed by the language syntax analyzer 214 forms extracted information 205 and is provided as input to the MIS 220 along with the utterance 202 itself.

[0093] As noted above, the utterance 202 can include more than one sentence. For the purposes of detecting multiple intents and explicit invocation, the utterance 202 can be treated as a single unit, even if it includes multiple sentences. However, in certain embodiments, pre-processing can be performed, e.g., by the pre-processing subsystem 210, to identify individual sentences in the multiple sentences for multiple intent analysis and explicit invocation analysis. Generally, the results produced by the MIS 220 and the EIS 230 are substantially the same whether the utterance 202 is processed at the level of individual sentences or as a single unit that includes multiple sentences.

[0094] MIS 220 determines whether the utterance 202 represents multiple intents. Although MIS 220 can detect that there are multiple intents in the utterance 202, the processing performed by MIS 220 does not involve determining whether the intents of the utterance 202 match any of the intents that have been configured for the robot. Rather, the processing of determining whether the intents of the utterance 202 match the intents of the robot can be performed by the intent classifier 242 of the MB system 200 or the intent classifier of the skill robot (e.g., as shown in the embodiments of Figure 3 ). The processing performed by MIS 220 assumes that there is a robot (e.g., a particular skill robot or the main robot itself) that can handle the utterance 202. Thus, the processing performed by MIS 220 does not need to know which robots are in the chatbot system (e.g., the identity of the skill robots registered with the main robot) or what intents have been configured for a particular robot.

[0095] To determine that the utterance 202 includes multiple intents, MIS 220 applies one or more rules of a set of rules 252 in the data store 250. The rules applied to the utterance 202 depend on the language of the utterance 202 and can include sentence patterns that indicate the presence of multiple intents. For example, a sentence pattern can include a coordinating conjunction that connects two parts of a sentence (e.g., a conjunction), where the two parts correspond to different intents. If the utterance 202 matches the sentence pattern, then it can be inferred that the utterance 202 represents multiple intents. It should be noted that an utterance with multiple intents does not necessarily have different intents (e.g., intents for different robots or different intents within the same robot). Rather, the utterance can have different instances of the same intent, such as “Place a pizza order using payment account X, then place a pizza order using payment account Y.”

[0096] As part of determining that the utterance 202 represents multiple intents, MIS 220 also determines which parts of the utterance 202 are associated with each intent. MIS 220 constructs a new utterance for individual processing in place of the original utterance for each intent represented in the utterance with multiple intents, e.g., as shown in Figure 2The original utterance 202 can be split into two or more separate utterances, each of which is handled at a time. The MIS 220 uses the extracted information 205 and / or analysis of the utterance 202 itself to determine which of the two or more utterances should be handled first. For example, the MIS 220 can determine that the utterance 202 contains a marker word that indicates that a particular intent should be handled first. The newly formed utterance that corresponds to that particular intent (e.g., one of the utterances 206 or 208) will be sent first to the EIS 230 for further processing. After the conversation triggered by the first utterance has ended (or has been temporarily paused), the next highest priority utterance (e.g., the other one of the utterances 206 or 208) can then be sent to the EIS 230 for processing.

[0097] The EIS 230 determines whether the utterance it receives (e.g., the utterance 206 or the utterance 208) contains a skill robot’s invocation name. In certain embodiments, each skill robot in the chatbot system is assigned a unique invocation name that distinguishes the skill robot from other skill robots in the chatbot system. The list of invocation names can be saved in the data store 250 as part of the skill robot information 254. When an utterance contains a word that matches an invocation name, the utterance is considered an explicit invocation. If the robot is not explicitly invoked, the utterance that the EIS 230 receives is considered an implicit invocation utterance 234 and is input to the main robot’s intent classifier (e.g., the intent classifier 242) to determine which robot to use to handle the utterance. In some instances, the intent classifier 242 will determine that the main robot should handle the implicit invocation utterance. In other instances, the intent classifier 242 will determine to route the utterance to a skill robot for handling.

[0098] The explicit invocation functionality provided by the EIS 230 has several advantages. It can reduce the amount of processing that the main robot has to perform. For example, when there is an explicit invocation, the main robot can not have to perform any intent classification analysis (e.g., using the intent classifier 242) or can have to perform a simplified intent classification analysis to select a skill robot. Thus, the explicit invocation analysis can enable selection of a particular skill robot without resorting to intent classification analysis.

[0099] Moreover, there can be instances of functional overlap between multiple skill robots. This can occur, for example, if the intents handled by two skill robots overlap or are very close to each other. In this case, it can be difficult for the main robot to identify which of the multiple skill robots to select based on intent classification analysis alone. In this scenario, explicit invocation makes it unambiguous as to which particular skill robot to use.

[0100] In addition to determining that the utterance is an explicit invocation, the EIS 230 is also responsible for determining whether any portion of the utterance should be used as input to the skill robot that was explicitly invoked. Specifically, the EIS 230 can determine whether a portion of the utterance is unrelated to the invocation. The EIS 230 can perform this determination by analyzing the utterance and / or analyzing the extracted information 205. The EIS 230 can send the portion of the utterance that is unrelated to the invocation to the invoked skill robot, rather than sending the entire utterance as received by the EIS 230. In some instances, the input to the invoked skill robot is simply formed by deleting any portion of the utterance that is associated with the invocation. For example, "I want to order pizza using Pizza Bot" can be shortened to "I want to order pizza" because "using Pizza Bot" is related to the invocation of the pizza robot, but is unrelated to any processing to be performed by the pizza robot. In some instances, the EIS 230 can reformat the portion to be sent to the invoked robot, e.g., to form a complete sentence. Thus, the EIS 230 not only determines that there is an explicit invocation, but also determines what to send to the skill robot when there is an explicit invocation. In some instances, there can be no text that can be input to the invoked robot. For example, if the utterance is "Pizza Bot," the EIS 230 can determine that the pizza robot is being invoked, but there is no text to be processed by the pizza robot. In such a scenario, the EIS 230 can indicate to the skill robot invoker 240 that there is nothing to send.

[0101] The skill robot invoker 240 invokes skill robots in various ways. For example, the skill robot invoker 240 can invoke a robot in response to receiving an indication 235 that a particular skill robot has been selected as a result of an explicit invocation. The indication 235 can be sent by the EIS 230 along with the input to the explicitly invoked skill robot. In this scenario, the skill robot invoker 240 hands over control of the conversation to the explicitly invoked skill robot. The explicitly invoked skill robot will determine an appropriate response to the input from the EIS 230 by treating the input as a standalone utterance. For example, the response can be to perform a particular action or to begin a new conversation in a particular state, where the initial state of the new conversation depends on the input sent from the EIS 230.

[0102] Another way in which the skill robot invoker 240 can invoke skill robots is through implicit invocation using an intent classifier 242. The intent classifier 242 can be trained using machine learning and / or rule-based training techniques to determine the likelihood that an utterance represents a task that a particular skill robot is configured to perform. The intent classifier 242 is trained on different classifications, one for each skill robot. For example, whenever a new skill robot is registered with the host robot, a list of example utterances associated with the new skill robot can be used to train the intent classifier 242 to determine the likelihood that a particular utterance represents a task that the new skill robot can perform. The parameters that result from this training (e.g., a set of values for the weights and biases of a machine learning model) can be stored as part of the skill robot information 254.

[0103] In certain embodiments, the intent classifier 242 is implemented using a machine learning model, as described in further detail herein. Training of the machine learning model can involve inputting at least one subset of utterances from example utterances associated with various skill robots to generate, as output of the machine learning model, an inference as to which robot is the correct robot for handling any particular training utterance. For each training utterance, an indication of the correct robot for the training utterance can be provided as ground truth information. The behavior of the machine learning model can then be adapted (e.g., through backpropagation) to minimize the difference between the generated inference and the ground truth information.

[0104] In certain embodiments, the intent classifier 242 determines, for each skill robot registered with the host robot, a confidence score indicating the likelihood that the skill robot can handle an utterance (e.g., a non-explicitly invoked utterance 234 received from the EIS 230). The intent classifier 242 can also determine a confidence score for each system-level intent that has been configured (e.g., help, exit). If a particular confidence score satisfies one or more conditions, the skill robot invoker 240 will invoke the robot associated with the particular confidence score. For example, it can be required that a threshold confidence score value be satisfied. Thus, the output 245 of the intent classifier 242 is either an identification of a system intent or an identification of a particular skill robot. In some embodiments, in addition to satisfying a threshold confidence score value, the confidence score must win over the next highest confidence score by a certain margin. Such a condition will enable routing to a particular skill robot when the confidence scores of multiple skill robots all exceed the threshold confidence score value.

[0105] After the robot is identified based on the confidence score evaluation, the skill robot invoker 240 hands off processing to the identified robot. In the case of a system intent, the identified robot is the host robot. Otherwise, the identified robot is a skill robot. Further, the skill robot invoker 240 will determine what to provide as input 247 to the identified robot. As described above, in the case of an explicit invocation, the input 247 can be based on the portion of the utterance that is independent of the invocation, or the input 247 can be nothing (e.g., an empty string). In the case of an implicit invocation, the input 247 can be the entire utterance.

[0106] The data store 250 includes one or more computing devices that store data used by the various subsystems of the host robot system 200. As explained above, the data store 250 includes rules 252 and skill robot information 254. The rules 252 include, for example, rules used by the MIS 220 to determine when an utterance represents multiple intents and how to split an utterance that represents multiple intents. The rules 252 further include rules used by the EIS 230 to determine which portions of an utterance that explicitly invokes a skill robot to send to the skill robot. The skill robot information 254 includes invocation names of skill robots in the chatbot system, e.g., a list of invocation names of all skill robots registered with a particular host robot. The skill robot information 254 can also include information used by the intent classifier 242 to determine confidence scores for each skill robot in the chatbot system, e.g., parameters of a machine learning model.

[0107] Figure 3 is a simplified block diagram of a skill robot system 300 in accordance with certain embodiments. The skill robot system 300 is a computing system that can be implemented in software only, in hardware only, or in a combination of hardware and software. In certain embodiments, the skill robot system 300 can be used to implement one or more skill robots within a digital assistant. Figure 1 As depicted in the depicted embodiment, the skill robot system 300 can be used to implement one or more skill robots within a digital assistant.

[0108] The skill robot system 300 includes a MIS 310, an intent classifier 320, and a session manager 330. The MIS 310 is similar to the MIS 220 in Figure 2 and provides similar functionality, including being operable to use rules 352 in a data store 350 to determine: (1) whether an utterance represents multiple intents, and if so, (2) how to split the utterance into separate utterances for each of the multiple intents. In certain embodiments, the rules applied by the MIS 310 to detect multiple intents and to split utterances are the same as the rules applied by the MIS 220. The MIS 310 receives an utterance 302 and extracted information 304. The extracted information 304 is similar to the extracted information 304 in Figure 1The information extracted from 205 can be generated using a language parser 214 or a language parser native to the skill robot system 300.

[0109] The intent classifier 320 can be combined with the above. Figure 2 The intent classifier 242 discussed in the embodiments is trained in a similar manner and is described in further detail herein. For example, in some embodiments, the intent classifier 320 is implemented using a machine learning model. For a specific skill robot, the machine learning model of the intent classifier 320 is trained using at least a subset of example utterances associated with that specific skill robot as training utterances. The underlying fact of each training utterance will be the specific robot intent associated with the training utterance.

[0110] Phras 302 can be received directly from the user or provided through the main bot. When phrasing 302 is provided through the main bot, for example, as via... Figure 2 The results processed by MIS 220 and EIS 230 in the embodiments depicted can bypass MIS 310 to avoid repeating the processing already performed by MIS 220. However, if utterance 302 is received directly from the user, for example, during a session after routing to a skill bot, MIS 310 can process utterance 302 to determine whether utterance 302 represents multiple intents. If so, MIS 310 applies one or more rules to split utterance 302 into separate utterances for each intent, such as utterance “D” 306 and utterance “E” 308. If utterance 302 does not represent multiple intents, MIS 310 forwards utterance 302 to intent classifier 320 for intent classification without splitting utterance 302.

[0111] Intent classifier 320 is configured to match received utterances (e.g., utterances 306 or 308) with intents associated with the skill robot system 300. As explained above, the skill robot may be configured with one or more intents, each intent including at least one example utterance associated with that intent and used to train the classifier. Figure 2In embodiments, the intent classifier 242 of the host robotic system 200 is trained to determine a confidence score for each skill robot and a confidence score for the system intent. Similarly, the intent classifier 320 can be trained to determine a confidence score for each intent associated with the skill robotic system 300. The classification performed by the intent classifier 242 is at the robot level, while the classification performed by the intent classifier 320 is at the intent level and thus more granular. The intent classifier 320 can have access to intent information 354. For each intent associated with the skill robotic system 300, the intent information 354 includes a list of utterances that are representative of the intent and that illustrate the meaning of the intent and are generally associated with tasks that can be performed by the intent. The intent information 354 can further include parameters that result from training on the list of utterances.

[0112] The session manager 330 receives an indication 322 of the particular intent as output of the intent classifier 320 that is identified by the intent classifier 320 as the best match to the utterance input to the intent classifier 320. In some instances, the intent classifier 320 is unable to determine any match. For example, if the utterance is for a system intent or an intent of a different skill robot, the confidence score computed by the intent classifier 320 can be below a threshold confidence score value. When this occurs, the skill robotic system 300 can submit the utterance to the host robot for handling, e.g., to route to a different skill robot. However, if the intent classifier 320 successfully identifies an intent within the skill robotic system 300, the session manager 330 will initiate a session with the user.

[0113] The session initiated by the session manager 330 is a session specific to the intent identified by the intent classifier 320. For example, the session manager 330 can be implemented using a state machine that is configured to execute a dialog flow for the identified intent. The state machine can include a default start state (e.g., when the intent is invoked without any additional input) and one or more additional states, where each state is associated with an action to be performed by the skill robot (e.g., execute a purchase transaction) and / or a dialog (e.g., questions, responses) to be presented to the user. Thus, the session manager 330 can determine the action / dialog 335 upon receiving the indication 322 that the intent was identified, and can determine additional actions or dialogs in response to subsequent utterances received during the session.

[0114] The data store 350 includes one or more computing devices that store data used by the various subsystems of the skill robotic system 300. As Figure 3 depicted in FIG. 3B, the data store 350 can include rules 352 and intent information 354. In certain embodiments, the data store 350 can be integrated into the data store of the host robot or digital assistant, e.g., the data store 250 of the host robotic system 200. Figure 2data store 250 in the memory 210.

[0115] Stop word data augmentation

[0116] Chatbots can be trained to determine user intent based on user utterances. When there is insufficient training data, chatbots can more frequently determine incorrect intents. For example, when there is insufficient training data associated with an unresolved class of intent, a trained chatbot can incorrectly identify certain utterances as positive or negative classes of that intent. In one example, if the training data for the “I need information about membership” intent includes “do you have a day pass so we can see if we would like a membership,” “do you have a senior citizen discount on membership,” and “do you have a tax-free shopping membership,” a trained chatbot can incorrectly classify any user utterance that includes “do you have” as high confidence associated with the “I need information about membership” intent. One existing option is to remove these high frequency words or a list of words selected for a particular domain (hereinafter referred to as stop words) from the training samples. However, removing or de-emphasizing stop words from training utterances limits the ability of machine learning models to learn features and context associated with intents.

[0117] To overcome this and other problems, various embodiments are directed to techniques to augment training data with stop words (e.g., non-stop words or non-stop word phrases can be randomly replaced with one or more stop words or one or more stop word phrases) to improve the accuracy of intent classification. Stop words generally refer to the most common words in a language, however, there is no single universal list of stop words used by all natural language processing tools. Stop words can include a list of words for a particular domain or intent, or can include high frequency words or any user selected words (e.g., a set of commonly used words in any language, not just English).

[0118] According to some embodiments, out-of-domain (OOD) training samples (e.g., sentences) are generated from in-domain training samples associated with a positive class of an intent and used as training samples associated with an unresolvable class of the intent. More specifically, a stopword augmentation technique (SWAT) automatically populates OOD training samples associated with an unresolvable intent using existing in-domain training samples, where existing stopword patterns in the existing in-domain training samples are preserved and non-stopwords or non-stopword phrases are randomly replaced with one or more stopword or one or more stopword phrases. For example, a training sentence “do you offer a one day trial membership?” can be transformed into a training sample associated with an unresolvable intent “do you to a one say a likely.” Thus, the original stopword “do you...a one...” is preserved and the non-stopword “offer...day trial membership” is replaced with a random stopword “to...say a likely” selected from a list of words from a particular domain or intent or including high frequency words or any user selected words.

[0119] Machine learning models trained on augmented text data can be implemented in chatbot systems as described with respect to Figure 1 、 Figure 2 and Figure 3 . Advantageously, these models and chatbots achieve better accuracy using SWAT because many utterances can include 20-50% stop words and machine learning models can pick up these stopword patterns incorrectly, as described above. By adding OOD sentences generated using SWAT, machine learning models can be prevented from picking up stopword patterns. SWAT can also help to define the boundaries of an intent. Another benefit is that a large number of unresolvable intent utterances can help machine learning models to perform better on OOD sentences. Preserving stopwords in utterances can help machine learning models to distinguish between similar utterances. Thus, higher accuracy and confidence scores can be achieved. Removing or de-emphasizing stopwords from training utterances can not achieve the above benefits.

[0120] Figure 4 A block diagram illustrating aspects of a chatbot system 400 configured to train and utilize a classifier based on text data 405 (e.g., an intent classifier 242 or 320 described with respect to Figure 2 and Figure 3 is shown. As Figure 4As shown, the text classification performed by the chatbot system 400 in this example includes various stages: a prediction model training stage 410; a skillbot invocation stage 415 for determining the likelihood that an utterance represents a task for which a particular skillbot is configured to perform; and an intent prediction stage 420 for classifying an utterance as one or more intents. The prediction model training stage 410 establishes and trains one or more prediction models 425a-425n (‘n’ representing any natural number) (which can be referred to individually as a prediction model 425 or collectively as the prediction models 425) for use by the other stages. For example, the prediction models 425 can include a model for determining the likelihood that an utterance represents a task for which a particular skillbot is configured to perform, another model for predicting an intent from an utterance of a first type of skillbot, and another model for predicting an intent from an utterance of a second type of skillbot. Other types of prediction models can be implemented in other examples in accordance with the present disclosure.

[0121] The prediction models 425 can be machine learning (“ML”) models, such as a convolutional neural network (“CNN”), e.g., an initial neural network, a residual neural network (“Resnet”), or a recurrent neural network, e.g., a long short-term memory (“LSTM”) model or a gated recurrent unit (“GRU”) model; other variants of deep neural networks (“DNNs”), e.g., a multi-label n-way DNN classifier or a multi-class DNN classifier for single-intent classification. The prediction models 425 can also be any other suitable ML model trained for natural language processing, such as a Naive Bayes classifier, a linear classifier, a support vector machine, a bagging model such as a random forest model, a boosting model, a shallow neural network, or a combination of one or more of such techniques, e.g., a CNN-HMM or a MCNN (multi-scale convolutional neural network). The chatbot system 400 can employ the same type of prediction model or different types of prediction models for determining the likelihood that an utterance represents a task for which a particular skillbot is configured to perform, for predicting an intent from an utterance of a first type of skillbot, and for predicting an intent from an utterance of a second type of skillbot. Other types of prediction models can be implemented in other examples in accordance with the present disclosure.

[0122] To train the various prediction models 425, the training phase 410 is composed of three main parts: dataset preparation 430, feature engineering 435, and model training 440. The dataset preparation 430 includes loading data assets 445, splitting the data assets 445 into training and validation sets 445a-n so that the system can train and test the prediction models 425, and performing processes of basic pre-processing. The data assets 445 can include at least one subset of in-domain utterances from example utterances associated with various skill robots. As noted above, the utterances can be provided in various ways including audio or text. The utterances can be sentence fragments, complete sentences, multiple sentences, etc. For example, if the utterances are provided as audio, the data preparation 430 can convert the audio to text using a speech-to-text converter (not shown) that inserts punctuation (e.g., commas, semicolons, periods, etc.) into the resulting text. In some instances, the example utterances are provided by a client or customer. In other instances, the example utterances are automatically generated from a previous corpus of utterances (e.g., identifying utterances from the corpus that are specific to a skill that the chat robot is to learn). The data assets 445 for the prediction models 425 can include input text or audio (or input features of text or audio frames) and labels 450 as matrices or tables of values corresponding to the input text or audio (or input features). For example, for each training utterance, an indication of the correct robot for the training utterance can be provided as basic fact information for the labels 450. The behavior of the prediction models 425 can then be adapted (e.g., through backpropagation) to minimize the difference between the generated inferences and the basic fact information. Alternatively, for a particular skill robot, the prediction models 425 can be trained using at least one subset of example utterances associated with that particular skill robot as training utterances. The basic fact information for the labels 450 for each training utterance would be the particular robot intent associated with the training utterance.

[0123] In various embodiments, data preparation 430 includes stopword augmentation 455 of data asset 445 to include replacing at least one non-stopword in a set of utterances with at least one stopword (e.g., replacing non-stopwords in original data with stopwords) to prevent the prediction model 425 from picking up stopword patterns and improve the accuracy of intent prediction. By augmenting data asset 445 with stopwords replacing non-stopwords, the prediction model 425 becomes better focused on the most important parts of the examples, i.e., tying them to their classification. Augmentation 455 is implemented using SWAT for incorporating stopwords into the original text of data asset 445. SWAT includes a number of operations, generally: (i) obtaining stopword text from a list of words, a corpus of text (e.g., a large structured set of text), a publication, or the original text of data asset 445, and (ii) incorporating the stopword text within data asset 445 relative to the original text at a predefined augmentation rate or augmentation ratio (i.e., the percentage of existing utterances within the original text of data asset 445 that are transformed into augmented utterances). For example, the stopword in each existing training sample can be preserved, while the non-stopword(s) or non-stopword pattern(s) (e.g., phrases that include only non-stopwords) in the existing training sample can be replaced with one or more random stopword or stopword pattern at an augmentation ratio of 1 :0.05 to 1 : 1 (e.g., 1 :0.1 or 1 :0.5)—original utterance: augmented utterance. The augmentation rate can be varied to generate a desired number of augmented samples.

[0124] In some embodiments, SWAT uses existing training samples associated with a positive or negative class of an intent to generate training samples associated with an un-resolved intent. More specifically, the stopword in each existing training sample associated with a positive or negative class of an intent is preserved, while each non-stopword or non-stopword pattern in the existing training sample is replaced with one or more random stopword or stopword pattern. For example, a training sentence with "do you..." can be transformed into an augmented training sentence:

[0125] • do you not what will be on with the empty after ours

[0126] • do you've a into front

[0127] • do you to a one say a likely

[0128] • do you 'll a thereupon to any way a she from the made

[0129] • do you have a yet always so we can see if we would like a nothing

[0130] • do you have a ye n’t against on off

[0131] • do you have a lew whoever thereupon for me

[0132] and so on. The new augmented training sentences generated in this way can be used as training data associated with the unresolved class of intents. The stop words can include a selected set of words for a particular domain, some high frequency words, some “non-content” or “nonsense” words, user-selected words, and so on. The augmentation rate (e.g., the percentage of existing utterances that are transformed) can be varied to generate a desired number of augmented samples.

[0133] In some embodiments, additional training samples associated with an unresolved intent are generated by searching for the longest contiguous stop word at the beginning of an existing utterance. For example, from the utterance “How do I get the refund?” the start pattern “how do i” can be extracted and used as at least a portion of a training sample associated with the unresolved intent. In other embodiments, additional training samples associated with an unresolved intent can be generated by searching for up to n contiguous stop words (referred to as n-grams) anywhere in an existing utterance. For example, in the utterance “How do I get there refund?” the n-grams of n = 5 can include “how,” “do,” “i,” “how do,” “do i,” “how do i,” and “the.” The n-grams can be used as at least a portion of a training sample associated with the unresolved intent.

[0134] With the introduction of the SWAT, the prediction model 425 performs better on utterances with common stop word patterns and regular utterances (without many stop words) because the prediction model 425 is able to better focus on the important parts of the utterance. Moreover, since the SWAT is applied automatically in a synthetic-agnostic manner, the customer or client does not need to worry about adding stop words in the training data. As shown in Table 1, the stop word patterns are learned incorrectly for intents associated with the UM Flowers skill. However, once the SWAT automatically populates the OOD training samples associated with the unresolved intent with stop word utterances generated from existing in-domain training samples, the expected confidence numbers for the prediction model 425 are achieved.

[0135] Table 1:

[0136]

[0137]

[0138]

[0139] In some instances, additional augmentations can be applied to the data assets 445 (using stopword augmentation). For example, a simple data augmentation (EDA) technique can be used to improve the performance of a text classification task. EDA includes four operations: synonym replacement, ransom insertion, random swap, and random deletion, which can prevent overfitting and help train more robust models. Note that, in comparison to stopword augmentation, EDA operations generally: (i) obtain words from the original text, and (ii) incorporate these words within each data asset 445 relative to the original text. For example, the synonym replacement operation includes randomly selecting n words from an original sentence (e.g., utterance) that are not stop words, and replacing each of these words with one of its randomly selected synonyms. The random insertion operation includes, n times, finding a random synonym of a random word in the original sentence that is not a stop word, and inserting the synonym into a random position in the sentence. The random swap operation includes, n times, randomly selecting two words in the sentence and swapping their positions. The random deletion operation includes randomly deleting each word in the sentence with probability p.

[0140] In various embodiments, the feature engineering 435 includes transforming the data assets 445 (using stopword augmentation) into feature vectors and / or creating new features that will be created using the data assets 445 (using stopword augmentation). The feature vectors can include count vectors as features, term frequency-inverse document frequency (TF-IDF) vectors as features (e.g., at word level, n-gram level, or character level), word embeddings as features, text / NLP as features, topic modeling as features, or a combination thereof. A count vector is a matrix notation of the data assets 445, where each row represents an utterance, each column represents a word from the utterance, and each cell represents the frequency count of a particular word in the utterance. A TF-IDF score represents the relative importance of a word in an utterance. Word embeddings are a form of representing words and utterances using dense vectors. The position of a word in the vector space is learned from the text and is based on the words surrounding the word when it is used. Text / NLP-based features can include the number of words in an utterance, the number of characters in an utterance, the average word density, the number of punctuation marks, the number of capital letters, the number of title words, the frequency distribution of part-of-speech tags (e.g., nouns and verbs), or any combination thereof. Topic modeling is a technique that identifies groups of words (called topics) from a set of utterances that contain the best information in the set.

[0141] In various embodiments, model training 440 includes training a classifier using the feature vectors and / or new features created in feature engineering 435. In some instances, the training process includes an iterative operation to find a set of parameters for the prediction model 425 that minimizes a loss or error function for the prediction model 425. Each iteration can involve finding a set of parameters for the prediction model 425 such that the value of the loss or error function using that set of parameters is less than the value of the loss or error function using another set of parameters in a previous iteration. The loss or error function can be constructed to measure the difference between the output predicted using the prediction model 425 and the labels 450 contained in the data asset 445. Once the set of parameters is identified, the prediction model 425 has been trained and can be used for prediction according to design.

[0142] In addition to the data asset 445, labels 450, feature vectors, and / or new features, other techniques and information can be employed to refine the training process for the prediction model 425. For example, the feature vectors and / or new features can be combined together to help improve the accuracy of the classifier or model. Additionally or alternatively, hyperparameters can be adjusted or optimized, for example, multiple parameters such as tree length, leaf, network parameters, etc. can be fine-tuned for the best fitting model. Although the training mechanisms described herein primarily focus on training the prediction model 425. These training mechanisms can also be used to fine-tune existing prediction models 425 trained from other data assets. For example, in some cases, the prediction model 425 can have been pre-trained using utterances specific to another skill robot. In these cases, the prediction model 425 can be retrained using the data asset 445 (augmented with stop words) as discussed herein.

[0143] The prediction model training stage 410 outputs trained prediction models 425, including a task prediction model 460 and an intent prediction model 465. The task prediction model 460 can be used in the skill robot invocation stage 415 to determine the likelihood that an utterance represents a task that a particular skill robot is configured to perform 470, and the intent prediction model 465 can be used in the intent prediction stage 420 to classify an utterance as one or more intents 475. In some instances, the skill robot invocation stage 415 and the intent prediction stage 420 can be performed independently by separate models in some examples. For example, the trained intent prediction model 465 can be used in the intent prediction stage 420 to predict the intent of a skill robot without first identifying the skill robot in the skill robot invocation stage 415. Similarly, the task prediction model 460 can be used in the skill robot invocation stage 415 to predict the task or skill robot to use for an utterance without identifying the intent of the utterance in the intent prediction stage 420.

[0144] Alternatively, the skill robot invocation stage 415 and the intent prediction stage 420 can be performed sequentially, with one stage using the output of the other stage as input, or for a particular skill robot, one stage is invoked based on the output of the other stage in a particular manner. For example, for a given textual data 405, the skill robot invoker can use the skill robot invocation stage 415 and a task prediction model 460 to invoke a skill robot through implicit invocation. The task prediction model 460 can be trained using machine learning and / or rule-based training techniques to determine the likelihood that an utterance is representative of a task that a particular skill robot 470 is configured to perform. The intent prediction stage 420 and an intent prediction model 465 can then be used to match the received utterance (e.g., an utterance within a given data asset 445) to an intent 475 associated with the skill robot for the identified or invoked skill robot and the given textual data 405. As explained herein, a skill robot can be configured with one or more intents, each intent including at least one example utterance associated with the intent and used to train a classifier. In some embodiments, the skill robot invocation stage 415 and the task prediction model 460 used in the host robot system are trained to determine a confidence score for each skill robot and a confidence score for a system intent. Similarly, the intent prediction stage 420 and the intent prediction model 465 can be trained to determine a confidence score for each intent associated with the skill robot system. The classification performed by the skill robot invocation stage 415 and the task prediction model 460 is at the robot level, while the classification performed by the intent prediction stage 420 and the intent prediction model 465 is at the intent level and is thus more granular.

[0145] Techniques for stop word data augmentation and utterance classification

[0146] Figure 5 is a flowchart illustrating a process 500 of augmenting a training dataset with stop words, in accordance with certain embodiments. Figure 5 The depicted processes can be implemented in software (e.g., code, instructions, programs) executed by one or more processing units (e.g., processors, cores) of the respective systems, hardware, or combinations thereof. The software can be stored on a non-transitory storage medium (e.g., on a memory device). Figure 5 The methods presented and described below are intended to be illustrative and not limiting. Although Figure 5 Various processing steps are depicted as occurring in a particular sequence or order, this is not intended to be limiting. In certain alternative embodiments, the steps can be performed in a different order or some steps can also be performed in parallel. In certain embodiments, as Figures 1-4 In the embodiments depicted in FIG. 4, Figure 5The depicted processing can be performed by a pre-processing subsystem (e.g., pre-processing subsystem 210 or prediction model training stage 410) to generate a stopword augmented dataset for training by one or more prediction models (e.g., intent classifier 242 or 320 or prediction model 425).

[0147] At 505, a set of utterance training is received. The set of utterance training includes in-domain utterances and is received for training an intent classifier to recognize one or more intents from one or more utterances. In-domain utterances are those utterances that are relevant to the one or more intents to be recognized by the classifier. For example, an in-domain utterance for a pizza ordering intent of a restaurant ordering skill can be “I want to order a pizza.” The set of utterance training is received by a data processing system (e.g., about chatbot system 400 described) and the intent classifier, such as prediction model 425, is implemented in one or more computing systems, such as chatbot system, as described about Figure 4 Figure 1 、 Figure 2 and Figure 3 .

[0148] At 510, the data processing system augments the set of utterance training with stop words to generate an augmented set of out-of-domain utterances for an out-of-parsed intent category corresponding to an out-of-parsed intent. In various embodiments, the augmentation includes: (i) selecting one or more utterances from the set of utterance training, and (ii) for each selected utterance, retaining existing stop words and / or stop word phrases within the utterance and replacing at least one non-stop word within the utterance with a stop word or stop word phrase (two or more stop words combined to produce a phrase) selected from a list of stop words to generate an out-of-domain utterance. Out-of-domain utterances are those utterances that are not relevant to the one or more intents to be recognized by the classifier. For example, an out-of-domain utterance for the same pizza ordering intent of a restaurant ordering skill can be “What’s the weather like today?” While the classifier can easily discern that the user utterance or message is unclassifiable as it is clearly different from the set of utterance training, the classifier should still learn to classify such utterances as an out-of-parsed intent. An out-of-parsed intent is defined for utterances that include out-of-domain utterances that should not be parsed by the classifier as any skill intent.

[0149] ​In some instances, for each selected utterance, existing stop words within the utterance are retained and all non-stop words within the utterance are replaced with a stop word or stop word phrase selected from a stop word list. In other instances, at least one non-stop word is one, two, three, four, or five of the non-stop words within the utterance. In other instances, for each selected utterance, existing stop words in the utterance are retained and at least one of the non-stop words within the utterance is randomly replaced with a stop word or stop word phrase selected from a stop word list. The stop word list can be generated from a list of common words, a corpus of text, publications, raw text from the utterance training set, or any combination thereof.

[0150] As should be appreciated, one or more utterances from the training instance set are retained to maintain the utterance training set for training the intent classifier to identify one or more intents from the one or more utterances, while establishing an additional ‘augmented’ out-of-domain utterance training set for the unresolved intent category corresponding to the unresolved intent. Further, one or more utterances are selected and processed (i.e., retaining stop words and / or replacing non-stop words) to generate the augmented out-of-domain utterance training set based on a predefined augmentation ratio. This can include: (i) iteratively selecting one or more utterances from the utterance training set and processing each utterance a single time to generate a corresponding single out-of-domain utterance based on the predefined augmentation ratio, (ii) selecting one or more utterances from the utterance training set and processing the one or more utterances multiple times to generate multiple out-of-domain utterances from each utterance based on the predefined augmentation ratio, or (iii) any combination thereof. The predefined augmentation ratio is between 1:0.05 to 1:1 (original utterance: augmented utterance).

[0151] In some embodiments, selecting the utterance includes: iteratively or randomly selecting utterances from the utterance training set. In some embodiments, selecting the utterance includes: searching for a contiguous stop word (i.e., a contiguous combination of two or more stop words) at the beginning of the utterances within the utterance training set and selecting the utterance having the longest contiguous stop word at the beginning of the utterance within the utterance training set. In some embodiments, selecting the utterance includes: searching for n contiguous stop words at any location in the utterances in the utterance training set and selecting the utterance having the n contiguous stop words at any location in the utterance. As a supplement or alternative to replacing at least one non-stop word within the utterance with a stop word or stop word phrase selected from a stop word list, the longest contiguous stop word at the beginning of the utterance and / or the n contiguous stop words at any location in the utterance can all be extracted and used as an out-of-domain utterance.

[0152] At point 515, an expanded utterance training set is used to train a predictive model to determine the probability that an utterance represents a task configured to be performed by the skill bot or to match the utterance with an intent associated with the skill bot. Subsequently, at point 520, the trained predictive model can be deployed within the chatbot system (e.g., as part of the skill bot) to determine the probability that an utterance represents a task configured to be performed by the skill bot or to match the utterance with an intent associated with the skill bot. For example, an utterance can be received, and the utterance can be analyzed to determine whether the utterance contains a call name for the skill bot. If no call name is found, the utterance is considered an indirect call, and the process continues using an intent classifier (such as the trained model). If a call name is determined to exist, the utterance is considered an explicit call, and the process continues to determine which parts of the utterance are associated with that call name.

[0153] In instances where the trained model is invoked, the entire received utterance is provided as input to the intent classifier. The intent classifier receiving the utterance can be the main robot's intent classifier (e.g., ...). Figure 2 The intent classifier (242) can be a machine learning-based or rule-based classifier trained using stop-word augmented data to determine whether the intent of a utterance matches a system intent (e.g., exit, help) or a specific skill bot. As explained herein, intent analysis performed by the master bot can be limited to matching against a specific skill bot without determining which intent within the specific skill bot is the best match for the utterance. Thus, the intent classifier of the received utterance can identify the specific skill bot to invoke. Alternatively, if the utterance represents a specific system intent (e.g., the utterance contains the words "exit" or "help"), then the intent classifier of the received utterance can identify the specific system intent to trigger a conversation between the master bot and the user based on the dialogue flow configured for that specific system intent.

[0154] In instances where the invocation name is present, one or more explicit invocation rules are applied to determine which parts of the utterance are associated with the invocation name. The determination can be based on an analysis of the sentence structure of the utterance using part-of-speech tags, dependency information, and / or other extracted information received with the utterance. For example, the parts associated with the invocation name can be the noun phrase that includes the invocation name or the prepositional object that corresponds to the invocation name. Any parts of the utterance that are determined to be associated with the invocation name based on the processing are removed. Other parts of the utterance that are not needed to convey the meaning of the utterance (e.g., prepositions) can also be removed. Removing certain parts of the utterance produces an input to the skill bot that is associated with the invocation name. If any parts of the received utterance remain after the removal, the remaining parts form a new utterance to input to the skill bot, e.g., as a text string. Otherwise, if the received utterance is completely removed, the input can be an empty string.

[0155] Thereafter, the skill bot associated with the invocation name is invoked and provided with the produced input. Upon receiving the produced input, the invoked skill bot disposes of the input, e.g., by performing intent analysis using an intent classifier of the skill bot that is trained with data augmented with stop words, to identify a robot intent that matches the intent expressed in the user input. The identification of the matching robot intent can cause the skill bot to perform a particular action or start a conversation with the user in accordance with a dialog flow associated with the matching robot intent. For example, if the input is an empty string, the conversation can start in a default state defined for the dialog flow, e.g., a welcome message. Alternatively, if the input is not an empty string, the conversation can start in some intermediate state, e.g., because the input contains values for entities or some other information that the skill bot no longer needs to ask the user for as a result of having received that information as part of the input. As another example, the skill bot can decide that it cannot dispose of the input (e.g., because the confidence score for each robot intent configured for the skill bot is below some threshold). In this case, the skill bot can return the input to the host bot for disposal (e.g., using an intent classifier of the host bot for intent analysis), or the skill bot can prompt the user for clarification.

[0156] In various embodiments, deploying and using an intent classifier within a chatbot system includes receiving, by the chatbot system, an utterance generated by a user interacting with the chatbot system; using an intent classifier deployed in the chatbot system, classifying the utterance into a resolved intent category corresponding to a resolved intent or an unresolved intent category corresponding to an unresolved intent; and using the intent classifier, outputting the resolved intent or the unresolved intent based on the classification. The intent classifier includes a plurality of model parameters identified using training data including: a set of utterance training data used to train the intent classifier to identify one or more resolved intents for one or more utterances; and an augmented out-of-domain utterance training set used to train the intent classifier to identify one or more unresolved intents for one or more utterances, wherein the augmented out-of-domain utterance training set is artificially generated to include utterances from the set of utterance training data, wherein existing stop word patterns within the utterances are preserved, and at least one non-stop word in each utterance is randomly replaced with a stop word. The plurality of model parameters are identified based on minimizing a loss function using the training data.

[0157] Illustrative system

[0158] Figure 6 A simplified diagram depicting a distributed system 600 is shown. In the illustrated example, the distributed system 600 includes one or more client computing devices 602, 604, 606, and 608 coupled to a server 612 over one or more communications networks 610. The client computing devices 602, 604, 606, and 608 can be configured to execute one or more applications.

[0159] In various examples, the server 612 can be adapted to execute one or more services or software applications that implement one or more embodiments described in the present disclosure. In certain examples, the server 612 can also provide other services or software applications that can include non-virtual and virtual environments. In some examples, these services can be provided as web-based services or cloud services (as under a software as a service (SaaS) model) to users of the client computing devices 602, 604, 606, and / or 608. Users operating the client computing devices 602, 604, 606, and / or 608 can in turn utilize one or more client applications to interact with the server 612 to utilize the services provided by these components.

[0160] In Figure 6In the depicted configuration, server 612 may include one or more components 618, 620, and 622 that implement the functions performed by server 612. These components may include software components that can be executed by one or more processors, hardware components, or a combination thereof. It should be understood that various different system configurations, different from distributed system 600, are possible. Therefore, Figure 6 The example shown is an example of a distributed system for implementing the example system and is not intended to be restrictive.

[0161] Users can use client computing devices 602, 604, 606, and / or 608 to execute one or more applications, models, or chatbots, which can generate one or more events or models that can then be implemented or serviced according to the teachings of this disclosure. The client device can provide an interface that enables users of the client device to interact with it. The client device can also output information to the user via this interface. Although Figure 6 It describes only four client computing devices, but can support any number of client computing devices.

[0162] Client devices can include various types of computing systems, such as portable handheld devices, general-purpose computers like personal computers and laptops, workstation computers, wearable devices, gaming systems, thin clients, various messaging devices, sensors, or other sensing devices. These computing devices can run various types and versions of software applications and operating systems (e.g., Microsoft). Apple Or UNIX-like operating systems, such as Google Chrome TM OS (such as Linux or Linux-like operating systems), including various mobile operating systems (e.g., Microsoft Windows) Windows Android TM , Palm Portable handheld devices can include cellular phones, smartphones (e.g., ), tablet computers (e.g., Wearable devices include Google's personal digital assistants (PDAs), personal digital assistants (PDAs), and more. Head-mounted displays and other devices. Gaming systems can include various handheld gaming devices, internet-enabled gaming devices (e.g., with or without internet access). Microsoft gesture input devices Game console, Sony System, by The various game systems, and other) provided. The client devices can be capable of executing a variety of different application programs, such as various Internet-related applications, communication applications (e.g., e-mail applications, short message service (SMS) applications), and can use a variety of communication protocols.

[0163] The network(s) 610 can be any type of network familiar to those skilled in the art that can support data communications using any of a variety of available protocols, including without limitation TCP / IP (transmission control protocol / Internet protocol), SNA (systems network architecture), IPX (Internet packet exchange), AppleTalk, and the like. Merely by way of example, the network(s) 610 can be a local area network (LAN), a wide-area network (WAN), the Internet, a virtual private network (VPN), an intranet, an extranet, a public switched telephone network (PSTN), an infra-red network, a wireless network (e.g., a network operating under any of the Institute of Electrical and Electronics (IEEE) 1002.11 suite of protocols, Bluetooth®, and / or any other wireless protocol), and / or any combination of these and / or other networks.

[0164] The server 612 can be composed of one or more general purpose computers, specialized server computers (including by way of example PC (personal computer) servers, UNIX® servers, mid-range servers, mainframe computers, rack-mounted servers, etc.), server arrays, server clusters, or any other suitable arrangement and / or combinations thereof. The server 612 can include one or more virtual machines running virtual operating systems, or other computing architectures involving virtualization, such as one or more flexible pools of virtual storage devices that can be virtualized to maintain virtual storage devices for the server. In various examples, the server 612 can be adapted to execute one or more services or software applications associated with the functionality described herein.

[0165] The computing systems in the server 612 can run a variety of operating systems including any of those mentioned above and / or any commercially available server operating system. The server 612 can also be running one or more additional server applications and / or mid-tier applications, including HTTP (hypertext transport protocol) servers, FTP (file transfer protocol) servers, CGI (common gateway interface) servers, JAVA® servers, database servers, and the like. Examples of database servers include without limitation those commercially available from Oracle, Microsoft, Sybase, IBM (IBM® DB2®, Informix®, DB2®, Solid Server, and the like). In some examples, the server 612 can be adapted to run the functionality described herein as one or more services of a web application. ​​Those database servers acquired through commercial purchases such as IBM.

[0166] In some implementations, server 612 may include one or more applications to analyze and merge data feeds and / or event updates received from users of client computing devices 602, 604, 606, and 608. As an example, data feeds and / or event updates may include, but are not limited to, those provided in the original text. feed, The server 612 may receive real-time updates from one or more third-party information sources and continuous data streams. These real-time updates may include real-time events related to sensor data applications, financial reporting machines, network performance measurement tools (e.g., network monitoring and traffic management applications), clickstream analysis tools, vehicle traffic monitoring, and the like. The server 612 may also include one or more applications to display data feeds and / or real-time events via one or more display devices of client computing devices 602, 604, 606, and 608.

[0167] The distributed system 600 may also include one or more data repositories 614, 616. In some examples, these data repositories may be used to store data and other information. For example, one or more of data repositories 614, 616 may be used to store information (such as information related to chatbot performance or generated models) for use by the chatbot, which is used by server 612 in performing various functions according to various embodiments. Data repositories 614, 616 may reside in various locations. For example, the data repository used by server 612 may be local to server 612 or may be located remotely to server 612 and communicate with server 612 via a network-based or dedicated connection. Data repositories 614, 616 may be of different types. In some examples, the data repository used by server 612 may be a database, such as a relational database, like one by Oracle. Databases from other vendors. One or more of these databases can be adapted to respond to SQL-formatted commands to store, update, and retrieve data from the database.

[0168] In some examples, one or more of the data stores 614 and 616 can also be used by the application to store application data. The data store used by the application can be of different types, such as a key-value store, an object store, or a general-purpose storage store supported by a file system.

[0169] In some examples, the functionality described in this disclosure can be provided as a service through a cloud environment. Figure 7 This is a simplified block diagram of a cloud-based system environment in which various services, based on certain examples, can be provided as cloud services.Figure 7 In the depicted example, cloud infrastructure system 702 can provide one or more cloud services that can be requested by a user using one or more client computing devices 704, 706, and 708. Cloud infrastructure system 702 may include one or more computers and / or servers, which may include those computers and / or servers described above with respect to server 612. The computers in cloud infrastructure system 702 may be organized as general-purpose computers, dedicated server computers, server groups, server clusters, or any other suitable arrangement and / or combination.

[0170] Multiple networks 710 can facilitate data communication and exchange between clients 704, 706, and 708 and cloud infrastructure system 702. Multiple networks 710 may include one or more networks. The networks may be of the same or different types. Multiple networks 710 may support one or more communication protocols (including wired and / or wireless protocols) to facilitate communication.

[0171] Figure 7 The example depicted is merely one example of a cloud infrastructure system and is not intended to be limiting. It should be understood that in other examples, the cloud infrastructure system 702 may have more than... Figure 7 The depicted components may have more or fewer components, may combine two or more components, or may have different component configurations or arrangements. For example, although... Figure 7 Three client computing devices are depicted, but in alternative examples, any number of client computing devices can be supported.

[0172] The term cloud service is generally used to refer to services that become available to users on demand through a service provider's systems (e.g., cloud infrastructure systems 702) and via communication networks such as the Internet. Typically, in a public cloud environment, the servers and systems that make up the cloud service provider's systems are different from the customer's own on-premises servers and systems. The cloud service provider's systems are managed by the cloud service provider. Therefore, customers can utilize cloud services provided by the cloud service provider without having to purchase separate licenses, support, or hardware and software resources for the services. For example, the cloud service provider's systems can host applications, and users can subscribe to and use the applications on demand via the Internet without having to purchase the infrastructure resources to run the applications. Cloud services are designed to provide easy, scalable access to applications, resources, and services. Several providers offer cloud services. For example, Oracle, based in Redwood Shores, California. It provides several cloud services, such as middleware services, database services, Java cloud services, and other services.

[0173] In some examples, cloud infrastructure system 702 may provide one or more cloud services using different models, such as Software as a Service (SaaS), Platform as a Service (PaaS), Infrastructure as a Service (IaaS), and other models (including hybrid service models). Cloud infrastructure system 702 may include a set of applications, middleware, databases, and other resources that enable the provisioning of various cloud services.

[0174] The SaaS model enables applications or software to be delivered as a service to customers via communication networks such as the Internet, without the customer having to purchase the underlying application's hardware or software. For example, the SaaS model can be used to provide customers with access to on-demand applications hosted by a cloud infrastructure system. Oracle Examples of the SaaS services offered include, but are not limited to, various services for human resources / capital management, customer relationship management (CRM), enterprise resource planning (ERP), supply chain management (SCM), enterprise performance management (EPM), analytics services, social applications, and others.

[0175] The IaaS model is typically used to provide customers with infrastructure resources (such as servers, storage, hardware, and networking resources) as cloud services to offer elastic computing and storage capabilities. (Produced by Oracle) It provides a variety of IaaS services.

[0176] The PaaS model is typically used to provide a platform and environment resources as a service that enable customers to develop, run, and manage applications and services without requiring them to purchase, build, or maintain such resources. (Provided by Oracle) Examples of PaaS services provided include, but are not limited to, Oracle Java Cloud Service (JCS), Oracle Database Cloud Service (DBCS), Data Management Cloud Service, various application development solutions services, and other services.

[0177] Cloud services are typically delivered in an on-demand, self-service, subscription-based, elastically scalable, reliable, highly available, and secure manner. For example, a customer may subscribe to one or more services provided by cloud infrastructure system 702 via a subscription order. Cloud infrastructure system 702 then performs processing to provide the service requested in the customer's subscription order. For instance, a user may use utterances to request the cloud infrastructure system to take an action (e.g., an intent) as described above and / or to provide services for a chatbot system as described herein. Cloud infrastructure system 702 can be configured to provide one or more cloud services.

[0178] Cloud infrastructure system 702 can provide cloud services through different deployment models. In a public cloud model, cloud infrastructure system 702 can be owned by a third-party cloud service provider, and cloud services are provided to any general public customer, which can be an individual or a business. In some other examples, under a private cloud model, cloud infrastructure system 702 can operate within an organization (e.g., within a business organization), and services are provided to customers within the organization. For example, customers can be various departments within the organization, such as human resources or payroll departments, or even individuals within the organization. In some other examples, under a community cloud model, cloud infrastructure system 702 and the services provided can be shared by several organizations in the relevant community. Various other models, such as hybrids of the models mentioned above, can also be used.

[0179] Client computing devices 704, 706, and 708 can be of different types (e.g., Figure 6 The depicted client computing devices (602, 604, 606, and 608) may also be capable of operating one or more client applications. Users can use the client devices to interact with the cloud infrastructure system 702, such as requesting services provided by the cloud infrastructure system 702. For example, a user can use the client device to request information or actions from a chatbot as described in this disclosure.

[0180] In some examples, the processing performed by cloud infrastructure system 702 to provide services may involve model training and deployment. This analysis may involve using, analyzing, and manipulating datasets to train and deploy one or more models. This analysis may be performed by one or more processors, potentially processing data in parallel, performing simulations using the data, etc. For example, big data analysis may be performed by cloud infrastructure system 702 to generate and train one or more models for a chatbot system. The data used for this analysis may include structured data (e.g., data stored in a database or structured according to a structured model) and / or unstructured data (e.g., data blocks (binary large objects)).

[0181] like Figure 7 As illustrated in the examples, cloud infrastructure system 702 may include infrastructure resources 730 used to facilitate the provision of various cloud services offered by cloud infrastructure system 702. Infrastructure resources 730 may include, for example, processing resources, storage or memory resources, networking resources, etc. In some examples, a storage virtual machine that can be used to service storage requested from an application may be part of cloud infrastructure system 702. In other examples, the storage virtual machine may be part of a different system.

[0182] In some examples, to facilitate the efficient provisioning of these resources to support the various cloud services provided by the cloud infrastructure system 702 to different customers, resources can be bound to resource groups or resource modules (also known as "pods"). Each resource module or pod may include a pre-integrated and optimized combination of one or more types of resources. In some examples, different pods may be pre-provisioned for different types of cloud services. For example, a first pod may be provisioned for a database service, and a second pod may be provisioned for a Java service (the second pod may include a different combination of resources than the pods in the first pod), and so on. For some services, resources allocated for provisioning the service can be shared between services.

[0183] The cloud infrastructure system 702 itself can internally use services 732 shared by different components of the cloud infrastructure system 702 and that facilitate the provision of services by the cloud infrastructure system 702. These internal shared services may include, but are not limited to, security and identity services, integration services, enterprise repository services, enterprise manager services, virus scanning and whitelisting services, high availability, backup and recovery services, services for enabling cloud support, email services, notification services, file transfer services, etc.

[0184] Cloud infrastructure system 702 may include multiple subsystems. These subsystems may be implemented in software or hardware, or a combination thereof. Figure 7 The depicted subsystem may include a user interface subsystem 712 that enables users or clients of the cloud infrastructure system 702 to interact with the cloud infrastructure system 702. The user interface subsystem 712 may include various interfaces, such as a web interface 714, an online store interface 716 (where advertising and the client can purchase cloud services provided by the cloud infrastructure system 702), and other interfaces 718. For example, a client may use a client device to request (service request 734) one or more services provided by the cloud infrastructure system 702 using one or more of the interfaces 714, 716, and 718. For example, a client may access an online store, browse cloud services provided by the cloud infrastructure system 702, and place a subscription order for one or more services provided by the cloud infrastructure system 702 that the client wishes to subscribe to. A service request may include information identifying the client and the one or more services the client wishes to subscribe to. For example, a client may place a subscription order for services provided by the cloud infrastructure system 702. As part of the order, the client may provide information identifying the service to be provided to a chatbot system and optionally provide one or more credentials for the chatbot system.

[0185] In some examples (such as) Figure 7In the illustrated example, cloud infrastructure system 702 may include an order management subsystem (OMS) 720 configured to process new orders. As part of this process, OMS 720 may be configured to: create an account for a customer (if not already created); receive billing and / or billing information from the customer to be used to bill the customer for the requested services; verify customer information; place an order for the customer after verification; and schedule various workflows to prepare the order for delivery.

[0186] Once correctly verified, OMS 720 can invoke the Order Provisioning Subsystem (OPS) 724, configured as order provisioning resources (including processing resources, storage resources, and networking resources). Provisioning may include allocating resources for an order and configuring resources to facilitate the service requested by the customer's order. The manner in which resources are provisioned for an order and the type of resources provided may depend on the type of cloud service the customer has subscribed to. For example, according to a workflow, OPS 724 may be configured to determine the specific cloud service being requested and identify the number of groups that may have been pre-configured for that specific cloud service. The number of groups allocated to an order may depend on the size / volume / tier / scope of the requested service. For example, the number of groups to be allocated may be determined based on the number of users the service is to support, the duration of the requested service, etc. The allocated groups can then be customized for a specific requesting customer to provide the requested service.

[0187] In some examples, the setup phase processing described above can be performed by cloud infrastructure system 702 as part of the provisioning process. Cloud infrastructure system 702 can generate application IDs and select storage virtual machines for applications from storage virtual machines provided by cloud infrastructure system 702 itself or from storage virtual machines provided by other systems besides cloud infrastructure system 702.

[0188] Cloud infrastructure system 702 may send a response or notification 744 to the requesting client to indicate when the requested service is now ready for use. In some instances, information (e.g., a link) enabling the client to begin using and taking advantage of the benefits of the requested service may be sent. In some examples, for the client requesting the service, the response may include a chatbot system ID generated by cloud infrastructure system 702 and information identifying the chatbot system selected by cloud infrastructure system 702 corresponding to the chatbot system with the chatbot system ID.

[0189] Cloud infrastructure system 702 can provide services to multiple customers. For each customer, cloud infrastructure system 702 is responsible for managing information related to one or more subscription orders received from the customer, maintaining customer data related to the orders, and providing the requested services to the customer. Cloud infrastructure system 702 can also collect usage statistics about customers' use of subscribed services. For example, it can collect statistics such as storage usage, data transfer volume, number of users, system uptime, and system downtime. This usage information can be used to issue bills to customers. Billing can be done, for example, on a monthly basis.

[0190] Cloud infrastructure system 702 can provide services to multiple customers in parallel. Cloud infrastructure system 702 can store information about these customers (potentially including proprietary information). In some examples, cloud infrastructure system 702 includes an Identity Management Subsystem (IMS) 728 configured to manage customer information and provide separation of the managed information so that information related to one customer cannot be accessed by another customer. IMS 728 can be configured to provide various security-related services, such as identity services, information access management, authentication and authorization services, services for managing customer identities and roles, and related functions.

[0191] Figure 8 An example of a computer system 800 is illustrated. In some examples, computer system 800 can be used to implement any digital assistant or chatbot system within a distributed environment, as well as the various servers and computer systems described above. Figure 8 As shown, the computer system 800 includes various subsystems, including a processing subsystem 804 that communicates with multiple other subsystems via a bus subsystem 802. These other subsystems may include a processing acceleration unit 806, an I / O subsystem 808, a storage subsystem 818, and a communication subsystem 824. The storage subsystem 818 may include non-transitory computer-readable storage media, including storage medium 822 and system memory 810.

[0192] Bus subsystem 802 provides a mechanism for allowing the various components and subsystems of computer system 800 to communicate with each other as intended. While bus subsystem 802 is schematically shown as a single bus, alternative examples of bus subsystems may utilize multiple buses. Bus subsystem 802 can be any of several types of bus structures, including a memory bus or memory controller, a peripheral bus, a local bus using any of a variety of bus architectures, etc. For example, such architectures may include an Industry Standard Architecture (ISA) bus, a Micro Channel Architecture (MCA) bus, an Enhanced ISA (EISA) bus, a Video Electronics Standards Association (VESA) local bus, and a Peripheral Component Interconnect (PCI) bus (which may be implemented as a mezzanine bus manufactured to the IEEE P1386.1 standard), etc.

[0193] Processing subsystem 804 controls the operation of computer system 800 and may include one or more processors, application-specific integrated circuits (ASICs), or field-programmable gate arrays (FPGAs). Processors may include single-core or multi-core processors. The processing resources of computer system 800 may be organized into one or more processing units 832, 834, etc. A processing unit may include one or more processors, one or more cores from the same or different processors, a combination of cores and processors, or other combinations of cores and processors. In some examples, processing subsystem 804 may include one or more dedicated coprocessors such as graphics processors or digital signal processors (DSPs). In some examples, some or all of the processing units of processing subsystem 804 may be implemented using custom circuitry such as ASICs or FPGAs.

[0194] In some examples, the processing units in processing subsystem 804 can execute instructions stored in system memory 810 or on computer-readable storage medium 822. In various examples, the processing units can execute various program or code instructions and can maintain multiple concurrently executing programs or processes. At any given time, some or all of the program code to be executed can reside in system memory 810 and / or on computer-readable storage medium 822 (potentially including residing on one or more storage devices). With suitable programming, processing subsystem 804 can provide the various functions described above. In an instance where computer system 800 is executing one or more virtual machines, one or more processing units can be assigned to each virtual machine.

[0195] In some examples, a processing acceleration unit 806 may optionally be provided for performing custom processing or for offloading some of the processing performed by the processing subsystem 804, thereby accelerating the overall processing performed by the computer system 800.

[0196] I / O subsystem 808 may include devices and mechanisms for inputting information to and / or outputting information from or via computer system 800. Generally, the term input device is intended to include all possible types of devices and mechanisms for inputting information to computer system 800. User interface input devices may include, for example, keyboards, pointing devices such as mice or trackballs, touchpads or touchscreens integrated into displays, scroll wheels, click wheels, dial pads, buttons, switches, keypads, audio input devices with voice command recognition systems, microphones, and other types of input devices. User interface input devices may also include motion sensing and / or gesture recognition devices, such as Microsoft's [unclear - possibly a specific technology or feature], which enables users to control and interact with input devices. Motion sensor, Microsoft The 360 ​​game controller provides an interface for receiving input using gestures and spoken commands. The user interface input device may also include eye gesture recognition devices, such as detecting eye movements from the user (e.g., "blinking" when taking a photo and / or making menu selections) and translating the eye gestures into the input device (such as Google). Google input Blink detector. Additionally, the user interface input device may include enabling the user to interact with a voice recognition system (e.g., ...) via voice commands. A voice recognition sensing device for interaction with navigators.

[0197] Other examples of user interface input devices include, but are not limited to, 3D mice, joysticks or pointing sticks, game controllers and graphics tablets, and audio / visual devices such as speakers, digital cameras, digital camcorders, portable media players, webcams, image scanners, fingerprint scanners, barcode readers, 3D scanners, 3D printers, laser rangefinders, and eye-tracking devices. Additionally, user interface input devices may include, for example, medical imaging input devices such as computed tomography (CT), magnetic resonance imaging (MRI), positron emission tomography (PET), and medical ultrasound examination equipment. User interface input devices may also include, for example, audio input devices such as MIDI keyboards, digital musical instruments, etc.

[0198] Generally, the term "output device" is intended to encompass all possible types of devices and mechanisms for outputting information from a computer system 800 to a user or other computer. User interface output devices may include display subsystems, indicator lights, or non-visual displays such as audio output devices. Display subsystems may be cathode ray tube (CRT), flat panel devices (such as those using liquid crystal displays (LCDs) or plasma displays), projection devices, touchscreens, etc. For example, user interface output devices may include, but are not limited to, various display devices that visually convey text, graphics, and audio / video information, such as monitors, printers, speakers, headsets, car navigation systems, plotters, voice output devices, and modems.

[0199] Storage subsystem 818 provides a repository or data store for storing information and data used by computer system 800. Storage subsystem 818 provides a tangible, non-transitory, computer-readable storage medium for storing basic programming and data constructs that provide some example functionality. Storage subsystem 818 may store software (e.g., programs, code modules, instructions) that provides the functionality described above when executed by processing subsystem 804. The software may be executed by one or more processing units of processing subsystem 804. Storage subsystem 818 may also provide certification according to the teachings of this disclosure.

[0200] The storage subsystem 818 may include one or more non-transitory memory devices, which may include volatile memory devices and non-volatile memory devices. For example... Figure 8 As shown, the storage subsystem 818 includes system memory 810 and computer-readable storage medium 822. System memory 810 may include multiple memories, including volatile main random access memory (RAM) for storing instructions and data during program execution and non-volatile read-only memory (ROM) or flash memory where fixed instructions are stored. In some embodiments, a basic input / output system (BIOS) containing basic routines that help transfer information between elements within the computer system 800 during startup may typically be stored in ROM. RAM typically contains data and / or program modules currently operated and executed by processing subsystem 804. In some embodiments, system memory 810 may include various different types of memory such as static random access memory (SRAM), dynamic random access memory (DRAM), etc.

[0201] By using examples rather than restrictions, such as Figure 8As depicted, system memory 810 can load an executing application 812 (which may include various applications such as web browsers, middleware applications, relational database management systems (RDBMS), etc.), program data 814, and operating system 816. By way of example, operating system 816 may include various versions of Microsoft... Apple and / or Linux operating system, and various commercially available... Or a UNIX-like operating system (including but not limited to various GNU / Linux operating systems, Google...) OS, etc.) and / or such as iOS, Telephone, OS OS OS operating system and other mobile operating systems.

[0202] Computer-readable storage medium 822 can store programming and data structures that provide some example functionality. Computer-readable storage medium 822 can provide storage for computer-readable instructions, data structures, program modules, and other data for computer system 800. Software (programs, code modules, instructions) that provides the functions described above, when executed by processing subsystem 804, can be stored in storage subsystem 818. By way of example, computer-readable storage medium 822 may include, for example, hard disk drives, disk drives, optical disc drives (such as CD ROMs, DVDs, Blu-ray discs, etc.). Non-volatile memory such as disks or other optical media. Computer-readable storage medium 822 may include, but is not limited to, [various types of storage media]. Drives, flash memory cards, Universal Serial Bus (USB) flash memory drives, Secure Digital (SD) cards, DVD discs, digital videotapes, etc. Computer-readable storage media 822 may also include SSDs based on flash memory, such as flash memory-based solid-state drives (SSDs), enterprise-class flash memory drives, SSDs based on non-volatile memory such as solid-state ROM, SSDs based on volatile memory such as solid-state RAM, dynamic RAM, static RAM, DRAM-based SSDs, magnetoresistive RAM (MRAM) SSDs, and hybrid SSDs using a combination of DRAM and flash memory-based SSDs.

[0203] In some examples, the storage subsystem 818 may also include a computer-readable storage medium reader 820 that can be further connected to a computer-readable storage medium 822. The reader 820 may receive data from a storage device such as a disk, flash memory drive, etc., and is configured to read data from said storage device.

[0204] In some examples, computer system 800 may support virtualization technologies, including but not limited to the virtualization of processing and memory resources. For example, computer system 800 may provide support for executing one or more virtual machines. In some examples, computer system 800 may execute programs such as hypervisors that facilitate the configuration and management of virtual machines. Each virtual machine may be allocated memory, computing (e.g., processor, core), I / O, and networking resources. Each virtual machine typically runs independently of other virtual machines. Virtual machines typically run their own operating system, which may be the same as or different from the operating systems executed by other virtual machines executed by computer system 800. Therefore, multiple operating systems may potentially be run simultaneously by computer system 800.

[0205] The communication subsystem 824 provides interfaces to other computer systems and networks. The communication subsystem 824 serves as an interface for receiving data from other systems and transmitting data from computer system 800 to other systems. For example, the communication subsystem 824 can enable computer system 800 to establish communication channels to one or more client devices via the Internet for receiving and sending information to client devices. For example, when computer system 800 is used to implement... Figure 1 When the robot system 120 is described, the communication subsystem can be used to communicate with a chatbot system selected for the application.

[0206] The communication subsystem 824 may support both wired and / or wireless communication protocols. In some examples, the communication subsystem 824 may include radio frequency (RF) transceiver components (e.g., advanced data network technologies such as cellular telephone technology, 3G, 4G, or EDGE (Global Evolution Enhanced Data Rate), WiFi (IEEE 802.XX Home Standard, or other mobile communication technologies, or any combination thereof), GPS receiver components, and / or other components for accessing wireless voice and / or data networks. In some examples, in addition to or as an alternative to a wireless interface, the communication subsystem 824 may provide wired network connectivity (e.g., Ethernet).

[0207] The communication subsystem 824 can receive and transmit data in various forms. In some examples, among others, the communication subsystem 824 can also receive input communications in the form of structured and / or unstructured data feeds 826, event streams 828, event updates 830, etc. For example, the communication subsystem 824 can be configured to receive (or send) data feeds 826 in real time from users of social media networks and / or other communication services, such as... feed, Updates, web feeds (such as rich site summary (RSS) feeds) and / or real-time updates from one or more third-party information sources.

[0208] In some examples, the communication subsystem 824 may be configured to receive data in the form of a continuous data stream, which may include an event stream 828 and / or event updates 830 of real-time events (which may be inherently continuous or unbounded and may not have an explicit end). Examples of applications that generate continuous data may include, for example, sensor data applications, financial reporting machines, network performance measurement tools (e.g., network monitoring and traffic management applications), clickstream analysis tools, vehicle traffic monitoring, etc.

[0209] The communication subsystem 824 can also be configured to transmit data from computer system 800 to other computer systems or networks. Data can be transmitted in various forms, such as structured and / or unstructured data feeds 826, event streams 828, event updates 830, etc., to one or more databases that can communicate with one or more streaming data source computers coupled to computer system 800.

[0210] The computer system 800 can be of a variety of types, including handheld portable devices (e.g., Cellular phone Computing tablets, PDAs), wearable devices (e.g., Google...) Head-mounted displays, personal computers, workstations, mainframes, self-service kiosks, server racks, or any other data processing systems. Due to the constantly evolving nature of computers and networks, [the following is relevant:] Figure 8 The description of the computer system 800 is intended only as a concrete example. It has a higher... Figure 8 Many other configurations with more or fewer components are possible for the system depicted. Based on this disclosure and the teachings provided herein, it should be understood that there are other ways and / or methods to implement the various examples.

[0211] While specific examples have been described, various modifications, alterations, alternative constructions, and equivalents are possible. The examples are not limited to operations in a particular data processing environment but are free to operate in multiple data processing environments. Furthermore, although certain examples have been described using specific series of transactions and steps, it will be apparent to those skilled in the art that this is not intended to be restrictive. While some flowcharts describe operations as sequential processes, many operations can be performed in parallel or simultaneously. Additionally, the order of operations can be rearranged. Processes may have additional steps not included in the diagrams. Various features and aspects of the examples described above can be used individually or in combination.

[0212] Furthermore, while certain examples have been described using specific combinations of hardware and software, it should be recognized that other combinations of hardware and software are also possible. Some examples may be implemented using only hardware, only software, or a combination thereof. The various processes described herein can be implemented on the same or different processors in any combination.

[0213] When a device, system, component, or module is described as being configured to perform certain operations or functions, this configuration can be accomplished, for example, by designing electronic circuitry to perform operations, by programming programmable electronic circuitry (such as a microprocessor) to perform operations (such as by executing computer instructions or code), or by a processor or core programmed to execute code or instructions stored on a non-transitory memory medium, or any combination thereof. Processes can communicate using a variety of techniques, including but not limited to conventional techniques for inter-process communication, and different pairs of processes may use different techniques, or the same pair of processes may use different techniques at different times.

[0214] Specific details are set forth in this disclosure to provide a thorough understanding of the examples. However, the examples can be practiced without these specific details. For example, well-known circuits, processes, algorithms, structures, and techniques have been shown without unnecessary details to avoid obscuring the examples. This description provides only examples and is not intended to limit the scope, applicability, or configuration of other examples. Rather, the foregoing description of the examples will provide those skilled in the art with enabling descriptions for implementing the various examples. Various changes can be made to the function and arrangement of the elements.

[0215] Therefore, this specification and drawings should be viewed in an illustrative rather than restrictive sense. However, it will be apparent that additions, omissions, deletions, and other modifications and changes may be made without departing from the broader spirit and scope set forth in the claims. Thus, while specific examples have been described, they are not intended to be restrictive. Various modifications and equivalents are within the scope of the following claims.

[0216] In the foregoing description, various aspects of this disclosure have been described with reference to specific examples; however, those skilled in the art will recognize that this disclosure is not limited thereto. Various features and aspects of the disclosure described above may be used individually or in combination. Furthermore, the examples may be utilized in any number of environments and applications other than those described herein without departing from the broader spirit and scope of the specification. Therefore, the specification and drawings should be considered illustrative rather than restrictive.

[0217] In the foregoing description, the methods have been described in a specific order for illustrative purposes. It should be understood that, in alternative examples, the methods may be performed in a different order than described. It should also be understood that the methods described above may be executed by hardware components or may be embodied in a sequence of machine-executable instructions that can be used to cause a machine (such as a general-purpose or special-purpose processor or logic circuit programmed with the instructions) to execute the methods. These machine-executable instructions may be stored on one or more machine-readable media, such as CD-ROMs or other types of optical discs, floppy disks, ROMs, RAMs, EPROMs, EEPROMs, magnetic or optical cards, flash memory, or other types of machine-readable media suitable for storing electronic instructions. Alternatively, the methods may be executed by a combination of hardware and software.

[0218] When a component is described as being configured to perform certain operations, such configuration can be accomplished, for example, by designing electronic circuitry or other hardware for performing the operations, by programming programmable electronic circuitry (e.g., a microprocessor or other suitable electronic circuitry) for performing the operations, or any combination thereof.

[0219] Although illustrative examples of this application have been described in detail herein, it should be understood that the inventive concepts may be embodied and employed in other ways, and the appended claims are intended to be construed as including such variations, except where limited by the prior art.

Claims

1. A method comprising: receiving, at a data processing system, a set of utterances for training an intent classifier to identify one or more intents of one or more utterances; augmenting, by the data processing system, the set of utterances with stop words to generate an augmented out-of-domain set of utterances for an unresolvable intent category corresponding to an unresolvable intent, wherein the augmenting comprises: selecting one or more utterances from the set of utterances, and for each selected utterance, preserving existing stop words within the utterance and replacing at least one non-stop word within the utterance with a stop word or stop word phrase selected from a list of stop words to generate an out-of-domain utterance; and training, by the data processing system, the intent classifier using the set of utterances and the augmented out-of-domain set of utterances.

2. The method of claim 1, wherein, for each selected utterance, preserving existing stop words within the utterance and replacing all non-stop words within the utterance with a stop word or stop word phrase selected from the list of stop words.

3. The method of claim 1, wherein, for each selected utterance, preserving existing stop words within the utterance and replacing at least one of the non-stop words within the utterance randomly with a stop word or stop word phrase selected from the list of stop words.

4. The method of claim 1, wherein, selecting the utterances comprises searching for consecutive stop words at the beginning of the utterances within the set of utterances and selecting utterances having the longest consecutive stop words at the beginning of the utterances within the set of utterances.

5. The method of claim 1, wherein, selecting the utterance includes searching for any position in the utterance within the training set of utterances that has n one consecutive stop word and selecting the utterance that has any position in the utterance that has n one consecutive stop word.

6. The method of claim 1, wherein, augmenting the set of utterances with the stop words further comprises (i) iteratively selecting the one or more utterances from the set of utterances and processing each utterance a single time to generate a corresponding single out-of-domain utterance based on a predefined augmentation ratio, (ii) selecting the one or more utterances from the set of utterances and processing the one or more utterances multiple times to generate multiple out-of-domain utterances from each utterance based on the predefined augmentation ratio, or (iii) any combination thereof.

7. The method of claim 6, wherein, the predefined augmentation ratio of original utterances to augmented utterances is between 1 :0.05 and 1 :

1.

8. A system comprising: one or more data processors; and a non-transitory computer-readable storage medium containing instructions that, when executed on the one or more data processors, cause the one or more data processors to perform actions comprising: receiving a set of utterances for training an intent classifier to identify one or more intents of one or more utterances; augmenting the set of utterances with stop words to generate an augmented out-of-domain set of utterances for an unresolvable intent category corresponding to an unresolvable intent, wherein the augmenting comprises: selecting one or more utterances from the set of utterances, and for each selected utterance, preserving existing stop words within the utterance and replacing at least one non-stop word within the utterance with a stop word or stop word phrase selected from a list of stop words to generate an out-of-domain utterance; and training the intent classifier using the set of utterances and the augmented out-of-domain set of utterances. for each selected utterance, preserving existing stop words within the utterance and replacing all non-stop words within the utterance with a stop word or stop word phrase selected from the list of stop words. for each selected utterance, preserving existing stop words within the utterance and replacing at least one of the non-stop words within the utterance randomly with a stop word or stop word phrase selected from the list of stop words. selecting the utterances comprises searching for consecutive stop words at the beginning of the utterances within the set of utterances and selecting utterances having the longest consecutive stop words at the beginning of the utterances within the set of utterances. augmenting the set of utterances with the stop words further comprises (i) iteratively selecting the one or more utterances from the set of utterances and processing each utterance a single time to generate a corresponding single out-of-domain utterance based on a predefined augmentation ratio, (ii) selecting the one or more utterances from the set of utterances and processing the one or more utterances multiple times to generate multiple out-of-domain utterances from each utterance based on the predefined augmentation ratio, or (iii) any combination thereof. the predefined augmentation ratio of original utterances to augmented utterances is between 1 :0.05 and 1 :

1.

9. The system of claim 8, wherein, For each selected utterance, retaining existing stop words within the utterance and replacing all non-stop words within the utterance with a stop word or stop word phrase selected from the list of stop words.

10. The system of claim 8, wherein, For each selected utterance, retaining existing stop words within the utterance and replacing at least one of the non-stop words within the utterance with a stop word or stop word phrase selected from the list of stop words.

11. The system of claim 8, wherein, Selecting the utterances includes searching for consecutive stop words at the beginning of the utterances within the training set of utterances and selecting utterances having the longest consecutive stop words at the beginning of the utterances within the training set of utterances.

12. The system of claim 8, wherein, selecting the utterance includes searching for any position in the utterance within the training set of utterances that has n one consecutive stop word and selecting the utterance that has any position in the utterance that has n one consecutive stop word.

13. The system of claim 8, wherein, Augmenting the training set of utterances with the stop words further includes (i) iteratively selecting the one or more utterances from the training set of utterances and processing each utterance a single time to generate a corresponding single out-of-domain utterance based on a predefined augmentation ratio, (ii) selecting the one or more utterances from the training set of utterances and processing the one or more utterances multiple times to generate multiple out-of-domain utterances from each utterance based on the predefined augmentation ratio, or (iii) any combination thereof.

14. The system of claim 13, wherein, The predefined augmentation ratio of original utterances to augmented utterances is between 1 :0.05 and 1 :

1.

15. A computer program product tangibly embodied in a non-transitory machine- readable storage medium, including instructions configured to cause one or more data processors to perform actions including: receiving a training set of utterances for training an intent classifier to identify one or more intents of one or more utterances; augmenting the training set of utterances with stop words to generate an augmented training set of out-of-domain utterances for an unresolvable intent category corresponding to an unresolvable intent, wherein, the augmenting includes: selecting one or more utterances from the training set of utterances, and for each selected utterance, retaining existing stop words within the utterance and replacing at least one non-stop word within the utterance with a stop word or stop word phrase selected from a list of stop words to generate an out-of-domain utterance; and training the intent classifier using the training set of utterances and the augmented training set of out-of-domain utterances.

16. The computer program product of claim 15, wherein, For each selected utterance, retaining existing stop words within the utterance and replacing all non-stop words within the utterance with a stop word or stop word phrase selected from the list of stop words.

17. The computer program product of claim 15, wherein, For each selected utterance, retaining existing stop words within the utterance and replacing at least one of the non-stop words within the utterance with a stop word or stop word phrase selected from the list of stop words.

18. The computer program product of claim 15, wherein, Selecting the utterances includes searching for consecutive stop words at the beginning of the utterances within the training set of utterances and selecting utterances having the longest consecutive stop words at the beginning of the utterances within the training set of utterances.

19. The computer program product of claim 15, wherein, selecting the utterance includes searching for any position in the utterance within the training set of utterances that has n one consecutive stop word and selecting the utterance that has any position in the utterance that has n one consecutive stop word.

20. The computer program product of claim 15, wherein, Augmenting the utterance training set with the stop words further comprises: (i) iteratively selecting the one or more utterances from the utterance training set and processing each utterance a single time to generate a corresponding single out-of-domain utterance based on a predefined augmentation ratio, (ii) selecting the one or more utterances from the utterance training set and processing the one or more utterances multiple times to generate multiple out-of-domain utterances from each utterance based on the predefined augmentation ratio, or (iii) any combination thereof.

21. A method for determining a resolved intent or an unresolved intent from an utterance, the method comprising: receiving, by a chatbot system, an utterance generated by a user interacting with the chatbot system; classifying, using an intent classifier deployed within the chatbot system, the utterance as a resolved intent category corresponding to a resolved intent or an unresolved intent category corresponding to an unresolved intent, wherein the intent classifier comprises a plurality of model parameters identified using training data, the training data comprising: an utterance training set for training the intent classifier to identify one or more resolved intents of one or more utterances; and an augmented out-of-domain utterance training set for training the intent classifier to identify one or more unresolved intents of one or more utterances, wherein the augmented out-of-domain utterance training set is artificially generated to include utterances from the utterance training set, wherein existing stop word patterns within the utterances are preserved and at least one non-stop word in each utterance is randomly replaced with a stop word, and wherein the plurality of model parameters are identified using training data based on minimizing a loss function; and outputting, using the intent classifier, the resolved intent or the unresolved intent based on the classification.

Images