Improved techniques for out-of-domain (OOD) detection

Abstract

To provide a method, system and program for identifying out-of-domain utterances.SOLUTION: A method includes: receiving an utterance and a target domain of a chatbot; generating a sentence embedding for the utterance; obtaining an embedding representation for each cluster of in-domain utterances associated with the target domain; predicting, using a metric learning model, a first probability that the utterance belongs to the target domain, based on a similarity or difference between the sentence embedding and each embedding representation for each cluster; predicting, using an outlier detection model, a second probability that the utterance belongs to the target domain, based on a determined distance or density deviation between the sentence embedding and embedding representations for neighboring clusters; and classifying the utterance as in-domain or out-of-domain based on a final probability determined by evaluating the first probability and the second probability.SELECTED DRAWING: Figure 5

Description

Technical Field

[0001] Priority Claim This application is the present application of U.S. Provisional Application No. 63 / 002,139, filed on March 30, 2021, and claims the benefit and priority thereof. The entire content of the above application is incorporated herein by reference for all purposes.

[0002] Field of the Invention The present disclosure generally relates to chatbot systems, and more particularly to improved techniques for identifying out-of-domain (OOD) utterances.

Background Art

[0003] Background Many users around the world are on instant messaging or chat platforms to get an immediate response. Organizations often use these instant messaging or chat platforms to communicate live with customers (or end users). However, having an organization use service representatives to communicate live with customers or end users can be very costly. In particular, chatbots or bots have begun to be developed to simulate conversations with end users over the Internet. End users can communicate with the bot via a messaging app that the end user has already installed and uses. Intelligent bots, generally realized by artificial intelligence (AI), are smarter and can communicate live based on context, enabling a more natural conversation between the bot and the end user and improving the conversation experience. Instead of the end user learning a fixed set of keywords or commands that the bot knows how to respond to, intelligent bots can understand the end user's intent based on the end user's utterance in natural language and respond accordingly.

[0004] However, chatbots are difficult to build because these automated solutions require specialized knowledge and the application of specific techniques in certain fields, which may be within the scope of the expertise of professional developers. As part of building such a chatbot, developers can first understand the needs of the company and the end users. Then, developers can, for example, select a dataset to be used for analysis, process this input dataset in preparation for analysis (e.g., clean the data, extract, format and / or transform the data before analysis, perform data feature engineering, etc.), identify an appropriate machine learning (ML) technique or model to perform the analysis, and perform analysis and decisions related to refining this technique or model and improving the results / outcomes based on feedback. The task of identifying an appropriate model may involve developing multiple models, possibly in parallel, and iteratively testing and experimenting with these models before identifying a specific model (or multiple models) to be used. Furthermore, supervised learning-based solutions generally include a training phase, a subsequent application (i.e., inference) phase, and an iterative loop between the training and application phases. Developers will be responsible for carefully executing and monitoring these phases to achieve the optimal solution. For example, to train an ML technique or model, accurate training data is needed to enable algorithms that understand and learn specific patterns or features that the ML technique or model uses to predict desired outcomes (e.g., inferring intent from utterances) (e.g., in a chatbot, intent extraction and careful parsing, rather than just raw language processing). [Overview of the project] [Means for solving the problem]

[0005] Brief Overview The technologies disclosed herein generally relate to chatbots. More specifically, the technologies disclosed herein relate to, but are not limited to, improved technologies for identifying OOD utterances. A chatbot (also referred to as a bot) includes an OOD detector that uses one or more algorithms to determine whether an utterance provided to the bot is outside the domain of the bot (e.g., a skills bot). When such an OOD utterance is detected, the bot can respond with an appropriate response, such as a message, that allows the user to identify that the utterance is not something the bot can process or address. In certain embodiments, technologies using various clustering-based algorithms and metrics-based algorithms, as well as combinations thereof, are used for OOD detection.

[0006] In various embodiments, a method is provided, the method comprising the steps of receiving an utterance and a target domain of a chatbot; generating sentence embeddings for the utterance; and obtaining embedding representations for each cluster of a plurality of clusters of in-domain utterances associated with the target domain, wherein the embedding representation for each cluster is the average of the sentence embeddings for each in-domain utterance in the cluster; the method further comprises the step of inputting the sentence embeddings for the utterance and the embedding representations for each cluster into a distance learning model, the distance learning model having trained model parameters configured to provide a first probability of whether the utterance belongs to the target domain; the method further comprises the step of using the distance learning model to determine the similarity or difference between the sentence embeddings for the utterance and the embedding representations for each cluster; and using the distance learning model to determine the similarity or difference between the sentence embeddings for the utterance and the embedding representations for each cluster The method comprises the steps of: predicting a first probability that the utterance belongs to the target domain based on the determined similarity or difference between the utterance and the target domain; inputting the sentence embedding for the utterance and the embedding representation for each cluster into an outlier detection model, wherein the outlier detection model is constructed with a distance or density algorithm for outlier detection; and further comprising the steps of: using the outlier detection model to determine the distance or density deviation between the sentence embedding for the utterance and the embedding representation for adjacent clusters; using the outlier detection model to predict a second probability that the utterance belongs to the target domain based on the determined distance or density deviation; evaluating the first and second probabilities to determine a final probability that the utterance belongs to the target domain; and classifying the utterance as in-domain or out-of-domain for the chatbot based on the final probability.

[0007] In some embodiments, the step of obtaining the embedding representation for each cluster comprises the steps of: obtaining an intradomain utterance based on the target domain; generating a sentence embedding for each intradomain utterance; and inputting the sentence embedding for each intradomain utterance into an unsupervised clustering model, wherein the unsupervised clustering model is configured to interpret the intradomain utterance to identify the plurality of clusters in the feature space of the intradomain utterance; and the step of obtaining the embedding representation for each cluster further comprises the steps of using the unsupervised clustering model to classify the sentence embedding for each intradomain utterance into one of the plurality of clusters based on the similarities and differences between the features of the sentence embedding and the features of the sentence embedding within each cluster; and calculating the centroid for each of the plurality of clusters. The method comprises the steps of: 1. Outputting the above-mentioned embedding representation and the above-mentioned centroid for each cluster of the plurality of clusters.

[0008] In some embodiments, the method further comprises the steps of: calculating a z-score for an utterance based on the distance or density deviation between the sentence embedding for the utterance and the embedding representation for the adjacent cluster; and determining a second probability of whether the utterance belongs to the target domain by applying a sigmoid function to the z-score.

[0009] In some embodiments, the sentence embeddings for the above utterances are generated using an embedding model that maps natural language elements, including sentences, words, and n-grams, to sequences of numbers, each of which is represented as a single point in a vector space.

[0010] In some embodiments, the step of determining the similarity or difference between the sentence embedding for the utterance and the embedding representation for each cluster comprises (i) calculating the absolute difference between the sentence embedding for the utterance and the embedding representation for each cluster, and (ii) inputting the absolute difference, the sentence embedding for the utterance and the embedding representation for each cluster into a wide and deep learning network, wherein the wide and deep learning network comprises a linear model and a deep neural network, and the step of determining the similarity or difference between the sentence embedding for the utterance and the embedding representation for each cluster further, (iii) Using the linear model and the absolute difference, predict a wide-based probability of whether the utterance belongs to the target domain; (iv) Using the deep neural network, the sentence embedding for the utterance, and the embedding representation for each cluster, determine the similarity or difference between the sentence embedding for the utterance and the embedding representation for each cluster, wherein the step of predicting the first probability comprises evaluating the wide probability and the similarity or difference between the sentence embedding for the utterance and the embedding representation for each cluster using the final layer of the wide and deep learning network.

[0011] In some embodiments, the linear model comprises a plurality of model parameters trained using a set of training data, the set of training data includes the absolute difference between sentence embeddings for utterances and each embedding representation for each cluster for intradomain utterances from multiple domains, and during the training of the linear model with the set of training data, a hypothesis function is used to learn a linear relationship between the sentence embeddings for utterances and each embedding representation for each cluster, and during the learning of the linear relationship, the plurality of model parameters are learned to minimize a loss function.

[0012] In some embodiments, the deep learning network comprises a set of model parameters trained using a set of training data, the set of training data includes sentence embeddings for intradomain utterances from multiple domains, and during training of the deep learning network using the set of training data, the high-dimensional features of the sentence embeddings for the intradomain utterances are converted into low-dimensional vectors, the low-dimensional vectors are then concatenated with features from the intradomain utterances and fed into the hidden layers of the deep neural network, the values ​​of the low-dimensional vectors are randomly initialized and learned together with the multiple model parameters to minimize a loss function.

[0013] In various embodiments, a computer program product is provided, the computer program product is tangibly embodied in a non-temporary machine-readable storage medium, 1 Includes instructions configured to cause one or more data processors to perform some or all of the methods disclosed herein.

[0014] In various embodiments, a system is provided which includes one or more data processors and a non-temporary computer-readable storage medium, wherein the non-temporary computer-readable storage medium includes instructions that, when executed on the one or more data processors, cause the one or more data processors to execute some or all of the methods disclosed herein.

[0015] The technologies described above and below can be implemented in various forms and contexts. Several exemplary implementations and contexts are provided with reference to the following drawings, as will be explained in more detail below. However, the following implementations and contexts represent only a small fraction of the many available. [Brief explanation of the drawing]

[0016] [Figure 1]This is a simplified block diagram of a distributed environment incorporating an exemplary embodiment. [Figure 2] This is a simplified block diagram of a computing system that implements a master bot according to a specific embodiment. [Figure 3] This is a simplified block diagram of a computing system that implements a skill bot according to a specific embodiment. [Figure 4] This is a simplified block diagram of a chatbot training and deployment system according to various embodiments. [Figure 5] This figure shows an ensemble architecture comprising a distance learning model and an outlier detection model for identifying OOD utterances, according to various embodiments. [Figure 6] This figure shows a process flow for identifying OOD utterances according to various embodiments. [Figure 7] This is a simplified diagram of a distributed system for realizing various embodiments. [Figure 8] This is a simplified block diagram of one or more components of a system environment in which services provided by one or more components of an embodiment of the system may be provided as cloud services, according to various embodiments. [Figure 9] This figure shows an exemplary computer system that can be used to realize various embodiments. [Modes for carrying out the invention]

[0017] Detailed explanation In the following description, specific details are set forth for the purpose of providing a thorough understanding of particular embodiments. It will be apparent, however, that the various embodiments may be practiced without these specific details. The drawings and description are not intended to be restrictive. The word "exemplary" as used herein means "serving as an example, instance, or illustration." Any embodiment or design described herein as "exemplary" is not necessarily to be construed as preferred or advantageous over other embodiments or designs.

[0018] Introduction A digital assistant is an AI-driven interface that helps users accomplish various tasks in natural language conversations. For each digital assistant, a customer can assemble one or more skills. Skills (also described herein as chatbots, bots, or skillbots) are individual bots specialized for specific types of tasks such as inventory tracking, time card submission, and expense report creation. When an end user engages with a digital assistant, the digital assistant evaluates the end user input and routes the conversation to and from appropriate chatbots. The digital assistant can be made available to end users via various channels such as Facebook (registered trademark) Messenger, Skype Mobile (registered trademark) Messenger, or Short Message Service (SMS). The channels enable chat to flow to and from the end user and the digital assistant and its various chatbots on various messaging platforms. These channels can also support user agent escalation, event-triggered conversations, and testing.

[0019] An intent enables a chatbot to understand what the user wants it to do. An intent is composed of permutations of typical user requests and utterances (e.g., get account balance, purchase, etc.), also referred to as utterances. As used herein, an utterance or message is a set of words (e.g., one or more sentences) exchanged during a conversation with a chatbot. An intent can be created by providing a name indicating some user action (e.g., order pizza) and compiling a set of real-world user utterances or utterances that are normally associated with causing that action. Since the chatbot's understanding is derived from these intents, each intent can be created from a robust (1 - 2 dozen utterances) and diverse dataset so that the chatbot can interpret ambiguous user input. A rich set of utterances enables the chatbot to understand what the user wants when it receives messages like "Please don't forget this order!" or "Please cancel the delivery!", i.e., messages that mean the same thing but are expressed differently. Intents and the utterances belonging to them collectively constitute a training corpus for the chatbot. By training a model using this corpus, the customer can essentially transform the model into a reference tool for decomposing end-user input into a single intent. The customer can improve the chatbot's understanding ability through multiple intent tests and intent trainings.

[0020] However, building a chatbot that can determine an end-user's intent based on user utterances is a challenging task, partly due to the nuances and ambiguities of natural language, the dimensions of the input space (e.g., possible user utterances), and the size of the output space (number of intents). Exemplary examples of this difficulty arise from the characteristics of natural language, such as the use of euphemisms, synonyms, or ungrammatical speech to express intent. For example, an utterance might express the intent to order a pizza without explicitly mentioning pizza, ordering, or delivery. For instance, in a particular local language, "pizza" might be referred to as "pie." These tendencies of natural language, such as inaccuracy or variability, introduce uncertainty, and reliability as a parameter for predicting intent, rather than an explicit representation of intent, is introduced, for example, by including keywords. Therefore, to improve chatbot performance and the user experience with the chatbot, it will be necessary to train, monitor, debug, and retrain the chatbot. Traditional systems include training systems for training and retraining machine learning models for digital assistants or chatbots using Speech and Language Understanding (SLU) and Natural Language Processing (NLP). Traditionally, models used in chatbot systems are trained in NLP using "fabricated" utterances for any intent. For example, the utterance "Shall I change the price?" can be used to train the chatbot system's classifier model to classify this type of utterance as the intent "Shall I offer a price match?" Training the model with fabricated utterances helps to initially train the chatbot system to provide a service, and then the chatbot system is deployed to users... They can be retrained once they begin to acquire actual speech.

[0021] Traditional training of text classification models begins with training a dataset of utterances labeled with a predefined list of intents (or categories, or classes). For example, a banking chatbot might be trained using predefined intents such as "open an account," "inquire about balance," "close an account," and "send money." These intents are generally considered to belong to the same domain that the chatbot can handle (e.g., the banking domain). Typically, chatbot training is done using training data that includes multiple examples of utterances and, for each training utterance, the intent associated with that utterance. Once training is complete, the chatbot can receive new utterances (e.g., in the real world or a production environment) and, for each utterance, infer the intent from the predefined intents.

[0022] However, the utterances that chatbots receive from actual users in real-world environments (e.g., production environments) can be quite diverse and noisy. Some of these received utterances may be very different from the utterances used to train the chatbot and may fall outside the scope of intents that the chatbot is trained to reason and address. For example, a banking chatbot may receive utterances unrelated to banking, such as "How do I book a trip to Italy?" Such utterances are called out-of-domain (OOD) utterances because they are outside the domain of intents for the trained chatbot. It is important that the chatbot system can identify such OOD utterances so that it can take an appropriate response action. For example, when a chatbot detects an OOD utterance, it can respond to the user in a way that indicates the utterance is not something the bot can process or address, rather than selecting the nearest matching intent.

[0023] Therefore, different approaches are needed to address these problems. This disclosure describes various embodiments for addressing these problems by identifying out-of-domain utterances. In various embodiments, a combination of clustering techniques and metrics-based techniques is used for OOD decisions. One exemplary technique includes the steps of: receiving an utterance and a target domain for a chatbot; generating a sentence embedding for the utterance; obtaining an embedding representation for each cluster of in-domain utterances associated with the target domain; using a distance learning model to predict a first probability that the utterance belongs to the target domain based on the similarity or difference between the sentence embedding and each embedding representation for each cluster; using an outlier detection model to predict a second probability that the utterance belongs to the target domain based on the obtained distance or density deviation between the sentence embedding and the embedding representation for adjacent clusters; evaluating the first and second probabilities to determine a final probability; and classifying the utterance as in-domain or out-of-domain for the chatbot based on the final probability.

[0024] In a particular embodiment, a method is provided which comprises the steps of: receiving an utterance and a target domain of a chatbot; generating sentence embeddings for the utterance; and obtaining embedding representations for each cluster of a plurality of clusters of in-domain utterances associated with the target domain, wherein the embedding representation for each cluster is the average of the sentence embeddings for each in-domain utterance in the cluster; and the method further comprises the steps of inputting the sentence embeddings for the utterance and the embedding representations for each cluster into a distance learning model, wherein the distance learning model provides a first probability of whether or not the utterance belongs to the target domain. The method further comprises the steps of: having a trained model parameter configured as follows; using the distance learning model to determine the similarity or difference between the sentence embedding for the utterance and each embedding representation for each cluster; using the distance learning model to predict the first probability of whether the utterance belongs to the target domain based on the determined similarity or difference between the sentence embedding for the utterance and each embedding representation for each cluster; and inputting the sentence embedding for the utterance and the embedding representation for each cluster into an outlier detection model, wherein the outlier detection model uses distance or density algorithms for outlier detection. The method is constructed using Gorhythm, and further comprises the steps of: using the outlier detection model to determine the distance or density deviation between the sentence embedding for the utterance and the embedding representation for adjacent clusters; using the outlier detection model to predict the second probability of whether the utterance belongs to the target domain based on the determined distance or density deviation; evaluating the first and second probabilities to determine the final probability of whether the utterance belongs to the target domain; and classifying the utterance as either in-domain or out-of-domain for the chatbot based on the final probability.

[0025] Bots and analytics systems A bot (also known as a skill, chatbot, chatterbot, or talkbot) is a computer program that can converse with end users. Bots can generally respond to natural language messages (e.g., questions or comments) through messaging applications that use natural language messaging. Businesses can use one or more bot systems to communicate with end users through messaging applications. The messaging application, which may be called a channel, can be a messaging application of the end user's choice that the end user has already installed and is familiar with. Therefore, end users do not need to download and install a new application to chat with the bot system. Messaging applications may include, for example, over-the-top (OTT) messaging channels (such as Facebook Messenger, Facebook WhatsApp, WeChat, Line, Kick, Telegram, Talk, Skype, Slack, or SMS), virtual private assistants (such as Amazon.com, Echo or Show, Google Home, or Apple HomePod), mobile and web app extensions that extend native or hybrid / responsive mobile or web applications with chat capabilities, or voice-based input (such as devices or apps with interfaces that use Siri, Cortana, Google Voice, or other voice input for interaction).

[0026] In some examples, a bot system may be associated with a Uniform Resource Identifier (URI). A URI may use a string to identify the bot system. A URI may be used as a webhook for one or more messaging application systems. A URI may include, for example, a Uniform Resource Locator (URL) or Uniform Resource Name (URN). A bot system may be designed to receive messages (e.g., Hypertext Transfer Protocol (HTTP) postcall messages) from a messaging application system. This HTTP postcall message may be directed to a URI from the messaging application system. In some embodiments, the message may be different from an HTTP postcall message. For example, a bot system may receive messages from the Short Message Service (SMS). In this description, the communication received by a bot system is referred to as a message, but a message may be an HTTP postcall message, an SMS message, or any other type of communication between two systems. It should be understood that this is also acceptable.

[0027] End users may interact with bot systems through conversational dialogue (sometimes referred to as a conversational user interface (UI)), similar to human-to-human interactions. In some cases, this dialogue may involve the end user saying "hello" to the bot, the bot responding "hi," and asking the end user what they need. In other cases, this dialogue may be transaction-related with a banking bot, such as transferring money from one account to another; information-related with an HR bot, such as checking vacation balances; or dialogue with a retail bot, such as discussing returning a purchased item or requesting technical support.

[0028] In some embodiments, a bot system can intelligently handle end-user interactions without interaction with the bot system's administrator or developer. For example, an end-user may send one or more messages to the bot system to achieve a desired goal. Messages may include specific content such as text, emojis, voice, images, video, or other means of conveying a message. In some embodiments, the bot system may generate a natural language response by converting this content into a standardized format (e.g., a Representative State Transfer (REST) ​​call to an enterprise service with appropriate parameters). The bot system may also request additional input parameters or other additional information from the end-user. Furthermore, in some embodiments, the bot system may initiate communication with the end-user rather than passively responding to the end-user utterance. Various techniques for identifying explicit calls to a bot system and determining the input to the bot system being called are described herein. In certain embodiments, parsing of explicit calls is performed by a master bot based on detecting a call name in the utterance. In response to the detection of a call name, the utterance can be refined for input to the skill bot associated with the call name.

[0029] A conversation with a bot may follow a specific conversational flow that includes multiple states. This flow may define what happens next based on the input. In some embodiments, a bot system may be implemented using a state machine that includes user-defined states (e.g., end-user intents) and actions to be taken in or for each state. The conversation may take various paths based on end-user input, which may influence the decisions the bot makes regarding the flow. For example, in each state, based on end-user input or utterance, the bot may determine the end-user intent to determine the next appropriate action to take. In the context of an utterance, the term “intent” as used herein refers to the intent of the user who provided the utterance. For example, a user may intend to converse with a bot to order a pizza, and therefore the user’s intent may be expressed by the utterance “Please order a pizza.” User intent can be directed to a specific task that the user wants the chatbot to perform on their behalf. Thus, an utterance can be expressed as a question, command, request, etc., that reflects the user’s intent. Intent may include goals that the end-user wants to achieve.

[0030] In the context of chatbot configuration, the term “intent” is used herein to refer to configuration information for mapping user utterances to specific tasks / actions or categories of tasks / actions that the chatbot can perform. To distinguish between utterance intents (i.e., user intents) and chatbot intents, the latter may also be referred to herein as “bot intents.” A bot intent may consist of one or more sets of utterances associated with the intent. For example, the intent to order a pizza may have various reorderings of utterances that express the desire to order a pizza. These associated utterances can be used to train the intent classifier so that it can later determine whether an input utterance from the user matches the pizza order intent. A bot intent may be associated with one or more dialogue flows to initiate a conversation with the user in a particular state. For example, the first message for the pizza order intent might be the question, "What kind of pizza do you prefer?" In addition to associated utterances, a bot intent may further comprise named entities associated with the intent. For example, the pizza order intent may include variables or parameters used to perform the task of ordering a pizza, such as topping 1, topping 2, pizza type, pizza size, pizza quantity, etc. The values ​​of entities are generally obtained through conversation with the user.

[0031] Figure 1 is a simplified block diagram of an environment 100 incorporating a chatbot system according to a particular embodiment. Environment 100 comprises a Digital Assistant Builder Platform (DABP) 102, which enables users of DABP 102 to create and deploy digital assistants or chatbot systems. DABP 102 can be used to create one or more digital assistants (or DAs) or chatbot systems. For example, as shown in Figure 1, a user 104 representing a particular company can use DABP 102 to create and deploy a digital assistant 106 for users of that particular company. For example, DABP 102 may be used by a bank to create one or more digital assistants for use by its customers. The same DABP 102 platform may be used by multiple companies to create digital assistants. As another example, the owner of a restaurant (e.g., a pizza shop) may use DABP 102 to create and deploy a digital assistant that enables customers of the restaurant to order food (e.g., order a pizza).

[0032] For the purposes of this disclosure, “digital assistant” is an entity that helps users of the digital assistant perform various tasks through natural language conversation. A digital assistant may be implemented using software alone (for example, a digital assistant is a digital entity implemented using a program, code, or instructions executable by one or more processors), using hardware, or using a combination of hardware and software. A digital assistant can be embodied or implemented in various physical systems or devices such as computers, mobile phones, watches, appliances, and vehicles. A digital assistant is sometimes referred to as a chatbot system. Therefore, for the purposes of this disclosure, the terms “digital assistant” and “chatbot system” are interchangeable.

[0033] A digital assistant, such as a digital assistant 106 built using DABP102, can be used to perform various tasks through natural language-based conversations between the digital assistant and its user 108. As part of the conversation, the user may provide one or more user inputs 110 to the digital assistant 106 and receive responses 112 from the digital assistant 106. The conversation may include one or more of the inputs 110 and responses 112. Through these conversations, the user requests one or more tasks to be performed by the digital assistant, and in response, the digital assistant is configured to perform the user-requested task and respond to the user with an appropriate response.

[0034] User input 110 is generally in natural language form and is referred to as utterances. User utterances 110 may also be in text form, for example, when the user types a sentence, question, text fragment, or single word and provides it as input to the digital assistant 106. In some embodiments, user utterances 110 may also be in voice input or speech form, for example, when the user says or speaks something and provides it as input to the digital assistant 106. Utterances are generally the language spoken by the user 108. For example, utterances may be in English or another language. If the utterance is in speech form, the speech input is converted into text utterances in that particular language, and these text utterances are then processed by the digital assistant 106. Various speech-to-text processing techniques may be used to convert speech or voice input into text utterances, and then process these text utterances by the digital assistant 106. In some embodiments, speech-to-text conversion may be performed by the digital assistant 106 itself.

[0035] Utterances, which may be text utterances or speech utterances, may include fragments, sentences, multiple sentences, one or more words, one or more questions, or combinations of the above types. The digital assistant 106 is configured to apply natural language understanding (NLU) techniques to utterances to understand the meaning of user input. As part of the NLU processing of an utterance, the digital assistant 106 is configured to perform processing to understand the meaning of the utterance, which includes identifying one or more intents and one or more entities corresponding to the utterance. When understanding the meaning of an utterance, the digital assistant 106 may perform one or more actions or behaviors in response to the understood meaning or intent. For the purposes of this disclosure, it is assumed that utterances are either text utterances provided directly by the user 108 of the digital assistant 106, or the result of converting input speech utterances into text format. However, this is not intended to be limiting or restrictive in any way.

[0036] For example, user input 108 might request to order a pizza by providing an utterance such as "I want to order a pizza." Upon receiving such an utterance, the digital assistant 106 is configured to understand the meaning of the utterance and take appropriate action. These appropriate actions may include responding to the user with questions that request user input regarding the type of pizza the user wants to order, the size of the pizza, the pizza toppings, etc. The responses provided by the digital assistant 106 may be in natural language form and may generally be in the same language as the input utterance. As part of the generation of these responses, the digital assistant 106 may perform natural language generation (NLG). If the user orders a pizza, through a conversation between the user and the digital assistant 106, the digital assistant may provide all the information necessary for ordering the pizza and then guide the user to order the pizza at the end of the conversation. The digital assistant 106 may terminate the conversation by outputting information to the user indicating that the pizza has been ordered.

[0037] At a conceptual level, the digital assistant 106 performs various processes in response to utterances received from the user. In some embodiments, this process includes a series of processing steps or a pipeline of processing steps, which include, for example, a step of understanding the meaning of the input utterance (sometimes referred to as natural language understanding (NLU)), a step of determining what action to take in response to the utterance, a step of performing the action as appropriate, a step of generating a response to be output to the user in response to the user utterance, and a step of outputting the response to the user. NLU processing may include parsing the received input utterance to understand its structure and meaning, and refining and improving the utterance to create a more understandable form (e.g., logical form) or structure of the utterance. The step of generating a response may include the use of NLG technology.

[0038] NLU processing performed by a digital assistant such as digital assistant 106 may include a variety of NLP-related processes, such as sentence parsing (e.g., tokenizing, digitizing, and identifying part-of-speech tags of utterances, identifying named entities in the sentence, generating dependency trees to represent sentence structure, splitting the sentence into clauses, parsing individual clauses, decomposing introductory repetitions, performing chunking, etc.). In certain embodiments, the NLU processing or a portion of it is performed by digital assistant 106 itself. In some other embodiments, digital assistant 106 may use other resources to perform a portion of the NLU processing. For example, the syntax and structure of an input utterance can be identified by processing the sentence using a parser, part-of-speech tagger, and / or named entity recognizer. In one implementation, for English, a parser, part-of-speech tagger, and named entity recognizer, such as those provided by the Stanford Natural Language Processing (NLP) Group, are used to analyze the sentence structure and syntax. These are provided as part of the Stanford CoreNLP Toolkit.

[0039] The various examples provided in this disclosure demonstrate English utterances, but these are intended to be examples only. In certain embodiments, the digital assistant 106 may also handle utterances in languages ​​other than English. The digital assistant 106 may provide subsystems (e.g., components implementing NLU functionality) configured to perform processing for various languages. These subsystems may be implemented as pluggable units that can be invoked using service calls from the NLU core server. This makes the NLU processing flexible and extensible for each language, which includes allowing various orders of processing. Language packs may be provided for individual languages, and language packs may register a list of subsystems that may be supplied from the NLU core server.

[0040] A digital assistant, such as the digital assistant 106 shown in Figure 1, may be made available or accessible to its user 108 through a variety of different channels, including but not limited to specific applications, social media platforms, various messaging services and applications, and other applications or channels. A single digital assistant may have several channels configured for it so that it runs simultaneously on different services and is simultaneously accessible by different services.

[0041] A digital assistant or chatbot system typically includes or is associated with one or more skills. In certain embodiments, these skills are individual chatbots (referred to as skillbots) configured to interact with the user and perform specific types of tasks, such as tracking inventory, submitting time cards, creating expense reports, ordering food, verifying bank accounts, making reservations, and purchasing widgets. For example, in the embodiment shown in Figure 1, the digital assistant or chatbot system 106 includes skills 116-1, 116-2, and so on. For the purposes of this disclosure, the terms “skill” and “multiple skills” are used synonymously with the terms “skillbot” and “multiple skillbots,” respectively.

[0042] Each skill associated with a digital assistant helps the user of the digital assistant complete tasks through a conversation with the user, which may include a combination of text or voice input provided by the user and responses provided by the skill bot. These responses may be in the form of text or voice messages to the user, and / or in the form of simple user interface elements (e.g., list selection) presented to the user for the user to make a choice.

[0043] There are various ways in which skills or skillbots can be associated with or added to digital assistants. In some cases, skillbots are developed by companies and then added to digital assistants using DABP102. In other cases, skillbots are developed and created using DABP102 and then added to digital assistants created using DABP102. In yet another case, DABP102 provides an online digital store (referred to as the "skill store") offering multiple skills directed towards a wide range of tasks. Skills offered through the skill store may also be published to various cloud services. To add skills to digital assistants generated using DABP102, DABP102 users can access the skill store via DABP102, select the desired skills, and indicate that the selected skills will be added to the digital assistant created using DABP102. Skills from the Skill Store may be added to the digital assistant as is, or in a modified form (for example, a DABP102 user may select a specific skill bot provided by the Skill Store, clone it, customize or modify the selected skill bot, and then add the modified skill bot to a digital assistant created using DABP102).

[0044] A digital assistant or chatbot system may be implemented using various different architectures. For example, in a particular embodiment, a digital assistant created and deployed using DABP102 may be implemented using a master-bot / child (or sub)bot paradigm or architecture. According to this paradigm, the digital assistant is implemented as a master bot that interacts with one or more child bots, which are skill bots. For example, in the embodiment shown in Figure 1, the digital assistant 106 comprises a master bot 114 and skill bots 116-1, 116-2, etc., which are child bots of the master bot 114. In a particular embodiment, the digital assistant 106 itself is considered to function as the master bot.

[0045] A digital assistant implemented according to a master-child bot architecture allows the user of the digital assistant to interact with multiple skills through a unified user interface, i.e., through the master bot. When a user engages with the digital assistant, user input is received by the master bot. The master bot then performs processing to determine the meaning of the user input utterance. The master bot then determines whether it can handle the task requested by the user in the utterance, and in addition, selects an appropriate skill bot to handle the user request and routes the conversation to the selected skill bot. This allows the user to converse with the digital assistant through a common, single interface, while still providing the ability to use several skill bots configured to perform specific tasks. For example, in a digital assistant developed for an enterprise, the master bot of this digital assistant may connect to skill bots with specific functions, such as a CRM bot to perform functions related to customer relationship management (CRM), an ERP bot to perform functions related to enterprise resource planning (ERP), and an HCM bot to perform functions related to human capital management (HCM). Thus, end-users or consumers of digital assistants only need to know how to access them through a common master bot interface, with multiple skill bots in place behind the scenes to handle user requests.

[0046] In a particular embodiment, the master bot / child bot infrastructure The master bot is configured to be aware of a list of available skill bots. The master bot can access metadata that identifies the various available skill bots and, for each skill bot, the capabilities of the skill bot, including the tasks it can perform. Upon receiving a user request in the form of an utterance, the master bot is configured to identify or predict from among several available skill bots which particular skill bot can best serve or process the user request. The master bot then routes this utterance (or part of an utterance) to that particular skill bot for further processing. Thus, control flows from the master bot to the skill bots. The master bot can support multiple input and output channels. In certain embodiments, routing may be performed with the help of processing performed by one or more available skill bots. For example, as described later, a skill bot can be trained to infer the intent of an utterance and determine whether the inferred intent matches an intent in which the skill bot is configured. Thus, routing performed by the master bot may include the skill bot communicating to the master bot whether it is configured with an intent suitable for processing the utterance.

[0047] The embodiment in Figure 1 shows that the digital assistant 106 comprises a master bot 114 and skill bots 116-1, 116-2, and 116-3, but this is not intended to be limiting. The digital assistant may include various other components (e.g., other systems and subsystems) that provide the functionality of the digital assistant. These systems and subsystems may be implemented solely in software (e.g., code, instructions stored on a computer-readable medium and executable by one or more processors), solely in hardware, or in implementations using a combination of software and hardware.

[0048] DABP102 provides infrastructure, as well as various services and features, that enable DABP102 users to create digital assistants that include one or more skillbots associated with the digital assistant. In some cases, skillbots can be created by cloning existing skillbots, for example, by cloning skillbots provided by the skill store. As described above, DABP102 provides a skill store or skill catalog that offers multiple skillbots for performing various tasks. DABP102 users can clone skillbots from this skill store. Modifications or customizations may be made to the cloned skillbots as needed. In some other cases, DABP102 users create skillbots from scratch using the tools and services provided by DABP102. As described above, the skill store or skill catalog provided by DABP102 may offer multiple skillbots for performing various tasks.

[0049] In a particular embodiment, creating or customizing a skill bot broadly involves the following steps:

[0050] (1) Steps to configure the new skillbot (2) Steps to configure one or more intents for the skillbot (3) Steps to create one or more entities for one or more intents (4) Steps to train the skill bot (5) Steps to create a dialog flow for the skill bot (6) Add custom components to the skillbot as needed. (7) Steps to test and deploy the skill bot Each of the above steps is briefly explained below.

[0051] (1) Steps to configure the settings for a new skillbot - various settings can be configured for a skillbot. For example, a skillbot designer can specify one or more invocation names for the skillbot being created. Users of the digital assistant can then explicitly invoke the skillbot using these invocation names. For example, a user can explicitly invoke the corresponding skillbot by typing the invocation name in their utterance.

[0052] (2) Steps to constitute one or more intents and associated utterance examples for the skillbot - The skillbot designer specifies one or more intents (also referred to as bot intents) for the skillbot being created. The skillbot is then trained based on these specified intents. These intents represent categories or classes of input utterances that the skillbot is trained to infer. Upon receiving an utterance, the trained skillbot infers the intent for the utterance, and the inferred intent is selected from a predefined set of intents used to train the skillbot. The skillbot then takes an appropriate action in response to the utterance based on the intent inferred for that utterance. In some cases, the skillbot's intents represent tasks that the skillbot can perform for the user of the digital assistant. Each intent is given an intent identifier or intent name. For example, for a skillbot trained for banking, the intents specified for this skillbot might be "CheckBalance," "TransferMoney," "DepositCheck," etc. It may include.

[0053] For each intent defined for a skillbot, the skillbot designer may also provide one or more utterance examples that represent and describe the intent. These utterance examples are intended to represent utterances that a user might input to the skillbot for that intent. For example, for the CheckBalance intent, utterance examples might include "What is the balance in my savings account?", "How much is in my checking account?", and "How much is in my account?". Thus, various reorders of typical user utterances may be designated as utterance examples for an intent.

[0054] Intents and their associated utterance examples are used as training data to train a skillbot. Various different training techniques may be used. As a result of this training, a predictive model is generated, which is configured to take an utterance as input and output the intent of that utterance as inferred by the predictive model. In some cases, the input utterance is provided to an intent analysis engine configured to predict or infer the intent of the input utterance using the trained model. The skillbot can then take one or more actions based on the inferred intent.

[0055] (3) Steps to construct entities for one or more intents of the skill bot - In some cases, additional context may be needed to enable the skill bot to respond appropriately to user utterances. For example, there may be situations where user input utterances are broken down into the same intent in the skill bot. For example, in the example above, the utterances "What is the balance in my savings account?" and "How much is in my checking account?" both break down into the same CheckBalance intent, but these utterances are different requests that ask for different things. To clarify such requests, one or more entities are added to the intent. Using the example of a banking skill bot, an entity called AccountType defines the values ​​called "checking" and "savings". This can enable the skillbot to parse user requests and respond appropriately. In the example above, the utterances are broken down into the same intent, but the AccountType entity is related. The associated values ​​differ between these two utterances. This allows the SkillBot to perform different actions for the two utterances, even though they break down into the same intent. One or more entities can specify a particular intent configured for the SkillBot. Therefore, entities are used to add context to the intent itself. Entities help to further elaborate on the intent, enabling the SkillBot to complete the user request.

[0056] In certain embodiments, there are two types of entities: (a) built-in entities provided by DABP102 and (2) custom entities that can be specified by the skillbot designer. Built-in entities are general-purpose entities that can be used with a wide variety of bots. Examples of built-in entities include, but are not limited to, time, date, address, number, email address, period, recurring period, currency, phone number, URL, etc. Custom entities are used for more customized applications. For example, in a banking skill, the AccountType entity enables various banking transactions by checking user input for keywords such as checking, savings, and credit card. It can be defined by the designer.

[0057] (4) Steps to train the skillbot - The skillbot is configured to receive user input in the form of utterances, parse or process the received input to identify or select intents associated with the received user input. As described above, the skillbot must be trained for this purpose. In certain embodiments, the skillbot is trained on intents configured for the skillbot and utterance examples associated with those intents (collectively, the training data) so that the skillbot can decompose user input utterances into one of its configured intents. In certain embodiments, the skillbot uses a predictive model that is trained using the training data to enable the skillbot to discern what the user is saying (or, in some cases, what the user is trying to say). DABP102 provides a variety of different training techniques that skillbot designers can use to train the skillbot, and these training techniques include a variety of machine learning-based training techniques, rule-based training techniques and / or combinations thereof. In certain embodiments, a portion of the training data (e.g., 80%) is used to train the skillbot model, and another portion (e.g., the remaining 20%) is used to test or validate the model. Once trained, a trained model (sometimes referred to as a trained skillbot) can be used to process and respond to user utterances. In certain cases, a user utterance may be a question that requires only one answer and does not require further conversation. To address such situations, a Q&A (question and answer) intent may be defined for the skillbot. This allows the skillbot to output an answer to a user request without having to update the dialogue definition. Q&A intents are created in the same manner as regular intents. The dialogue flow for Q&A intents may differ from that of regular intents.

[0058] (5) Step to create a dialogue flow for the skillbot - The dialogue flow specified for the skillbot describes how the skillbot will react when its various intents are broken down in response to user input received. This dialogue flow defines the actions or behaviors the skillbot will perform, such as how the skillbot will respond to user utterances, how the skillbot will request input from the user, and how the skillbot will return data. A skillbot's dialogue flow is similar to a flowchart that a skillbot follows. Skillbot designers specify the dialogue flow using a language such as Markdown. In certain embodiments, a version of YAML called OBotML may be used to specify the skillbot's dialogue flow. The skillbot's dialogue flow definition serves as a model of the conversation itself, that is, a model that allows the skillbot designer to construct the interaction between the skillbot and the user it serves.

[0059] In a particular embodiment, the SkillBot's dialog flow definition includes the following three sections:

[0060] (a) Context section (b) Default transition section (c) State section Context Section - Skillbot designers can define variables used in the conversation flow within the context section. Other variables that can be named within the context section include, but are not limited to, variables for error handling, variables for built-in or custom entities, and user variables that allow Skillbot to recognize and adhere to user preferences.

[0061] Default Transition Section - Skillbot transitions can be defined in either a dialog flow state section or a default transition section. Transitions defined in the default transition section act as fallbacks and are triggered when there are no applicable transitions defined within a state, or when the conditions necessary to trigger a state transition cannot be met. The default transition section can be used to define routing that allows Skillbot to deal directly with unexpected user actions.

[0062] The State Section – Dialog Flow and its associated behavior are defined as a set of transient states that govern the logic within the dialog flow. Each state node in the dialog flow definition names a component that provides the functionality required for the dialog at that moment. Thus, states are built around components. States contain characteristics specific to the component and define transitions to other states that are triggered after the component has been executed.

[0063] The state section may be used to address exceptional scenarios. For example, you might want to temporarily leave the first skill the user is involved with in place and give the user the option to do something in a second skill within the digital assistant. For instance, a user might be conversing with a shopping skill (e.g., making a purchase selection), then want to jump to a banking skill (e.g., to confirm they have enough money for the purchase), and then return to the shopping skill to complete the user's order. To address this, the action in the first skill could be configured to return to the original flow after initiating an interaction with a second, different skill within the same digital assistant.

[0064] (6) Step of adding custom components to the skillbot - As described above, the states specified in the skillbot's dialog flow name the components that provide the necessary functionality corresponding to those states. The components enable the skillbot to perform the functions. In certain embodiments, DABP102 provides a set of pre-configured components for performing a wide range of functions. The skillbot designer can select one or more of these pre-configured components and associate them with states in the skillbot's dialog flow. The designer can use the tools provided by DABP102 to create custom or new components and associate these custom components with one or more states in the SkillBot's dialog flow.

[0065] (7) Steps to test and deploy the skill bot - DABP102 provides several features that allow the skill bot designer to test the skill bot being developed. The skill bot can then be deployed and included in the digital assistant.

[0066] The above description explains how to create a skillbot, but similar techniques may also be used to create a digital assistant (or masterbot). At the masterbot or digital assistant level, built-in system intents may be configured for the digital assistant. These built-in system intents are used to identify common tasks that the digital assistant itself (i.e., the masterbot) can handle without calling the skillbot associated with the digital assistant. Examples of system intents defined for a masterbot include: (1) Exit: applied when the user expresses a desire to exit the current conversation or context in the digital assistant; (2) Help: applied when the user requests help or orientation; and (3) UnresolvedIntent: applied to user input that does not well match the exit and help intents. The digital assistant also stores information about one or more skillbots associated with it. This information allows the masterbot to select a specific skillbot to process the utterance.

[0067] At the master bot or digital assistant level, when a user inputs a phrase or utterance into the digital assistant, the digital assistant is configured to perform processing to determine how to route the utterance and the associated conversation. The digital assistant makes this determination using a routing model, which may be rule-based, AI-based, or a combination thereof. Using this routing model, the digital assistant determines whether the conversation corresponding to the user-input utterance is routed to a specific skill for processing, processed by the digital assistant or master bot itself on a per-built-in system intent basis, or processed as a different state in the current conversation flow.

[0068] In certain embodiments, as part of this process, the digital assistant determines whether the user input utterance explicitly identifies a skillbot using its invocation name. If an invocation name is present in the user input, it is treated as an explicit invocation of the skillbot corresponding to the invocation name. In such scenarios, the digital assistant may route the user input to the explicitly invoked skillbot for further processing. If there is no specific or explicit invocation, in certain embodiments, the digital assistant evaluates the received user input utterance to calculate a confidence score for the system intent and the skillbot associated with the digital assistant. The calculated score for a skillbot or system intent indicates how likely the user input is to represent a task that the skillbot is configured to perform, or to represent a system intent. Any system intent or skillbot whose associated calculated confidence score exceeds a threshold (e.g., a confidence threshold routing parameter) is selected as a candidate for further evaluation. The digital assistant then selects a specific system intent or skillbot from the identified candidates for further processing of the user input utterance. In certain embodiments, after one or more skillbots are identified as candidates, they are associated with those candidate skills. The intents are evaluated (according to the intent model for each skill), and a confidence score is calculated for each intent. Generally, any intent with a confidence score exceeding a threshold (e.g., 70%) is treated as a candidate intent. When a specific skill bot is selected, the user utterance is routed to that skill bot for further processing. When a system intent is selected, one or more actions are performed by the master bot itself according to the selected system intent.

[0069] Figure 2 is a simplified block diagram of a master bot (MB) system 200 according to a particular embodiment. The MB system 200 may be implemented in software only, in hardware only, or in a combination of hardware and software. The MB system 200 includes a preprocessing subsystem 210, a multiple intent subsystem (MIS) 220, an explicit call subsystem (EIS) 230, a skillbot caller 240, and a data store 250. The MB system 200 shown in Figure 2 is merely one example of the arrangement of components in a master bot. Those skilled in the art will recognize many possible modifications, alternatives, and variations. For example, in some implementations, the MB system 200 may have more or fewer systems or components than those shown in Figure 2, may combine two or more subsystems, or may have different configurations or arrangements of subsystems.

[0070] The preprocessing subsystem 210 receives the utterance "A" 202 from the user and processes this utterance via the language detector 212 and the language syntactic analyzer 214. As described above, the utterance can be provided in various forms, including speech or text. The utterance 202 may be a sentence fragment, a complete sentence, multiple sentences, etc. The utterance 202 may contain punctuation. For example, if the utterance 202 is provided as speech, the preprocessing subsystem 210 may convert this speech to text using a speech-to-text converter (not shown), which inserts punctuation marks, such as commas, semicolons, periods, etc., into the resulting text.

[0071] The language detector 212 detects the language of utterance 202 based on the text of utterance 202. The manner in which utterance 202 is processed is language-dependent because each language has its own grammar and semantics. Differences between languages ​​are taken into account when analyzing the syntax and structure of the utterance.

[0072] The language parser 214 parses the utterance 202 and extracts part-of-speech (POS) tags for each linguistic unit (e.g., word) within the utterance 202. POS tags include, for example, nouns (NN), pronouns (PN), verbs (VB), etc. The language parser 214 may also tokenize the linguistic units of the utterance 202 (e.g., converting each word into a separate token) to create headwords. Headwords are the primary form of a set of words, as they appear in a dictionary (e.g., "run" is a headword such as run, runs, ran, running, etc.). Other types of preprocessing that the language parser 214 can perform include chunking compound expressions, for example, combining "credit" and "card" into a single expression, "credit_card". The language parser 214 may also identify relationships between words within the utterance 202. For example, in some embodiments, the language parser 214 generates a dependency tree indicating which parts of the utterance (e.g., a specific noun) are direct objects, which parts of the utterance are prepositions, and so on. The results of the processing performed by the language parser 214 form extracted information 205, which is provided as input to the MIS 220 along with the utterance 202 itself.

[0073] As described above, utterance 202 may contain two or more sentences. Multiple intents and explicit statements. For the purpose of detecting explicit calls, utterance 202 can be treated as a single unit even if it contains multiple sentences. However, in certain embodiments, preprocessing may be performed, for example, by the preprocessing subsystem 210, to identify a single sentence among multiple sentences for multiple intent parsing and explicit call parsing. In general, whether utterance 202 is processed at the level of individual sentences or as a single unit containing multiple sentences, the results produced by MIS220 and the results produced by EIS230 are substantially identical.

[0074] MIS220 determines whether utterance 202 represents multiple intents. While MIS220 can detect the presence of multiple intents in utterance 202, the processing performed by MIS220 does not include determining whether the intents in utterance 202 match any intent configured for the bot. Instead, the processing to determine whether the intents in utterance 202 match a bot intent can be performed by the intent classifier 242 of the MB system 200 or the skill bot's intent classifier (shown in the embodiment of Figure 3). The processing performed by MIS220 assumes the existence of a bot capable of processing utterance 202 (e.g., a specific skill bot or the master bot itself). Therefore, the processing performed by MIS220 does not require knowledge of what bots are in the chatbot system (e.g., the identity of a skill bot registered with the master bot) or what intents are configured for a particular bot.

[0075] To determine if utterance 202 contains multiple intents, MIS220 applies one or more rules from a set of rules 252 in datastore 250. The rules applied to utterance 202 depend on the language of utterance 202 and may include sentence patterns indicating the presence of multiple intents. For example, a sentence pattern may include a coordinating conjunction (e.g., a conjunction) that connects two parts of a sentence, where both parts correspond to separate intents. If utterance 202 matches a sentence pattern, it can be inferred that utterance 202 represents multiple intents. Note that utterances with multiple intents do not necessarily have different intents (e.g., intents directed to different bots or intents directed to different intents within the same bot). Instead, the utterance may have separate instances of the same intent, for example, "order a pizza using payment account X, then order a pizza using payment account Y."

[0076] As part of its determination that utterance 202 represents multiple intents, MIS220 also determines which part of utterance 202 is associated with each intent. For each intent represented in an utterance containing multiple intents, MIS220 constructs a new utterance for alternative processing, such as utterances "B" 206 and "C" 208 shown in Figure 2, to replace the original utterance. Thus, the original utterance 202 can be split into two or more separate utterances, each processed one at a time. Using extracted information 205 and / or from the analysis of utterance 202 itself, MIS220 determines which of the two or more utterances should be processed first. For example, MIS220 may determine that utterance 202 contains a marker word indicating that a particular intent should be processed first. The newly formed utterance corresponding to this particular intent (e.g., either utterance 206 or utterance 208) is sent first for further processing by EIS230. After the conversation initiated by the first utterance has ended (or been temporarily interrupted), the next highest priority utterance (for example, utterance 206 or the other of utterance 208) may be sent to the EIS230 for processing.

[0077] The EIS230 determines whether the utterance it receives (for example, utterance 206 or utterance 208) contains the invocation name of the skillbot. In certain embodiments, Each skillbot in the chatbot system is assigned a unique invocation name that distinguishes it from other skillbots in the chatbot system. A list of invocation names can be maintained in the data store 250 as part of the skillbot information 254. An utterance is considered an explicit invocation if it contains a word match with an invocation name. If a bot is not explicitly invoked, the utterance received by the EIS 230 is considered an implicit invocation utterance 234 and is fed into the master bot's intent classifier (e.g., intent classifier 242) to determine which bot to use to process the utterance. In some cases, the intent classifier 242 determines that the master bot should process the implicit invocation utterance. In other cases, the intent classifier 242 determines which skillbot the utterance is routed to for processing.

[0078] The explicit call functionality provided by EIS230 offers several advantages. It can reduce the amount of processing the master bot must perform. For example, with explicit calls, the master bot does not need to perform intent classification analysis (e.g., using intent classifier 242), or it can reduce the amount of intent classification analysis it must perform to select a skill bot. Therefore, explicit call analysis can enable the selection of a specific skill bot without relying on intent classification analysis.

[0079] Furthermore, there may be overlapping functionalities among multiple skill bots. This can occur, for example, when two skill bots process the same or very similar intents. In such situations, it would be difficult for the master bot to identify which of the multiple skill bots to select based solely on intent classification analysis. In such scenarios, an explicit call clarifies which skill bot should be used.

[0080] In addition to determining whether an utterance is an explicit invocation, the EIS230 is responsible for determining whether any part of the utterance should be used as input to the explicitly invoked skill bot. Specifically, the EIS230 can determine whether any part of the utterance is not associated with an invocation. The EIS230 can make this determination through parsing the utterance and / or parsing the extracted information 205. Instead of sending the entire utterance received by the EIS230, the EIS230 can send only the parts of the utterance that are not associated with an invocation to the invoked skill bot. In some cases, the input to the invoked skill bot is formed simply by removing any part of the utterance that is associated with the invocation. For example, "I want to order a pizza using PizzaBot" can be shortened to "I want to order a pizza" because "using PizzaBot" is related to the invocation of PizzaBot but irrelevant to the process performed by PizzaBot. In some cases, the EIS230 may reformat the parts sent to the invoked bot, for example, to form a complete sentence. Therefore, the EIS230 determines not only whether there is an explicit call, but also what to send to the skillbot if there is an explicit call. In some cases, there may be no text to input to the bot being called. For example, if the utterance is "pizzabot", the EIS230 would determine that pizzabot is being called, but there is no text to process by pizzabot. In such a scenario, the EIS230 may inform the skillbot caller 240 that there is nothing to send.

[0081] The skillbot caller 240 can call skillbots in various ways. For example, the skillbot caller 240 can call a bot in response to receiving an indication 235 that a particular skillbot has been selected as a result of an explicit call. The instruction 235 may be transmitted by the EIS230 along with the input to the explicitly invoked skillbot. In this scenario, the skillbot caller 240 delegates control of the conversation to the explicitly invoked skillbot. The explicitly invoked skillbot determines an appropriate response to the input from the EIS230 by treating the input as a standalone utterance. For example, this response may involve performing a specific action or initiating a new conversation in a specific state where the initial state of the new conversation depends on the input transmitted from the EIS230.

[0082] Another way in which the SkillBot Invocation 240 can invoke a SkillBot is through an implicit invocation using the Intent Classifier 242. The Intent Classifier 242 may be trained using machine learning and / or rule-based training techniques to determine the likelihood that an utterance represents a task that a particular SkillBot is configured to perform. The Intent Classifier 242 is trained for different classes, i.e., one class for each SkillBot. For example, whenever a new SkillBot is registered with the MasterBot, the Intent Classifier 242 can be trained using a list of utterance examples associated with this new SkillBot to determine the likelihood that a particular utterance represents a task that the new SkillBot can perform. Parameters generated as a result of this training (e.g., a set of parameter values ​​for a machine learning model) may be stored as part of the SkillBot Information 254.

[0083] In certain embodiments, the intent classifier 242 is implemented using a machine learning model, as described further in this specification. Training the machine learning model may involve inputting at least a subset of utterances from utterance examples associated with various skill bots to generate inferences as the output of the machine learning model about which bot is the correct bot to process any particular training utterance. For each training utterance, an indication that the correct bot is used for this training utterance may be provided as ground truth information. The behavior of the machine learning model can then be adapted (e.g., via backpropagation) to minimize the difference between the generated inferences and the ground truth information.

[0084] In certain embodiments, the intent classifier 242 determines a confidence score for each skill bot registered with the master bot, indicating the likelihood that the skill bot can process an utterance (e.g., an implicit calling utterance 234 received from the EIS 230). The intent classifier 242 may also determine a confidence score for each configured system-level intent (e.g., help, exit). If a particular confidence score satisfies one or more conditions, the skill bot caller 240 invokes the bot associated with that particular confidence score. For example, it might need to meet a threshold confidence score value. Thus, the output 245 of the intent classifier 242 is either an identification of a system intent or an identification of a particular skill bot. In some embodiments, in addition to meeting a threshold confidence score value, the confidence score must exceed the next highest confidence score by a certain win margin. Imposing such conditions would allow routing to a particular skill bot if each of the confidence scores of multiple skill bots exceeds a threshold confidence score value.

[0085] After identifying a bot based on the confidence score evaluation, the skillbot caller 240 hands over processing to the identified bot. In the case of a system intent, the identified bot is the master bot. Otherwise, the identified bot is a skillbot. Furthermore, the skillbot caller 240 determines what to provide as input 247 to the identified bot. As described above, in the case of an explicit call, input 247 may be based on a portion of an utterance not associated with this call, or input 247 may be nothing (e.g., an empty string). In the case of an implicit call, input 247 is the entire utterance. obtain.

[0086] The data store 250 comprises one or more computing devices that store data used by various subsystems of the master bot system 200. As described above, the data store 250 includes rules 252 and skill bot information 254. Rules 252 include rules for the MIS 220 to determine, for example, when an utterance represents multiple intents and how to divide an utterance that represents multiple intents. Rules 252 further include rules for the EIS 230 to determine which part of an utterance that explicitly invokes a skill bot should be sent to the skill bot. Skill bot information 254 includes the invocation names of skill bots in the chatbot system, for example, a list of invocation names of all skill bots registered with a particular master bot. Skill bot information 254 may also include confidence scores for each skill bot in the chatbot system, for example, information used by the intent classifier 242 to determine parameters for a machine learning model.

[0087] Figure 3 is a simplified block diagram of a SkillBot System 300 according to a particular embodiment. The SkillBot System 300 is a computing system that can be implemented with software only, hardware only, or a combination of hardware and software. In certain embodiments, such as the embodiment shown in Figure 1, the SkillBot System 300 can be used to implement one or more SkillBots in a digital assistant.

[0088] The Skillbot system 300 includes an MIS 310, an intent classifier 320, and a conversation manager 330. The MIS 310 is similar to the MIS 220 in Figure 2 and provides similar functionality, including being operable to determine (1) whether an utterance represents multiple intents and, if so, (2) how to split this utterance into separate utterances for each of the multiple intents, using rules 352 in the data store 350. In a particular embodiment, the rules applied by the MIS 310 to detect multiple intents and to split an utterance are identical to the rules applied by the MIS 220. The MIS 310 receives an utterance 302 and extracted information 304. The extracted information 304 is similar to the extracted information 205 in Figure 1 and can be generated using a language parser 214 or a language parser local to the Skillbot system 300.

[0089] The intent classifier 320 can be trained in a manner similar to that of the intent classifier 242 described above in relation to the embodiment of Figure 2, as described in further detail herein. For example, in a particular embodiment, the intent classifier 320 is implemented using a machine learning model. The machine learning model of the intent classifier 320 is trained for a particular skill bot using at least a subset of utterance examples associated with that particular skill bot as training utterances. The ground truth for each training utterance would be the particular bot intent associated with that training utterance.

[0090] Utterance 302 may be received directly from the user or supplied via a master bot. If utterance 302 is supplied via a master bot as a result of processing via MIS220 and EIS230 in the embodiment shown in Figure 2, for example, MIS310 can be bypassed to avoid repeating processing already performed by MIS220. However, if utterance 302 is received directly from the user during a conversation that occurs after routing to a skill bot, for example, MIS310 can process utterance 302 to determine whether utterance 302 represents multiple intents. If so, MIS310 applies one or more rules to split utterance 302 into separate utterances for each intent, for example, utterance "D" 306 and utterance "E" 308. If the utterance does not represent multiple intents, the MIS310 forwards the utterance 302 to the intent classifier 320 for intent classification without splitting the utterance 302.

[0091] The intent classifier 320 is configured to match received utterances (e.g., utterances 306 or 308) with intents associated with the skillbot system 300. As described above, a skillbot may consist of one or more intents, each intent including at least one example utterance associated with the intent and used to train the classifier. In the embodiment of Figure 2, the intent classifier 242 of the masterbot system 200 is trained to determine confidence scores for individual skillbots and system intents. Similarly, the intent classifier 320 may be trained to determine confidence scores for each intent associated with the skillbot system 300. The classification performed by the intent classifier 242 is bot-level, while the classification performed by the intent classifier 320 is intent-level and therefore more granular. The intent classifier 320 can access intent information 354. The intent information 354 includes a list of utterances for each intent associated with the skillbot system 300, where this list of utterances represents and explains the meaning of the intent and is generally associated with the tasks that can be performed by that intent. The intent information 354 may further include parameters generated as a result of training on this list of utterances.

[0092] The conversation manager 330 receives an indication 322 as output from the intent classifier 320, which indicates that a particular intent identified by the intent classifier 320 best matches the utterance input to the intent classifier 320. In some cases, the intent classifier 320 may not be able to determine any match. For example, if the utterance is directed to a system intent or an intent of a different skillbot, the confidence score calculated by the intent classifier 320 may fall below the threshold confidence score value. When this happens, the skillbot system 300 may direct the utterance to the master bot for processing, for example, to route it to a different skillbot. However, if the intent classifier 320 successfully identifies an intent within the skillbot, the conversation manager 330 initiates a conversation with the user.

[0093] A conversation initiated by the conversation manager 330 is a conversation specific to an intent identified by the intent classifier 320. For example, the conversation manager 330 may be implemented using a state machine configured to execute a dialogue flow about the identified intent. This state machine may include a default initiation state (e.g., the intent is invoked without any additional input) and one or more further states, each associated with an action performed by the skillbot (e.g., executing a purchase transaction) and / or a dialogue presented to the user (e.g., a question, a response). Thus, the conversation manager 330 can determine an action / dialogue 335 upon receiving an indication 322 that identifies an intent, and can determine further actions or dialogues in response to subsequent utterances received during the conversation.

[0094] The data store 350 comprises one or more computing devices that store data used by various subsystems of the skillbot system 300. As shown in Figure 3, the data store 350 includes rules 352 and intent information 354. In certain embodiments, the data store 350 can be integrated with the data store of the master bot or digital assistant, for example, the data store 250 in Figure 2.

[0095] Systems and architectures for OOD detection When an utterance is received by a chatbot, the chatbot must accurately determine whether the utterance is in-domain or out-of-domain. Models used to classify utterances as intents have been found to be overconfident and may provide poor results for text that is OOD (Good for Knowledge). To overcome this problem, various embodiments have been directed towards techniques that use clustering-based and metrics-based approaches to calculate the probability of whether an utterance belongs to a target domain (e.g., a given skillbot). The probabilities calculated from the clustering-based and metrics-based approaches are then combined into an ensemble approach to obtain the best from both approaches. This ensemble approach ultimately classifies the utterance as in-domain or out-of-domain to the target domain based on the final combined probabilities.

[0096] Figure 4 is a block diagram showing a view of a chatbot system 400 configured to train and utilize a classifier (e.g., the intent classifiers 242 or 320 described with respect to Figures 2 and 3) based on text data 405. As shown in Figure 4, the text classification performed by the chatbot system 400 in this example includes several stages: a predictive model training stage 410; a skillbot invocation stage 415 to determine whether the utterance represents a task (e.g., in-domain or out-of-domain) that a particular skillbot is configured to perform; and an intent prediction stage 420 to classify the utterance as one or more intents. The predictive model training stage 410 constructs and trains one or more predictive models 425a to 425n (where "n" represents any natural number) (which may be referred to herein individually or collectively as predictive models 425) to be used by the other stages. For example, predictive model 425 may include one or more models (or an ensemble of models) for determining the likelihood that an utterance represents a task configured to be performed by a particular skillbot (e.g., calculating the probability of whether or not the utterance belongs to a target domain), another model for predicting intent from an utterance for a first type of skillbot, and another model for predicting intent from an utterance for a second type of skillbot. Other types of predictive models may be implemented in other examples relating to this disclosure.

[0097] The predictive model 425 may be a machine learning ("ML") model such as a convolutional neural network ("CNN") (e.g., an inception neural network, a residual neural network ("Resnet")), or a recurrent neural network (e.g., a long-short-term memory ("LSTM") model or a gated recurrent unit ("GRU") model), or another variant of a deep neural network ("DNN") (e.g., a stacked highway network, a wide and deep learning network with linear models and deep neural networks, a multi-label n-binary DNN classifier, or a multi-class DNN classifier for single-intent classification). Alternatively, the predictive model 425 may be another suitable ML model trained for natural language processing, such as a naive Bayes classifier, a linear classifier, a support vector machine, a bagging model such as a random forest model, a boosting model, a shallow neural network, or a combination of one or more such techniques (e.g., a CNN-HMM or MCNN (multiscale convolutional neural network)). The chatbot system 400 may use the same type of predictive model or a different type of predictive model to predict intents from utterances for a first type of skillbot and for a second type of skillbot, in order to determine the likelihood that the utterance represents a task configured to be performed by a particular skillbot. Other types of predictive models may be implemented in other examples relating to this disclosure.

[0098] To train various predictive models 425, the training phase 410 consists of three main components: dataset preparation 430, feature engineering 435, and model training 440. Dataset preparation 430 includes the process of loading data assets 445 and performing basic preprocessing by dividing the data assets 445 into training and validation sets 445a-n so that the system can train and test the predictive models 425. Data assets 445 may include at least a subset of utterances from utterance examples associated with various skillbots. As described above, utterances can be provided in various forms, including speech or text. Utterances may be sentence fragments, complete sentences, multiple sentences, etc. For example, if utterances are provided as speech, data preparation 430 may convert this speech to text using a speech-to-text converter (not shown), which inserts punctuation marks, such as commas, semicolons, periods, etc., into the resulting text. In some cases, utterance examples are provided by a client or customer. In other cases, utterance examples are automatically generated from a pre-existing library of utterances (for example, identifying utterances specific to the skill the chatbot is learning from the library). The data asset 445 for the predictive model 425 may include input text or speech (or input features of text or speech frames) and labels 450 corresponding to this input text or speech (or input features) as a matrix or table of values. For example, for each training utterance, ground truth information may be provided for label 450 indicating that the correct bot is used for this training utterance. The behavior of the predictive model 425 can then be adapted (for example, via backpropagation) to minimize the difference between the generated inference and the ground truth information. Alternatively, the predictive model 425 may be trained for a particular skill bot by using a subset of utterance examples associated with at least a particular skill bot as training utterances. The ground truth information for label 450 for each training utterance would be the specific bot intent associated with this training utterance.

[0099] In various embodiments, data preparation 430 includes OOD data augmentation 455 of data asset 445 to make the predictive model 425 more resilient to OOD utterances by including OOD utterance examples in various contexts. By augmenting data asset 445 with OOD examples in various contexts, the predictive model 425 becomes better able to focus on the most important parts of these examples and the contexts that link them to those classes, including the OOD classes. The augmentation 455 may be implemented using OOD augmentation techniques to combine OOD utterances in various contexts with the original utterances in data asset 445. OOD augmentation techniques may include four operations, which generally include: (i) generating a dataset containing multiple OOD examples; (ii) filtering out OOD examples with contexts too similar to the context of the original utterance; and (iii) supplying OOD examples to the model during training in a batch process to balance OOD examples with in-domain examples, as the number of OOD examples may be much larger than the number of in-domain utterances, with batch supply starting with batches containing easier OOD examples and progressing to batches containing more difficult OOD examples.

[0100] In some cases, further augmentation (by OOD augmentation) may be applied to data asset 445. For example, Easy Data Augmentation (EDA) techniques may be used to improve the performance of a text classification task. EDA includes four operations that help prevent overfitting and train a more robust model: synonym substitution, random insertion, random swap, and random deletion. Note that, in contrast to OOD augmentation, EDA operations generally (i) take words from the original text and (ii) incorporate these words into each data asset 445 relative to the original text. For example, the synonym substitution operation is stock The random insertion operation involves randomly selecting n non-stop words from the original sentence (e.g., an utterance) and replacing each of these words with one of its randomly selected synonyms. The random insertion operation involves finding a random synonym for a random non-stop word in the original sentence n times and inserting that synonym at a random position in the sentence. The random swap operation involves randomly selecting two words in the sentence n times and swapping their positions. The random deletion operation involves randomly removing each word in the sentence with probability p.

[0101] In various embodiments, feature engineering 435 involves using an encoding model such as a Multilingual Universal Sentence Encoder (MUSE) to transform a data asset 445 into a feature vector and / or create new features using the data asset 445. The encoding model is a model that can map natural language elements such as sentences, words, and n-grams (a collection of n characters / words) to an array of numbers. Thus, each natural language element can be represented as a single point in the vector space. The goal is to obtain a representation of sentences, words, and n-grams that a computing device can use for data processing without losing too much information. The feature vector may include count vectors as features, term frequency inverse document frequency (TF-IDF) vectors such as word-level, n-gram-level, or character-level, word embeddings as features, text / NLP as features, topic models as features, or a combination thereof. Count vectors are a matrix representation of data asset 445, where each row represents an utterance, each column represents a term from the utterance, and each cell represents the frequency count of a particular term within the utterance. TF-IDF scores represent the relative importance of terms within an utterance. Word embeddings are a form of representing words and utterances using dense vector representations. The position of a word in the vector space is learned from the text and is based on the words surrounding that word at the time of use. Text / NLP-based features may include the number of words in an utterance, the number of characters in an utterance, mean word density, punctuation count, capitalization count, headword count, frequency distribution of part-of-speech tags (e.g., nouns and verbs), or any combination thereof. Topic modeling is a technique for identifying groups of words (called topics) from a collection of utterances that contain the best information.

[0102] In various embodiments, model training 440 includes training a predictive model 425 using sentence embeddings with feature vectors and / or new features created in feature engineering 435. In some cases, the training process includes iterative operations to find a set of parameters for the predictive model 425 that minimizes the loss or error function of the predictive model 425. Each iteration may include finding a set of parameters such that the value of the loss or error function using that set of parameters for the predictive model 425 is less than the value of the loss or error function using a different set of parameters in a previous iteration. The loss or error function can be constructed to measure the difference between the output predicted using the predictive model 425 and the labels 450 contained in the data asset 445. Once the set of parameters is identified, the predictive model 425 is fully trained and can be used for prediction as designed.

[0103] In addition to the data asset 445, labels 450, feature vectors, and / or new features, other techniques and information can also be used to refine the training process of the predictive model 425. For example, feature vectors and / or new features may be combined to help improve the accuracy of the classifier or model. Furthermore or alternatively, hyperparameters may be tuned or optimized, for example, multiple parameters such as tree length, leaves, and network parameters may be fine-tuned to obtain the best-fit model. While the training mechanisms described herein primarily focus on training the predictive model 425, these training mechanisms can be trained from other data assets. It can also be used to fine-tune existing predictive models 425. For example, in some cases, predictive model 425 may have been pre-trained using utterances specific to a different skill bot. In such cases, predictive model 425 can be retrained using data asset 445 (for example, by OOD augmentation).

[0104] The predictive model training stage 410 outputs a trained predictive model 425 which includes a task predictive model 460 and an intent predictive model 465. The task predictive model 460 may be used in the skillbot invocation stage 415 to determine the likelihood (470) that an utterance represents a task that a particular skillbot is configured to perform, and the intent predictive model 465 may be used in the intent predictive stage 420 to classify an utterance as one or more intents (475). In some cases, the skillbot invocation stage 415 and the intent predictive stage 420 may proceed to separate models independently in some cases. For example, the trained intent predictive model 465 may be used in the intent predictive stage 420 to predict the intent of a skillbot without first identifying the skillbot in the skillbot invocation stage 415. Similarly, the task predictive model 460 may be used in the skillbot invocation stage 415 to predict the task or skillbot used in an utterance without first identifying the intent of the utterance in the intent predictive stage 420.

[0105] Alternatively, the skillbot invocation stage 415 and the intent prediction stage 420 may be executed sequentially, with one stage using the output of the other as input, or one stage being invoked in a manner specific to a particular skillbot based on the output of the other. For example, for a given text data 405, a skillbot caller can invoke a skillbot via an implicit invocation using the skillbot invocation stage 415 and the task prediction model 460. The task prediction model 460 may be trained using machine learning and / or rule-based training techniques to determine the likelihood that an utterance represents a task that a particular skillbot 470 is configured to perform. Then, for the identified or invoked skillbot and the given text data 405, the intent prediction stage 420 and the intent prediction model 465 can be used to match the received utterance (e.g., an utterance in a given data asset 445) with an intent 475 associated with the skillbot. As described herein, a skillbot may consist of one or more intents, each intent including at least one utterance example associated with the intent and used to train a classifier. In some embodiments, the skillbot invocation stage 415 and task prediction model 460 used in a masterbot system are trained to determine confidence scores for individual skillbots and system intents. Similarly, the intent prediction stage 420 and intent prediction model 465 may be trained to determine confidence scores for each intent associated with the skillbot system. The classification performed by the skillbot invocation stage 415 and task prediction model 460 is bot-level, while the classification performed by the intent prediction stage 420 and intent prediction model 465 is intent-level and therefore more granular.

[0106] Figure 5 is a block diagram showing a phase of a model architecture 500 that provides clustering-based and metrics-based approaches for calculating the probability of whether an utterance belongs to a target domain (for example, the skillbot invocation stage 415 described with reference to Figure 4). The model architecture 500 comprises a clustering component 505, a classification component 510, and an ensemble component 515. The clustering component 505 comprises two stages, namely (i) an unsupervised clustering model 520 and (ii) an outlier detection model 525. The unsupervised clustering model 520 finds clusters within the in-domain data 535. (530), the centroid is calculated (540), and the clustering-based approach is shared with the metrics-based approach to generate an embedding representation for the cluster (545). An outlier detection model 525 is constructed with a distance or density algorithm (e.g., Z-score, K-means, DBSCAN, local outlier detection (LOF), isolated forest, etc.) to provide a probability 550 (e.g., a second probability) of whether the input utterance 555 belongs to the target domain. The classification component 510 comprises two stages: (i) an unsupervised clustering model 520 and (ii) a distance learning model 560. The distance learning model 560 is constructed with a deep learning network 575 having trained model parameters configured to compute an absolute difference 565 between a sentence embedding 570 for the input utterance 555 and an embedding representation 545 for the cluster, and to provide a probability 580 (e.g., a first probability) of whether the input utterance 555 belongs to the target domain. The ensemble component 515 is configured to evaluate probabilities 580 and 550 to determine a final probability 585 regarding whether the input utterance 555 belongs to the target domain, and to classify the input utterance 555 as either in-domain or out-of-domain for the chatbot based on the final probability 585.

[0107] With respect to the clustering-based approach performed by the clustering component 505, the in-domain data 535 used to train an unsupervised clustering algorithm consists of in-domain utterances associated with a particular domain or skill bot (e.g., only pizza order training data). An embedding model 590 (e.g., MUSE) may be used to generate sentence embeddings for each in-domain utterance by mapping natural language elements, including sentences, words, and n-grams, to arrays of digits. Each natural language element is represented as a single point in the vector space. Thus, each sentence embedding is a vector of values ​​representing natural language elements. An unsupervised clustering algorithm (e.g., K-means, affinity propagation, cohesive clustering, balanced iterative shrinking and clustering (BIRCH), DBSCAN, mean shift, ordering of points to identify clustering structures (OPTICS), etc.) takes data points (i.e., sentence embeddings for each in-domain utterance) as input and groups them into clusters. This grouping process is the training phase of the unsupervised clustering algorithm. The result would be an unsupervised clustering model 520 that, following the training it has undergone, takes a data sample (e.g., sentence embeddings for a new intradomain utterance) as input and returns the cluster to which the new data point belongs. Once clusters are determined for the intradomain data 535, an embedding representation 545 is generated for each cluster. The embedding representation 545 is the average of the sentence embeddings for each intradomain utterance in the cluster. The clustering process narrows the intradomain data 535 down to more manageable-sized embedding representations 545 for each cluster (e.g., 1000 or fewer embedding representations, 500 or fewer embedding representations, or 250 or fewer embedding representations).

[0108] The outlier detection model 525 optionally includes an unsupervised clustering algorithm (e.g., K-means, affinity propagation, cohesive clustering, BIRCH, DBSCAN, mean shift, OPTICS, etc.) that takes data points (i.e., embedding representations 545 and centroid calculations for clusters obtained by centroid calculation 535 and cluster detection 530) as input and further groups them into elaborate clusters. This grouping process is the training phase of the unsupervised clustering algorithm. The result would be an unsupervised clustering model that, according to the training the unsupervised clustering model has undergone, takes data samples (e.g., embedding representations and centroid calculations for clusters) as input and returns the elaborate cluster to which the new data point belongs. Once elaborate clusters are obtained for the embedding representations 545, for each elaborate cluster... An elaborate embedding representation is generated. This elaborate embedding representation is the average of the embedding representations 545 for each elaborate cluster. The outlier detection model 525 comprises a distance or density algorithm (e.g., Z-score, K-means, DBSCAN, local outlier detection (LOF), isolated forest, etc.) configured to determine the distance or density deviation between the sentence embedding 570 for the input utterance 555 and the embedding representation (or modified embedding representation) for adjacent clusters. Based on the determined distance or density deviation, the outlier detection model 525 predicts a probability 525 about whether the input utterance 555 belongs to the target domain. For example, the outlier detection model 525 may consider an input utterance 555 to be an outlier if it is at a considerable distance from any adjacent cluster or has a substantially lower density than any adjacent cluster, and then use this outlier to provide a probability 525 about whether the input utterance 555 belongs to the target domain.

[0109] With respect to the metrics-based approach performed by the classification component 510, the deep learning network 575 may be trained using a set of training data, which includes (i) sentence embeddings for input utterances, (ii) embedding representations for each cluster consisting of sentence embeddings for in-domain data, and (iii) the absolute difference between each embedding representation for each cluster consisting of sentence embeddings for input utterances and sentence embeddings for in-domain data. The in-domain data 535 used to train the deep learning network 575 comprises in-domain utterances associated with various domains or skill bots (for example, not only pizza order training data, but also training data from other available domains or bots such as payroll roster bots, weather bots, bank account bots, etc.).

[0110] In some embodiments, the deep learning network 575 is a stacked highway network having a nonlinear transformation as part of the gating function. Model parameters for the stacked highway network can be learned using a set of training data. During training of the distance learning model 560 with the set of training data, the high-dimensional features of sentence embeddings and embedding representations for each cluster are transformed into low-dimensional vectors, which are then concatenated with features from in-domain utterances and fed into the hidden layers of the deep neural network, where the values ​​of the low-dimensional vectors are randomly initialized and learned, along with the model parameters, to minimize the loss function.

[0111] Once trained, the stacked highway network can determine similarities or differences between sentence embeddings 570 for an input utterance 555 and each embedding representation 545 for each cluster. An embedding model 595 (e.g., MUSE) can be used to generate sentence embeddings 570 for an input utterance 555 by mapping natural language elements, including sentences, words, and n-grams, to arrays of digits. Each natural language element is represented as a single point in the vector space. Thus, a sentence embedding is a vector of values ​​representing natural language elements.

[0112] A layered highway network can be formalized as follows:

[0113]

number

[0114] The similarity or difference between the sentence embedding 570 for the input utterance 555 and the embedding representation 545 for each cluster can be determined by (i) calculating the absolute difference 565 between the sentence embedding for the utterance and the embedding representation for each cluster, (ii) inputting the absolute difference 565, the sentence embedding 570 for the input utterance, and the embedding representation 545 for each cluster into a stacked highway network, and (iii) using the stacked highway network, the absolute difference 565, the sentence embedding 570 for the input utterance 555, and the embedding representation 545 for each cluster to determine the similarity or difference between the sentence embedding 570 for the input utterance 555 and the embedding representation 545 for each cluster. As shown in Figure 5, the absolute difference between the sentence embedding 570 for the input utterance 555 and each embedding representation 545 for each cluster is calculated by taking the absolute difference (e.g., |UV|=[0.2,0.3,0.9]|) between the vector value of the sentence embedding 570 for the input utterance 555 (e.g., V=[0.1,0.4,-0.5]) and the vector value of each embedding representation 545 for each cluster composed of sentence embeddings for the intradomain data 535 (e.g., U=[0.3,0.1,0.4]). The probability 580 of whether 55 belongs to the target domain can be predicted by a stacked highway network based on the similarities or differences found between the sentence embedding 570 for the input utterance 555 and each embedding representation 545 for each cluster.

[0115] In another embodiment, the deep learning network 575 is a wide and deep learning network having a linear model and a deep neural network. The linear model comprises model parameters trained using a set of training data. The set of training data includes the absolute difference between sentence embeddings for utterances and each embedding representation for each cluster for intradomain utterances from multiple domains. During the training of the linear model with the set of training data, a linear relationship between sentence embeddings for utterances and each embedding representation for each cluster is learned using a hypothetical function. During the learning of the linear relationship, the multiple model parameters are learned to minimize a loss function. The deep learning network comprises model parameters trained using a set of training data. The set of training data includes sentence embeddings for intradomain utterances from multiple domains. During the training of a deep learning network using a set of training data, high-dimensional features of sentence embeddings for in-domain utterances are converted into low-dimensional vectors. These low-dimensional vectors are then concatenated with features from the in-domain utterances and fed into the hidden layers of the deep neural network. The values ​​of the low-dimensional vectors are randomly initialized and, along with several model parameters, are learned to minimize the loss function.

[0116] Once trained, the wide and deep learning network can determine similarities or differences between the sentence embeddings 570 for the input utterance 555 and each embedding representation 545 for each cluster. An embedding model 595 (e.g., MUSE) can be used to generate the sentence embeddings 570 for the input utterance 555 by mapping natural language elements, including sentences, words, and n-grams, to arrays of digits. Each natural language element is represented as a single point in the vector space. Thus, a sentence embedding is a vector of values ​​representing natural language elements.

[0117] Determining the similarity or difference between the sentence embedding 570 for the input utterance 555 and each embedding representation 545 for each cluster may involve (i) calculating the absolute difference 565 between the sentence embedding 570 for the input utterance 555 and each embedding representation 545 for each cluster; (ii) inputting the absolute difference 565, the sentence embedding 570 for the input utterance 565, and each embedding representation 545 for each cluster into a wide and deep learning network; (iii) using the linear model and the absolute difference 565 to predict the wide-based probability of whether the input utterance 555 belongs to the target domain; and (iv) using the deep neural network, the sentence embedding 570 for the input utterance 555, and each embedding representation 545 for each cluster to determine the similarity or difference between the sentence embedding 570 for the input utterance 555 and each embedding representation 545 for each cluster. The probability 580 of whether the input utterance 555 belongs to the target domain can be predicted by using the final layer of a wide and deep learning network to evaluate the wide probability and the similarity or difference between the sentence embedding 570 for the input utterance 555 and each embedding representation 545 for each cluster.

[0118] Ensemble component 515 evaluates probabilities 580 and 550 to obtain a final probability 585 regarding whether the input utterance 555 belongs to the target domain, and based on the final probability 585, classifies the input utterance 555 as either in-domain or out-of-domain for the chatbot. In certain cases, ensemble component 515 uses the following in_domain_prob function (in_domain_prob(ensemble,x) = max(in_domain_prob(cluster-based,x),in_domain_prob(metric-based,x)), which in_domain_prob function is: This function returns the in-domain probability of utterance x, considering both clustering-based and metrics-based approaches. Essentially, an utterance is within the target domain if either approach indicates that x is within the domain, and outside the target domain if both approaches indicate that x is outside the domain (the error on the utterance x side is within the domain).

[0119] Techniques for OOD judgment Figure 6 is a flowchart of a process 600 for identifying OOD utterances according to a particular embodiment. The process shown in Figure 6 may be implemented by software (e.g., code, instructions, programs) executed by one or more processing units (e.g., processors, cores) of each system, by hardware, or by a combination thereof. The software may be stored in a non-temporary storage medium (e.g., a memory device). The method shown in Figure 6 and described below is intended to be illustrative and non-limiting. Figure 6 shows various processing steps performed in a particular sequence or order, but this is not intended to be limiting. In certain alternative embodiments, these steps may be performed in a different order, or some steps may be performed in parallel. In certain embodiments, such as the embodiments shown in Figures 1 to 5, the process shown in Figure 6 may be performed by a trained model architecture (e.g., model architecture 500) for identifying OOD utterances.

[0120] In 605, the utterance and the chatbot's target domain are received (for example, the skill or chatbot described in Figures 1, 2, and 3). The target domain is defined for chatbots that specialize in specific types of tasks, such as inventory tracking, time card submission, and expense report creation.

[0121] In 610, a sentence embedding is generated for an utterance. This sentence embedding for an utterance can be generated using an embedding model that maps natural language elements, including sentences, words, and n-grams, to arrays of numbers. Each natural language element is represented as a single point in the vector space. Thus, each sentence embedding is a vector of values ​​representing natural language elements.

[0122] In 615, embedding representations are obtained for each cluster of multiple clusters of intradomain utterances associated with the target domain. Each embedding representation for each cluster is the average of the sentence embeddings for each intradomain utterance in the cluster. Obtaining embedding representations for each cluster may involve obtaining intradomain utterances based on the target domain (for example, if the target domain is ordering pizza, then all intradomain utterances relate to utterances associated with ordering pizza, such as "I want to order a cheese pizza"), generating sentence embeddings for each intradomain utterance, inputting the sentence embeddings for each intradomain utterance into an unsupervised clustering model configured to interpret the intradomain utterances and identify multiple clusters in the feature space of the intradomain utterances, using the unsupervised clustering model to classify the sentence embeddings for each intradomain utterance into one of the multiple clusters based on the similarity or difference between the features of the sentence embeddings and the features of the sentence embeddings within each cluster, calculating the centroid for each of the multiple clusters, and outputting the embedding representations and the centroids for each of the multiple clusters. Unsupervised clustering models may include K-means clustering, affinity propagation, agglomeration clustering, BIRCH, DBSCAN, mean shift, and OPTICS.

[0123] Sentence embeddings for each domain utterance can be generated using an embedding model that maps natural language elements, including sentences, words, and n-grams, to arrays of numbers. Each natural language element is represented as a single point in the vector space. Thus, each sentence embedding is a vector of values ​​representing natural language elements.

[0124] In step 620, sentence embeddings for the utterance and embedding representations for each cluster are input to a distance learning model, which has trained model parameters configured to provide a first probability of whether the utterance belongs to the target domain. In step 625, the distance learning model is used to determine the similarity or difference between the sentence embeddings for the utterance and each embedding representation for each cluster. In step 630, the distance learning model is used to predict a first probability of whether the utterance belongs to the target domain, based on the determined similarity or difference between the sentence embeddings for the utterance and each embedding representation for each cluster.

[0125] In some embodiments, the distance learning model comprises a stacked highway network with a nonlinear transformation as part of the gating function. Determining the similarity or difference between the sentence embedding for an utterance and each embedding representation for each cluster involves (i) calculating the absolute difference between the sentence embedding for an utterance and each embedding representation for each cluster, (ii) inputting the absolute difference, the sentence embedding for an utterance and each embedding representation for each cluster into the stacked highway network, and (iii) inputting the stacked highway network, the absolute difference, the sentence embedding for an utterance and each embedding representation for each cluster The system may include using expressions to determine the similarity or difference between sentence embeddings for utterances and each embedding expression for each cluster.

[0126] Model parameters for a stacked highway network can be learned using a set of training data, which includes (i) sentence embeddings for utterances, (ii) embedding representations for each cluster consisting of sentence embeddings for intradomain utterances from multiple domains, and (iii) the absolute difference between each embedding representation for each cluster consisting of sentence embeddings for utterances and sentence embeddings for intradomain utterances. During training of a distance learning model using the training data set, the high-dimensional features of the sentence embeddings and embedding representations for each cluster are transformed into low-dimensional vectors, which are then concatenated with features from intradomain utterances and fed into the hidden layers of a deep neural network, where the values ​​of the low-dimensional vectors are randomly initialized and learned, along with the model parameters, to minimize a loss function.

[0127] In other embodiments, the distance learning model comprises a wide and deep learning network having a linear model and a deep neural network. Determining the similarity or difference between a sentence embedding for an utterance and each embedding representation for each cluster may comprise: (i) calculating the absolute difference between the sentence embedding for an utterance and each embedding representation for each cluster; (ii) inputting the absolute difference, the sentence embedding for an utterance, and each embedding representation for each cluster into the wide and deep learning network; (iii) using the linear model and the absolute difference to predict a wide-based probability of whether the utterance belongs to a target domain; and (iv) using the deep neural network, the sentence embedding for an utterance, and each embedding representation for each cluster to determine the similarity or difference between the sentence embedding for an utterance and each embedding representation for each cluster. Predicting a first probability comprises using the final layer of the wide and deep learning network to evaluate the wide probability and the similarity or difference between the sentence embedding for an utterance and each embedding representation for each cluster.

[0128] A linear model comprises model parameters trained using a set of training data. This training data includes the absolute difference between sentence embeddings for utterances and each embedding representation for each cluster, for intradomain utterances from multiple domains. During training of the linear model with this training data, a linear relationship between sentence embeddings for utterances and each embedding representation for each cluster is learned using a hypothetical function. While learning this linear relationship, multiple model parameters are learned to minimize a loss function.

[0129] A deep learning network has model parameters trained using a set of training data. The training data set includes sentence embeddings for intradomain utterances from multiple domains. During training of the deep learning network with the training data set, the high-dimensional features of the sentence embeddings for intradomain utterances are transformed into low-dimensional vectors. These low-dimensional vectors are then concatenated with features from the intradomain utterances and fed into the hidden layers of the deep neural network. The values ​​of the low-dimensional vectors are randomly initialized and, along with multiple model parameters, are learned to minimize the loss function.

[0130] In 635, sentence embeddings for utterances and embedding representations for each cluster are input to an outlier detection model, which is constructed using a distance or density algorithm for outlier detection. This distance or density algorithm may be Z-score, K-means algorithm, DBSCAN, local outlier detection (LOF), or isolated forest.

[0131] In step 640, an outlier detection model is used to determine the distance or density deviation between the sentence embedding for the utterance and the embedding representation for adjacent clusters. In step 645, an outlier detection model is used to predict a second probability of whether the utterance belongs to the target domain, based on the determined distance or density deviation. This prediction may comprise calculating a z-score for the utterance based on the distance or density deviation between the sentence embedding for the utterance and the embedding representation for adjacent clusters, and then applying a sigmoid function to the z-score to determine the second probability of whether the utterance belongs to the target domain.

[0132] In step 650, the first and second probabilities are evaluated to determine the final probability of whether the utterance belongs to the target domain. In step 655, based on the final probability, the utterance is classified as either in-domain or out-of-domain for the chatbot. The probabilities calculated from the clustering-based approach and the distance-based approach are combined as an ensemble approach to obtain the best of both approaches. In a specific case, the ensemble approach has (in_domain_prob(ensemble,x) = max(in_domain_prob(cluster-based,x),in_domain_prob(metric-based,x)), where this in_domain_prob function is clustering This function returns the in-domain probability of utterance x, considering both a metrics-based approach and a metrics-based approach. Basically, an utterance is within the target domain if either approach says x is within the domain, and outside the target domain if both approaches say x is outside the domain (the error on the utterance x side is within the domain).

[0133] Exemplary System Figure 7 is a simplified diagram of a distributed system 700. In the example shown, the distributed system 700 includes one or more client computing devices 702, 704, 706, and 708, which are connected to a server 712 via one or more communication networks 710. The client computing devices 702, 704, 706, and 708 may be configured to run one or more applications.

[0134] In various examples, server 712 may be adapted to run one or more services or software applications that enable one or more embodiments described in this disclosure. In certain examples, server 712 may also provide other services or software applications that may include non-virtual and virtual environments. In some examples, these services may be provided to users of client computing devices 702, 704, 706 and / or 708 as web-based services or cloud services, such as under a Software-as-a-Service (SaaS) model. Users operating client computing devices 702, 704, 706 and / or 708 may then interact with server 712 using one or more client applications to access the services provided by these components.

[0135] In the configuration shown in Figure 7, server 712 may include one or more components 718, 720, and 722 that implement the functions performed by server 712. These components may include software components, hardware components, or a combination thereof that can be executed by one or more processors. It should be understood that various different system configurations are possible, which may differ from the distributed system 700. The example shown in Figure 7 is therefore an example of a distributed system for implementing an exemplary system and is not intended to be limiting. do not have.

[0136] A user may use client computing devices 702, 704, 706, and / or 708 to run one or more applications, models, or chatbots, which may generate one or more events or models that may be run or supplied in accordance with the teachings of this disclosure. A client device may provide an interface that allows a user of the client device to interact with the client device. The client device may also output information to the user through this interface. Although Figure 7 shows only four client computing devices, any number of client computing devices may be supported.

[0137] Client devices may include various types of computing systems, such as portable handheld devices, general-purpose computers (personal computers and laptops, etc.), workstation computers, wearable devices, gaming systems, thin clients, various messaging devices, sensors or other sensing devices. These computing devices may run various types and versions of software applications and operating systems (e.g., Microsoft Windows®, Apple Macintosh®, UNIX® or UNIX-like operating systems, Linux® or Linux-like operating systems (e.g., Google Chrome® OS)), including various mobile operating systems (e.g., Microsoft Windows Mobile®, iOS®, Windows Phone®, Android®, BlackBerry®, Palm OS®). Portable handheld devices may include mobile phones, smartphones (e.g., iPhone®), tablets (e.g., iPad®), personal digital assistants (PDAs), etc. Wearable devices may include Google Glass® head-mounted displays and other devices. Gaming systems may include a variety of handheld gaming devices, internet-enabled gaming devices (for example, Microsoft Xbox® game consoles with or without Kinect® gesture input devices, Sony PlayStation® systems, and various gaming systems offered by Nintendo®). Client devices may include a variety of internet-related applications and communication applications (for example, email applications, short message services). It may be possible to run various different applications, such as Sage Service (SMS) applications, and may use various communication protocols.

[0138] Network 710 could be any type of network familiar to a person skilled in the art, capable of supporting data communication using any of the various available protocols, including, but not limited to, TCP / IP (Transmission Control Protocol / Internet Protocol), SNA (System Network Architecture), IPX (Internet Packet Switching), AppleTalk®, etc. Just as an example, Network 710 could be a Local Area Network (LAN), an Ethernet®-based network, Token Ring, a Wide Area Network (WAN), the Internet, a Virtual Network, a Virtual Private Network (VPN), an Intranet, an Extranet, a Public Switched Telephone Network (PSTN), an Infrared Network, a Wireless Network (for example, a network operating under any of the IEEE 1002.11 protocols, Bluetooth®, and / or other wireless protocols), and / or any combination of these and / or other networks. .

[0139] Server 712 may consist of one or more general-purpose computers, specialized server computers (including, for example, PC (personal computer) servers, UNIX® servers, midrange servers, mainframe computers, rack-mount servers), server farms, server clusters, or other appropriate configurations and / or combinations. Server 712 may include other computing architectures, including virtualization, such as one or more virtual machines running a virtual operating system, or one or more flexible pools of logical storage devices that can be virtualized to maintain virtual storage devices for the server. In various examples, Server 712 may be adapted to run one or more services or software applications that provide the functionality described in the foregoing disclosure.

[0140] The computing system in Server 712 may run one or more operating systems, including any of the above, and any server operating system available on the market. Server 712 may also run any of a variety of other server applications and / or middle-tier applications, including HTTP (Hypertext Transfer Protocol) servers, FTP (File Transfer Protocol) servers, CGI (Common Gateway Interface) servers, Java® servers, and database servers. Exemplary database servers include, but are not limited to, those available on the market from Oracle Corporation®, Microsoft Corporation®, Sybase Corporation®, IBM Corporation® (International Business Machines), and others.

[0141] In some implementations, server 712 may include one or more applications for analyzing and integrating data feeds and / or event update information received from users of client computing devices 702, 704, 706, and 708. For example, data feeds and / or event update information may include, but are not limited to, Twitter® feeds, Facebook® updates, or real-time updates received from one or more third-party sources and continuous data streams, and may include real-time events related to sensor data applications, financial stock market boards, network performance measurement tools (e.g., network monitoring and traffic management applications), clickstream analysis tools, and automotive traffic monitoring. Server 712 may also include one or more applications for displaying data feeds and / or real-time events via one or more display devices of client computing devices 702, 704, 706, and 708.

[0142] The distributed system 700 may also include one or more data repositories 714,716. In a particular example, these data repositories may be used to store data and other information. For example, one or more of the data repositories 714,716 may be used to store information such as information related to generated models used by the chatbot, which are used by the server 712 when performing the chatbot's performance or various functions according to different embodiments. The data repositories 714,716 may be in various locations. For example, the data repository used by the server 712 may be local to the server 712, or it may be remote from the server 712 and communicate with the server 712 via a network-based connection or a dedicated connection. The data repositories 714,716 may be of different types. In a particular example, the data repository used by the server 712 may be a database, such as a relational database, provided by Oracle Corporation and other vendors. These databases may also be databases. One or more of these databases may be adapted to allow the storage, updating, and retrieval of data to and from databases that respond to SQL-formatted commands.

[0143] In a particular example, one or more of the data repositories 714,716 may also be used by an application to store application data. The data repositories used by the application may be of different types, such as a key-value storage repository, an object storage repository, or a general storage repository supported by the file system.

[0144] In a particular example, the functions described herein may be provided as services through a cloud environment. Figure 8 is a simplified block diagram of a cloud-based system environment in a particular example in which various services may be provided as cloud services. In the example shown in Figure 8, the cloud infrastructure system 802 may provide one or more cloud services that may be requested by a user using one or more client computing devices 804, 806 and 808. The cloud infrastructure system 802 may comprise one or more computers and / or servers, which may include the above-described server 812. The computers in the cloud infrastructure system 802 may be organized as general-purpose computers, specialized server computers, server farms, server clusters, or other appropriate configurations and / or combinations.

[0145] Network 810 can facilitate the transmission and exchange of data between clients 804, 806, and 808 and the cloud infrastructure system 802. Network 810 may include one or more networks. These networks may be of the same type or of different types. Network 810 may support one or more communication protocols, including wired and / or wireless protocols, to facilitate communication.

[0146] The example shown in Figure 8 is merely one example of a cloud infrastructure system and is not intended to be limiting. It should be understood that in several other examples, the cloud infrastructure system 802 may have more or fewer components than those shown in Figure 8, may combine two or more components, or may have different configurations or arrangements of components. For example, while Figure 8 shows three client computing devices, any number of client computing devices may be supported in alternative examples.

[0147] The term "cloud service" is generally used to refer to services made available to users on demand via communication networks such as the Internet by a service provider's system (e.g., cloud infrastructure system 802). Generally, in a public cloud environment, the servers and systems that make up the cloud service provider's system are different from the customer's own on-premises servers and systems. The cloud service provider's system is managed by the cloud service provider. Therefore, customers can use these services without having to purchase separate licenses, support, or hardware and software resources for the cloud services provided by the cloud service provider. For example, the cloud service provider's system may host an application, and a user can order this application on demand via the Internet without having to purchase the infrastructure resources to run it. It can be used as follows. Cloud services are designed to provide easy and scalable access to applications, resources, and services. Several providers offer cloud services. For example, some cloud services, such as middleware services, database services, and Java cloud services, are offered by Oracle Corporation (registered trademark) in Redwood Shores, California.

[0148] In a particular example, the cloud infrastructure system 802 may provide one or more cloud services using various models, including a hybrid service model, a software-as-a-service (SaaS) model, a platform-as-a-service (PaaS) model, and an infrastructure-as-a-service (IaaS) model. The cloud infrastructure system 802 may include a set of applications, middleware, databases, and other resources that enable the delivery of various cloud services.

[0149] The SaaS model allows applications or software to be delivered to customers as a service over a communication network such as the internet, without requiring customers to purchase the hardware or software for the basic application. For example, the SaaS model can be used to provide customers with access to on-demand applications hosted by a cloud infrastructure system. Examples of SaaS services offered by Oracle Corporation (registered trademark) include, but are not limited to, a variety of services for human resources / capital management, customer relationship management (CRM), enterprise resource planning (ERP), supply chain management (SCM), enterprise performance management (EPM), analytics services, and social applications.

[0150] The IaaS model is generally used to provide customers with resilient computing and storage capabilities by offering infrastructure resources (e.g., servers, storage, hardware, and networking resources) as cloud services. Various IaaS services are offered by Oracle Corporation®.

[0151] The PaaS model is generally used to provide resources as a service that enable customers to develop, run, and manage applications and services without having to procure, build, or maintain platform and environmental resources themselves. Examples of PaaS services offered by Oracle Corporation (registered trademark) include, but are not limited to, Oracle Java Cloud Services (JCS), Oracle Database Cloud Services (DBCS), Data Management Cloud Services, and various application development solution services.

[0152] Cloud services are generally provided in an on-demand self-service, subscription-based, elastically scalable, reliable, highly available, and secure manner. For example, a customer may order one or more services provided by the cloud infrastructure system 802 through a subscription order. The cloud infrastructure system 802 then processes to provide the services requested in the customer's subscription order. For example, a user may use utterances to request the cloud infrastructure system to take a specific action (e.g., an intent) as described above, and / or to provide services to a chatbot system as described herein. The cloud infrastructure system 802 may be configured to provide one or more cloud services.

[0153] Cloud infrastructure system 802 supports various deployment models Cloud services can be provided through a third-party cloud service provider. In the public cloud model, the cloud infrastructure system 802 may be owned by a third-party cloud service provider, and the cloud services are provided to any general public customer, who may be an individual or a company. In certain other examples, under the private cloud model, the cloud infrastructure system 802 may be operated within an organization (e.g., within a corporate organization), and services are provided to customers within that organization. For example, customers may be various departments within the company, such as the human resources department or the payroll department, or individuals within that company. In certain other examples, under the community cloud model, the cloud infrastructure system 802 and the services provided may be shared by several organizations within the relevant community. Various other models, such as hybrids of the above models, may also be used.

[0154] The client computing devices 804, 806, and 808 may be of different types (such as the client computing devices 702, 704, 706, and 708 shown in Figure 7) and may be capable of running one or more client applications. Users may use the client devices to interact with the cloud infrastructure system 802, such as requesting services provided by the cloud infrastructure system 802. For example, users may use the client devices to request information or actions from a chatbot as described in this disclosure.

[0155] In some examples, the processing performed by the cloud infrastructure system 802 to provide a service may include training and deploying models. This analysis may include using, analyzing, and manipulating a dataset to train and deploy one or more models. This analysis may be performed by one or more processors that process the data in parallel or use the data to perform simulations. For example, big data analysis may be performed by the cloud infrastructure system 802 to generate and train one or more models for a chatbot system. The data used for this analysis may include structured data (e.g., data stored in a database or data structured according to a structured model) and / or unstructured data (e.g., data blobs (binary large objects)).

[0156] As shown in the example in Figure 8, the cloud infrastructure system 802 may include infrastructure resources 830 that are used to facilitate the provision of various cloud services offered by the cloud infrastructure system 802. Infrastructure resources 830 may include, for example, processing resources, storage or memory resources, networking resources, etc. In a particular example, a storage virtual machine available to supply storage requested by an application may be part of the cloud infrastructure system 802. In other examples, the storage virtual machine may be part of a different system.

[0157] In a particular example, to facilitate the efficient provisioning of these resources to support various cloud services provided to various customers by the cloud infrastructure system 802, resources may be bundled into sets of resources or resource modules (also referred to as "pods"). Each resource module or pod may contain a pre-integrated and optimized combination of one or more types of resources. In a particular example, different pods may be pre-provisioned for different types of cloud services. For example, a first set of pods may be provisioned for database services, and a second set of pods may contain a different combination of resources than the pods in the first set of pods, for Java services. For example, resources can be provisioned. In some services, the resources allocated to provisioning a service can be shared among services.

[0158] The cloud infrastructure system 802 itself may internally use services 832 shared by various components of the cloud infrastructure system 802, thereby facilitating the provisioning of services by the cloud infrastructure system 802. These internally shared services may include, but are not limited to, security and identity services, integration services, enterprise repository services, enterprise manager services, virus scanning and whitelisting services, high availability, backup and recovery services, services to enable cloud support, email services, notification services, and file transfer services.

[0159] The cloud infrastructure system 802 may comprise multiple subsystems. These subsystems may be implemented in software, hardware, or a combination thereof. As shown in Figure 8, these subsystems may include a user interface subsystem 812 that enables users or customers of the cloud infrastructure system 802 to interact with it. The user interface subsystem 812 may include various different interfaces, such as a web interface 814, an online store interface 816 (where cloud services provided by the cloud infrastructure system 802 are advertised and available for purchase by consumers), and other interfaces 818. For example, a customer may use a client device to request one or more services provided by the cloud infrastructure system 802 using one or more of interfaces 814, 816, and 818 (a service request 834). For example, a customer may access an online store to browse cloud services provided by the cloud infrastructure system 802 and place a subscription order for one or more services provided by the cloud infrastructure system 802 that the customer wishes to subscribe to. This service request may include information identifying the customer and the one or more services the customer wishes to subscribe to. For example, a customer may place a subscription order for services provided by the cloud infrastructure system 802. As part of the order, the customer may provide information identifying the chatbot system on which the service is provided, and optionally, one or more credentials for said chatbot system.

[0160] In certain examples, such as the example shown in Figure 8, the cloud infrastructure system 802 may include an Order Management Subsystem (OMS) 820 configured to process new orders. As part of this process, the OMS 820 may be configured to create a customer account if one has not already been made, receive invoice issuance and / or billing information from the customer which will be used to issue an invoice to the customer in order to provide the requested services to the customer, verify the customer information, confirm the customer order upon verification, orchestrate various workflows, and prepare the order for provisioning.

[0161] Once properly authenticated, OMS820 may then invoke the Order Provisioning Subsystem (OPS)824. OPS824 is configured to provision resources for an order, including processing, memory, and networking resources. Provisioning may involve allocating resources for an order and configuring these resources to facilitate the services requested by the customer order. The manner in which resources are provisioned for an order and the types of resources provided may depend on the type of cloud service ordered by the customer. For example, following a workflow, the OPS824 may be configured to determine that a specific cloud service is being requested and to identify the number of pods that would have been pre-configured for that specific cloud service. The number of pods allocated to an order may depend on the size / volume / level / scope of the requested service. For example, the number of pods allocated may be determined based on the number of users supported by the service, the duration for which the service is requested, etc. The allocated pods may then be customized to suit the specific request-issuing customer to provide the requested service.

[0162] In a specific example, the setup phase processing described above may be performed by the cloud infrastructure system 802 as part of the provisioning process. The cloud infrastructure system 802 may generate an application ID and select a storage virtual machine for the application from among the storage virtual machines provided by the cloud infrastructure system 802 itself, or from storage virtual machines provided by other systems other than the cloud infrastructure system 802.

[0163] The cloud infrastructure system 802 may send a response or notification 844 to the requesting customer indicating when the requested service will be available. In some cases, information (e.g., a link) that enables the customer to begin using and utilizing the benefits of the requested service may be sent to the customer. In a particular example, when a customer requests a service, the response may include a chatbot system ID generated by the cloud infrastructure system 802 and information identifying the chatbot system selected by the cloud infrastructure system 802 that corresponds to this chatbot system ID.

[0164] The cloud infrastructure system 802 may provide services to multiple customers. For each customer, the cloud infrastructure system 802 is responsible for managing information related to one or more subscription orders received from the customer, maintaining customer data related to these orders, and providing the requested services to the customer. The cloud infrastructure system 802 may also collect usage statistics regarding the customer's use of the subscribed services. For example, statistics may be collected on the amount of storage used, the amount of data transferred, the number of users, and system uptime and system downtime. This usage information may be used to issue invoices to customers. Invoice issuance may occur, for example, on a monthly cycle.

[0165] The cloud infrastructure system 802 may provide services to multiple customers in parallel. The cloud infrastructure system 802 may store information about these customers, which may include proprietary information. In a particular example, the cloud infrastructure system 802 includes an Identity Management Subsystem (IMS) 828 configured to manage customer information and isolate managed information so that information related to one customer is inaccessible to another customer. The IMS 828 may be configured to provide a variety of security-related services, such as identity services (information access management, authentication and authorization services, services for managing customer identity and roles and related functions, etc.).

[0166] Figure 9 shows an example of computer system 900. In some examples, computer system 900 may be used to implement a digital assistant or chatbot system in a distributed environment, as well as any of the various servers and computer systems described above. As shown in Figure 9, computer system 900 includes various subsystems, including processing subsystem 904, and processing subsystem 904 communicates with several other subsystems via the bus subsystem 902. These other subsystems may include the processing acceleration unit 906, the I / O subsystem 908, the storage subsystem 918, and the communication subsystem 924. The storage subsystem 918 may include a non-temporary computer-readable storage medium including a storage medium 922 and system memory 910.

[0167] The bus subsystem 902 provides a mechanism for various components and subsystems of the computer system 900 to communicate with each other as intended. Although the bus subsystem 902 is schematically shown as a single bus, alternative examples of the bus subsystem may utilize multiple buses. The bus subsystem 902 can be one of several types of bus structures, these bus structures include memory buses or memory controllers, peripheral buses, local buses, etc., using one of various bus architectures. For example, such architectures may include industry standard architecture (ISA) buses, microchannel architecture (MCA) buses, enhanced ISA (EISA) buses, video electronics standards association (VESA) local buses, and peripheral component interconnect (PCI) buses, which can be implemented as mezzanine buses manufactured according to the IEEE P1386.1 standard.

[0168] The processing subsystem 904 controls the operation of the computer system 900 and may comprise one or more processors, application-specific integrated circuits (ASICs), or field-programmable gate arrays (FPGAs). These processors may include single-core or multi-core processors. The processing resources of the computer system 900 can be organized into one or more processing units 932, 934, etc. A processing unit may comprise one or more processors, one or more cores from the same or different processors, a combination of cores and processors, or other combinations of cores and processors. In some examples, the processing subsystem 904 may comprise one or more special-purpose coprocessors, such as graphics processors or digital signal processors (DSPs). In some examples, some or all of the processing units of the processing subsystem 904 may be implemented using customized circuits such as application-specific integrated circuits (ASICs) or field-programmable gate arrays (FPGAs).

[0169] In some cases, processing units within the processing subsystem 904 can execute instructions stored in system memory 910 or on computer-readable storage media 922. In various cases, processing units can execute various programs or code instructions and can maintain multiple concurrently running programs or processes. At any given time, some or all of the program code to be executed may reside in system memory 910 and / or on computer-readable storage media 922 (and possibly on one or more storage devices). Through suitable programming, the processing subsystem 904 can provide the various functions described above. In cases where the computer system 900 is running one or more virtual machines, one or more processing units may be assigned to each virtual machine.

[0170] In a particular example, the processing acceleration unit 906 may be optionally configured to perform customized processing to accelerate the entire processing performed by the computer system 900, or to offload a portion of the processing performed by the processing subsystem 904.

[0171] The I / O subsystem 908 inputs information to and / or inputs information from the computer system 900. This may include devices and mechanisms for outputting via the input device. Generally, the use of the term input device is intended to include all possible types of devices and mechanisms for inputting information into the computer system 900. User interface input devices may include, for example, pointing devices such as keyboards, mice or trackballs, touchpads or touchscreens integrated into displays, scroll wheels, click wheels, dials, buttons, switches, keypads, voice input devices with voice command recognition systems, microphones, and other types of input devices. User interface input devices may also include motion sensing and / or gesture recognition devices such as Microsoft Kinect® motion sensors that allow a user to control and interact with the input device, Microsoft Xbox® 360 game controllers, and devices that provide an interface for receiving input using gestures and verbal commands. User interface input devices may also include eye gesture recognition devices, such as the Google Glass® Blink Detector, which detects eye movements from the user (e.g., blinking while taking a picture and / or making a menu selection) and translates the eye gestures into input to an input device (e.g., Google Glass®). Furthermore, user interface input devices may include voice recognition sensing devices that enable the user to interact with a voice recognition system (e.g., Siri® Navigator) via voice commands.

[0172] Other examples of user interface input devices include, but are not limited to, three-dimensional (3D) mice, joysticks or pointing sticks, gamepads and graphic tablets, and audio / visual devices (such as speakers, digital cameras, digital camcorders, portable media players, webcams, image scanners, fingerprint scanners, barcode readers, 3D scanners, 3D printers, laser rangefinders, and eye-tracking devices). Furthermore, user interface input devices may include, for example, medical imaging input devices such as computed tomography, magnetic resonance imaging, positional emission tomography, and medical ultrasound devices. User interface input devices may also include, for example, audio input devices such as MIDI keyboards and digital musical instruments.

[0173] Generally, the use of the term “output device” is intended to include all possible types of devices and mechanisms for outputting information from the computer system 900 to a user or another computer. User interface output devices may include non-visual displays such as display subsystems, indicator lights, or audio output devices. Display subsystems may include flat panel devices such as those using cathode ray tubes (CRTs), liquid crystal displays (LCDs), or plasma displays, projection devices, touchscreens, etc. For example, user interface output devices may include, but are not limited to, a variety of display devices that visually convey text, graphics, and audio / video information, such as monitors, printers, speakers, headphones, car navigation systems, plotters, audio output devices, and modems.

[0174] The storage subsystem 918 provides a repository or datastore for storing information and data used by the computer system 900. The storage subsystem 918 provides a tangible, non-temporary, computer-readable storage medium for storing basic programming and data structures that provide some example functions. The storage subsystem 918 may store software (e.g., programs, code modules, instructions) that, when executed by the processing subsystem 904, provides the above functions. This software is stored in one or more processing units of the processing subsystem 904. This may be performed by the storage subsystem 918. The storage subsystem 918 may also provide the authentication relating to the teachings of this disclosure.

[0175] The storage subsystem 918 may include one or more non-temporary memory devices, including volatile and non-volatile memory devices. As shown in Figure 9, the storage subsystem 918 includes system memory 910 and computer-readable storage medium 922. The system memory 910 may include several memories, which include volatile main random access memory (RAM) for storing instructions and data during program execution, and non-volatile read-only memory (ROM) or flash memory for storing fixed instructions. In some implementations, a basic input / output system (BIOS), which includes basic routines that help transfer information between elements within the computer system 900, such as during startup, may generally be stored in ROM. The RAM generally includes data and / or program modules currently operating and executing by the processing subsystem 904. In some implementations, the system memory 910 may include several different types of memory, such as static random access memory (SRAM) and dynamic random access memory (DRAM).

[0176] As an example, as shown in Figure 9, system memory 910 may, but is not limited to, load running application programs 912 (which may include various applications such as web browsers, middle-tier applications, and relational database management systems (RDBMS)), program data 914, and operating system 916. As an example, operating system 916 may include various versions of Microsoft Windows®, Apple Macintosh®, and / or Linux operating systems, UNIX® or UNIX-like operating systems available in various markets (including, but not limited to, various GNU / Linux operating systems and Google Chrome® OS), and / or mobile operating systems such as iOS, Windows® Phone, Android® OS, BlackBerry® OS, and Palm® OS.

[0177] The computer-readable storage medium 922 may store programming and data structures that provide several example functions. The computer-readable storage medium 922 may provide the computer system 900 with computer-readable instructions, data structures, program modules, and other data storage. Software (programs, code modules, instructions) that provides the above functions when executed by the processing subsystem 904 may be stored in the storage subsystem 918. As an example, the computer-readable storage medium 922 may include non-volatile memory such as hard disk drives, magnetic disk drives, optical disk drives such as CD-ROMs, DVDs, Blu-ray® discs, or other optical media. The computer-readable storage medium 922 may include, but is not limited to, Zip® drives, flash memory cards, Universal Serial Bus (USB) flash drives, Secure Digital (SD) cards, DVD discs, digital videotapes, etc. Computer-readable storage media 922 may also include flash memory-based SSDs, enterprise flash drives, solid-state drives (SSDs) based on non-volatile memory such as solid-state ROM, SSDs based on volatile memory such as solid-state RAM, dynamic RAM, and static RAM, DRAM-based SSDs, magnetoresistive RAM (MRAM) SSDs, and hybrid SSDs that use a combination of DRAM-based SSDs and flash memory-based SSDs.

[0178] In certain examples, the storage subsystem 918 may also include a computer-readable storage medium reader 920 that can be further connected to the computer-readable storage medium 922. The reader 920 may be configured to receive and read data from memory devices such as disks and flash drives.

[0179] In certain cases, computer system 900 may support virtualization technologies, including but not limited to the virtualization of processing and memory resources. For example, computer system 900 may provide support for running one or more virtual machines. In certain cases, computer system 900 may run programs such as hypervisors that facilitate the configuration and management of virtual machines. Each virtual machine may have allocated memory, computing resources (e.g., processors, cores), I / O, and networking resources. Each virtual machine generally operates independently of other virtual machines. A virtual machine generally runs its own operating system, which may be identical or different from the operating systems run by other virtual machines run by computer system 900. Thus, in some cases, multiple operating systems may run simultaneously by computer system 900.

[0180] The communication subsystem 924 provides interfaces to other computer systems and networks. It acts as an interface for sending and receiving data between other systems and the computer system 900. For example, the communication subsystem 924 may enable the computer system 900 to establish a communication channel with one or more client devices over the Internet for sending and receiving information. For instance, if the computer system 900 is used to implement the bot system 120 shown in Figure 1, the communication subsystem may be used to communicate with a chatbot system selected to suit the application.

[0181] The communication subsystem 924 may support both wired and / or wireless communication protocols. In certain examples, the communication subsystem 924 may include radio frequency (RF) transceiver components for accessing wireless voice and / or data networks (e.g., using cellular telephone technology, advanced data network technologies such as 3G, 4G, or EDGE (Enhanced Data Communication Speed ​​for Global Evolution), WiFi (IEEE 802.XX family standards), or other mobile communication technologies, or any combination thereof), a global positioning system (GPS) receiver component, and / or other components. In some examples, the communication subsystem 924 may provide a wired network connection (e.g., Ethernet) in addition to, or instead of, a wireless interface.

[0182] The communication subsystem 924 can send and receive data in various formats. In some examples, in addition to other formats, the communication subsystem 924 may receive input communications in the form of structured and / or unstructured data feeds 926, event streams 928, event update information 930, etc. For example, the communication subsystem 924 may be configured to receive (or send) data feeds 926 in real time from users of social media networks and / or other communication services, such as web feeds like Twitter® feeds, Facebook® updates, rich site summary (RSS) feeds, and / or real-time update information from one or more third-party sources.

[0183] In a specific example, the communication subsystem 924 receives data in the form of a continuous data stream. The system may be configured to receive a continuous data stream, which may include an event stream 928 of real-time events and / or event update information 930, which may be continuous or infinite and have no definite end. Examples of applications that generate continuous data include, for example, sensor data applications, financial stock market boards, network performance measurement tools (e.g., network monitoring and traffic management applications), clickstream analysis tools, and automotive traffic monitoring.

[0184] Furthermore, the communication subsystem 924 may be configured to transmit data from the computer system 900 to other computer systems or networks. This data may be transmitted to one or more databases in various different formats, such as structured and / or unstructured data feeds 926, event streams 928, and event update information 930, and these one or more databases may communicate with one or more streaming data source computers coupled to the computer system 900.

[0185] The computer system 900 can be of various types, including handheld portable devices (e.g., iPhone® mobile phones, iPad® computing tablets, PDAs), wearable devices (e.g., Google Glass® head-mounted displays), personal computers, workstations, mainframes, kiosks, server racks, or other data processing systems. Because the nature of computers and networks is constantly changing, the description of the computer system 900 shown in Figure 9 is intended only as an example. Many other configurations are possible, having more or fewer components than the system shown in Figure 9. It should be understood that, based on the disclosures and teachings herein, there are other embodiments and / or methods for realizing various examples.

[0186] While specific examples have been described, various modifications, alterations, alternative configurations, and equivalents are possible. The examples are not limited to operation within a specific data processing environment, but can freely operate within multiple data processing environments. Furthermore, while specific examples have been described using a particular set of transactions and steps, it should be apparent to those skilled in the art that this is not intended to be limiting. Some flowcharts illustrate operations as sequential processes, but many of these operations may be performed in parallel or concurrently. The order of operations may also be rearranged. Processes may have additional steps not included in the diagrams. The various features and aspects of the above examples may be used individually or together.

[0187] Furthermore, while specific examples have been described using particular combinations of hardware and software, it should be recognized that other combinations of hardware and software are also possible. A particular example may be implemented using hardware alone, software alone, or a combination thereof. The various processes described herein may be implemented on the same processor in any combination, or on different processors in any combination.

[0188] A device, system, component, or module is described as being configured to perform a particular operation or function, but such configuration is not programmed to perform an operation, for example, by designing electronic circuits to perform an operation, or by executing computer instructions or code, or a processor or core, that are programmed to execute code or instructions stored in a non-temporary memory medium. This can be achieved by programming possible electronic circuits (such as microprocessors) or by any combination thereof. Processes can communicate using a variety of techniques, including but not limited to conventional techniques for inter-process communication, and different process pairs may use different techniques, or the same process pair may use different techniques at different times.

[0189] Specific details are provided in this disclosure to ensure that the examples are fully understood. However, the examples can be implemented without these specific details. For example, well-known circuits, processes, algorithms, structures, and techniques are shown without unnecessary detail to avoid obscuring the examples. This description is merely illustrative and is not intended to limit the scope, applicability, or configuration of other examples. Rather, the above description of the examples will provide a practical description for implementing various examples for those skilled in the art. Various modifications may be made in terms of the function and arrangement of the elements.

[0190] Therefore, the specification and drawings should be considered illustrative rather than restrictive. However, it will be clear that additions, subtractions, deletions, and other variations and modifications may be made without departing from the broader spirit and scope set forth in the claims. Thus, while specific examples have been described, they are not intended to be restrictive. Various variations and equivalents are within the scope of the claims below.

[0191] While the above specification illustrates aspects of the disclosure with reference to specific examples, those skilled in the art will recognize that the disclosure is not limited thereto. The various features and aspects of the above disclosure may be used individually or together. Furthermore, the examples may be used in many more environments and applications beyond those described herein without departing from the broader spirit and scope of the specification. Accordingly, the specification and drawings should be considered illustrative rather than restrictive.

[0192] In the above description, the methods are presented in a specific order for illustrative purposes. It should be understood that, in alternative examples, these methods may be executed in a different order than described. It should also be understood that the above methods may be executed by hardware components, or embodied in a sequence of machine-executable instructions that can be used to cause a machine, such as a general-purpose or special-purpose processor or logic circuit programmed with instructions, to execute the above methods. These machine-executable instructions may be stored in one or more machine-readable media, such as a CD-ROM or other type of optical disk, a floppy diskette, ROM, RAM, EPROM, EEPROM, magnetic or optical card, flash memory, or other type of machine-readable media suitable for storing electronic instructions. Alternatively, these methods may be executed by a combination of hardware and software.

[0193] When a component is described as being configured to perform a particular operation, such a configuration can be achieved, for example, by designing electronic circuits or other hardware to perform the operation, by programming programmable electronic circuits (e.g., a microprocessor or other suitable electronic circuit) to perform the operation, or by any combination thereof.

[0194] While this specification has described in detail exemplary examples of the present application, it should be understood that the concepts of the present invention may be embodied and used in various ways, and the appended claims are intended to be interpreted to encompass such modifications, except as limited by the prior art.

Claims

1. A method performed by a computer, Steps to access the target domain for speech and chatbots, The steps include generating a sentence embedding for the aforementioned utterance, A step of predicting a second probability of whether an utterance belongs to the target domain of the chatbot, based on the distance or density deviation between the sentence embedding for the utterance and the embedding representation for a cluster among a plurality of clusters of intradomain utterances associated with the target domain of the chatbot. A step of predicting a first probability of whether an utterance belongs to the target domain of the chatbot, based on the similarity or difference between the sentence embedding for the utterance and the embedding representation for a cluster among the clusters of intradomain utterances associated with the target domain of the chatbot, A step of determining the final probability of whether the utterance belongs to the target domain of the chatbot, based on the second and first probabilities, A method comprising the step of classifying the utterance as either within or outside the domain for the chatbot, based on the aforementioned final probability.

2. The method according to claim 1, wherein the embedding representation for a given cluster of the plurality of intradomain utterances associated with the target domain of the chatbot is the average of the sentence embeddings for each intradomain utterance in the given cluster.

3. The method according to claim 1 or 2, wherein the second probability is predicted by inputting the sentence embedding for the utterance and the embedding representation for a given cluster of the plurality of intradomain utterances associated with the target domain of the chatbot into an outlier detection model constructed with a distance or density algorithm for outlier detection.

4. The method according to claim 3, wherein the outlier detection model is configured to calculate the distance or density deviation.

5. The method according to any one of claims 1 to 4, wherein the first probability is predicted by inputting the sentence embedding for the utterance and the embedding representation for a given cluster of the plurality of intradomain utterances associated with the target domain of the chatbot into a distance learning model having trained model parameters configured to provide a probability of whether or not the utterance belongs to the target domain.

6. The method according to claim 5, wherein the distance learning model is configured to calculate the similarity or difference.

7. The method according to any one of claims 1 to 6, wherein the sentence embedding for the utterance is generated using an embedding model that maps natural language elements, including sentences, words, and n-grams, to a sequence of numbers, and each of the natural language elements is represented as a single point in a vector space.

8. A computer program for causing one or more data processors to perform the method described in any one of claims 1 to 7.

9. It is a system, One or more data processors, The system comprises a computer-readable storage medium, the computer-readable storage medium, when executed on the one or more data processors, includes instructions that cause the one or more data processors to perform an action, the action being, Accessing the target domain of speech and chatbots, To generate sentence embeddings for the aforementioned utterances, A second probability of whether an utterance belongs to the target domain of the chatbot is predicted based on the distance or density deviation between the sentence embedding for the utterance and the embedding representation for a cluster among a plurality of clusters of intradomain utterances associated with the target domain of the chatbot, Based on the similarity or difference between the sentence embedding for the utterance and the embedding representation for a cluster among the clusters of intradomain utterances associated with the chatbot's target domain, a first probability is predicted regarding whether the utterance belongs to the chatbot's target domain. Based on the second and first probabilities, the final probability of whether the utterance belongs to the target domain of the chatbot is determined, A system that includes classifying the utterance as either within or outside the domain for the chatbot, based on the aforementioned final probability.

10. The system according to claim 9, wherein the embedding representation for a given cluster of the plurality of intradomain utterances associated with the target domain of the chatbot is the average of the sentence embeddings for each intradomain utterance in the given cluster.

11. The system according to claim 9 or 10, wherein the second probability is predicted by inputting the sentence embedding for the utterance and the embedding representation for a given cluster of the plurality of intradomain utterances associated with the target domain of the chatbot into an outlier detection model constructed with a distance or density algorithm for outlier detection.

12. The system according to claim 11, wherein the outlier detection model is configured to calculate the distance or density deviation.

13. The system according to any one of claims 9 to 12, wherein the first probability is predicted by inputting the sentence embedding for the utterance and the embedding representation for a given cluster of the plurality of intradomain utterances associated with the target domain of the chatbot into a distance learning model having trained model parameters configured to provide a probability of whether or not the utterance belongs to the target domain.

14. The system according to claim 13, wherein the distance learning model is configured to compute the similarity or difference.

Images