Improved Out-of-Domain (OOD) Detection Technology

Abstract

This disclosure relates to techniques for identifying out-of-domain discourse. One particular technique includes receiving discourse and a target domain of a chatbot; generating sentence embeddings for the discourse; obtaining embedding representations for each cluster of in-domain discourse associated with the target domain; using a metric learning model to predict a first probability that the discourse belongs to the target domain based on the similarity or difference between the sentence embeddings and each embedding representation of each cluster; using an outlier detection model to predict a second probability that the discourse belongs to the target domain based on the distance or density deviation between the determined sentence embeddings and the embedding representations of neighboring clusters; evaluating the first probability and the second probability to determine a final probability; and classifying the discourse as in-domain or out-of-domain based on the final probability.

Description

[0001] Priority requirements

[0002] This application is a non-provisional application filed on March 30, 2020, under U.S. Provisional Application No. 63 / 002,139, and claims the benefit and priority of the stated U.S. Provisional Application. The entire contents of the foregoing application are incorporated herein by reference for all purposes. Technical Field

[0003] This disclosure generally relates to chatbot systems, and more specifically, to improved techniques for recognizing out-of-domain (OOD) discourse. Background Technology

[0004] To achieve immediate responses, many users around the world utilize instant messaging or chat platforms. Organizations frequently use these platforms to engage in real-time conversations with customers (or end users). However, hiring service personnel to communicate with customers or end users in real time can be very expensive for organizations. Chatbots or bots have begun to be developed to simulate conversations with end users, especially via the Internet. End users can interact with bots through messaging applications they have already installed and are using. Intelligent bots (typically powered by artificial intelligence (AI)) can communicate more intelligently and context-sensitively in real-time conversations, thus allowing for more natural conversations between bots and end users to improve the conversational experience. Instead of end users learning a fixed set of keywords or commands to know how to respond, intelligent bots can understand the end user's intent based on natural language utterances and respond accordingly.

[0005] However, chatbots are difficult to build because these automated solutions require domain-specific knowledge and the application of certain technologies that may only be within the capabilities of specialized developers. As part of building such a chatbot, developers can begin by understanding the needs of the enterprise and end-users. Developers can then analyze and make decisions related to, for example: selecting the dataset to be used for analysis; preparing the input dataset for analysis (e.g., cleaning, extracting, formatting, and / or transforming data before analysis, performing data feature engineering, etc.); identifying appropriate machine learning (ML) techniques or (multiple) ML models for performing the analysis; and improving the techniques or models to improve results / effects based on feedback. The task of identifying appropriate models may include developing multiple models (possibly in parallel), iteratively testing and experimenting with these models, and then identifying specific models (or multiple models) for use. Furthermore, supervised learning-based solutions typically involve a training phase, followed by an application (i.e., inference) phase, and an iterative cycle between the training and application phases. Developers can be responsible for carefully implementing and monitoring these phases to achieve the optimal solution. For example, in order to train (multiple) ML techniques or (multiple) models, precise training data is needed so that the algorithm can understand and learn certain patterns or features (e.g., for chatbots—intent extraction and careful syntactic analysis are needed, not just raw language processing). These ML techniques or models will use these patterns or features to predict desired outcomes (e.g., inferring intent from utterances). Summary of the Invention

[0006] The techniques disclosed herein generally relate to chatbots. More specifically and non-limitingly, the techniques disclosed herein relate to improved techniques for identifying out-of-order (OOD) utterances. Chatbots (also called bots) are equipped with an OOD detector that uses one or more algorithms to determine whether a utterance provided to the bot is outside the domain of the bot (e.g., a skill bot). When such an OOD utterance is detected, the bot can respond with an appropriate response, such as sending a message to the user that the utterance is not something the bot can handle or dispose of. In some embodiments, various clustering-based algorithms and metric-based algorithms, and combinations thereof, are used for OOD detection.

[0007] In various embodiments, a method is provided, the method comprising: receiving utterances and a target domain of a chatbot; generating sentence embeddings for the utterances; obtaining embedding representations for each cluster of multiple clusters of in-domain utterances associated with the target domain, wherein the embedding representation of each cluster is the average of the sentence embeddings of each in-domain utterance in the cluster; inputting the sentence embeddings of the utterances and the embedding representations of each cluster into a metric learning model having learned model parameters, the metric learning model being configured to provide a first probability about whether the utterance belongs to the target domain; using the metric learning model to determine a similarity or difference between the sentence embeddings of the utterances and each embedding representation of each cluster; and using the metric learning model, based on the determined sentence embeddings of the utterances... The system calculates a first probability that the utterance belongs to the target domain by using similarity or difference between the utterance and each embedding representation of each cluster; inputs the sentence embedding of the utterance and the embedding representation of each cluster into an outlier detection model constructed using a distance or density algorithm for outlier detection; uses the outlier detection model to determine the distance or density deviation between the sentence embedding of the utterance and the embedding representation of neighboring clusters; uses the outlier detection model to predict a second probability that the utterance belongs to the target domain based on the determined distance or density deviation; evaluates the first probability and the second probability to determine a final probability that the utterance belongs to the target domain; and classifies the utterance as in-domain or out-of-domain based on the final probability.

[0008] In some embodiments, obtaining the embedding representation of each cluster includes: obtaining the domain-specific utterance based on the target domain; generating a sentence embedding for each domain-specific utterance; inputting the sentence embedding of each domain-specific utterance into an unsupervised clustering model configured to interpret the domain-specific utterance and identify multiple clusters within the feature space of the domain-specific utterance; using the unsupervised clustering model, classifying the sentence embedding of each domain-specific utterance into one of multiple clusters based on the similarity and difference between the features of the sentence embedding and the features of the sentence embedding within each cluster; calculating the centroid of each of the multiple clusters; and outputting the embedding representation and centroid of each of the multiple clusters.

[0009] In some embodiments, the method further includes: calculating a z-score of the utterance based on the distance or density deviation between the sentence embedding of the utterance and the embedding representation of the neighboring clusters; and determining a second probability regarding whether the utterance belongs to the target domain by applying a sigmoid function to the z-score.

[0010] In some embodiments, the sentence embedding of the utterance is generated using an embedding model that maps natural language elements, including sentences, words, and n-grams, to an array of numbers, wherein each of the natural language elements is represented as a single point in a vector space.

[0011] In some embodiments, determining the similarity or difference between the sentence embedding of the utterance and each embedding representation of each cluster includes: (i) calculating the absolute difference between the sentence embedding of the utterance and each embedding representation of each cluster; (ii) inputting the absolute difference, the sentence embedding of the utterance, and the embedding representation of each cluster into a wide and deep learning network, wherein the wide and deep learning network includes a linear model and a deep neural network; (iii) using the linear model and the absolute difference to predict a broad-based probability regarding whether the utterance belongs to the target domain; and (iv) using the deep neural network, the sentence embedding of the utterance, and the embedding representation of each cluster to determine the similarity or difference between the sentence embedding of the utterance and each embedding representation of each cluster; and predicting the first probability includes: using the final layer of the wide and deep learning network to evaluate the broad probability and the similarity or difference between the sentence embedding of the utterance and each embedding representation of each cluster.

[0012] In some embodiments, the linear model includes multiple model parameters trained using a training dataset; the training dataset includes the absolute difference between sentence embeddings of utterances and each embedding representation of each cluster of intra-domain utterances from multiple domains; during training the linear model using the training dataset, a hypothesis function is used to learn a linear relationship between the sentence embeddings of the utterances and each embedding representation of each cluster; and during the learning of the linear relationship, the multiple model parameters are learned to minimize a loss function.

[0013] In some embodiments, the deep learning network includes multiple model parameters trained using a training dataset; the training dataset includes sentence embeddings of intra-domain utterances from multiple domains; and during training the deep learning network using the training dataset, high-dimensional features of the sentence embeddings of the intra-domain utterances are converted into lower-dimensional vectors, which are then concatenated with the features of the intra-domain utterances and fed into the hidden layers of the deep neural network, the values ​​of the lower-dimensional vectors being randomly initialized and learned together with the multiple model parameters to minimize a loss function.

[0014] In various embodiments, a computer program product is provided, which is tangibly embodied in a non-transitory machine-readable storage medium and includes instructions configured to cause one or more data processors to perform some or all of the methods disclosed herein.

[0015] In various embodiments, a system is provided that includes one or more data processors and a non-transitory computer-readable storage medium containing instructions that, when executed on the one or more data processors, cause the one or more data processors to perform some or all of the methods disclosed herein.

[0016] The techniques described above and below can be implemented in various ways and in various contexts. Several example implementations and contexts are provided with reference to the following figures, as described in more detail below. However, the following implementations and contexts are only a few of many implementations and contexts. Attached Figure Description

[0017] Figure 1 This is a simplified block diagram of a distributed environment incorporating exemplary embodiments.

[0018] Figure 2 This is a simplified block diagram of a computing system for implementing a master robot according to certain embodiments.

[0019] Figure 3 This is a simplified block diagram of a computing system for a skill-implementing robot according to certain embodiments.

[0020] Figure 4 This is a simplified block diagram of a chatbot training and deployment system according to various embodiments.

[0021] Figure 5 The illustration shows an integrated architecture, including a metric learning model and an outlier detection model, for identifying OOD discourse according to various embodiments.

[0022] Figure 6 The illustration shows the process flow for identifying OOD utterances according to various embodiments.

[0023] Figure 7 A simplified diagram of a distributed system for implementing various embodiments is depicted.

[0024] Figure 8 It is a simplified block diagram of one or more components of a system environment according to various embodiments, through which services provided by one or more components of the embodiment system can be provided as cloud services.

[0025] Figure 9 An example computer system that can be used to implement various embodiments is illustrated. Detailed Implementation

[0026] In the following description, specific details are set forth for purposes of explanation in order to provide a thorough understanding of certain embodiments. However, it will be apparent, however, that various embodiments may be practiced without these specific details. The accompanying drawings and description are not intended to be limiting. The word “exemplary” is used herein to mean “serving as an example, instance, or illustration.” Any embodiment or design described herein as “exemplary” is not necessarily to be construed as being more preferred or advantageous than other embodiments or designs.

[0027] introduction

[0028] A digital assistant is an AI-driven interface that helps users complete various tasks in natural language conversations. For each digital assistant, a client can assemble one or more skills. Skills (also described in this document as chatbots, bots, or skill bots) are individual bots focused on specific types of tasks such as tracking inventory, submitting timecards, and creating expense reports. When an end user interacts with a digital assistant, the assistant evaluates the end user's input and routes the conversation to and from the appropriate chatbot. This can be achieved through methods such as... Messenger, Skype Various channels, such as Messenger or Short Message Service (SMS), make digital assistants available to end users. Channels enable chat to be sent back and forth between end users and digital assistants and their chatbots via various messaging platforms. Channels can also support user agent upgrades, event-initiated sessions, and testing.

[0029] Intents allow chatbots to understand what users want them to do. Intents comprise a sequence of typical user requests and statements, also known as utterances (e.g., getting account balance, making a purchase, etc.). As used herein, an utterance or message can refer to a set of words (e.g., one or more sentences) exchanged during a conversation with the chatbot. Intents can be created by providing a name describing a user action (e.g., ordering pizza) and compiling a set of real-life user statements or utterances typically associated with triggering the action. Because the chatbot's cognition stems from these intents, each intent can be created and varied from a robust dataset (one to two dozen utterances), allowing the chatbot to interpret ambiguous user input. A rich set of utterances enables the chatbot to understand what a user wants when it receives messages such as "Forget this order!" or "Cancel delivery!" (meaning the same thing expressed differently). Intents and the utterances that belong to intents together constitute the chatbot's training corpus. By training the model with the corpus, the client essentially turns the model into a reference tool for parsing end-user input into individual intents. Customers can improve the cognitive acuity of chatbots through multiple rounds of intent testing and intent training.

[0030] However, building chatbots that can determine the end-user's intent based on user utterances is a challenging task, partly due to the subtlety and ambiguity of natural language, as well as the dimensionality of the input space (e.g., possible user utterances) and the size of the output space (the number of intents). Illustrative examples of this difficulty arise from the characteristics of natural language, such as the use of euphemisms, synonyms, or ungrammatical expressions to convey intent. For example, an utterance might express the intent to order pizza without explicitly mentioning pizza, ordering, or delivery. For instance, in some local dialects, "pizza" is referred to as "pie." These tendencies in natural language (such as imprecision or variability) introduce uncertainty and introduce confidence as a parameter for intent prediction, rather than explicitly indicating the intent by including keywords, for example. Thus, improving chatbot performance and the user experience may require training, monitoring, debugging, and retraining the chatbot. In traditional systems, training systems are provided for training and retraining machine learning models of digital assistants or chatbots in Spoken Language Understanding (SLU) and Natural Language Processing (NLP). Traditionally, models used in chatbot systems are trained in NLP using utterances "manufactured" for any intent. For example, the utterance "Do you do price changes?" can be used to train a classifier model for a chatbot system to categorize this type of utterance as the intent—"Do you offer a price match?" Using manufactured utterances to train the model helps in initially training the chatbot system to provide a service, and then once the chatbot system is deployed and begins to receive real utterances from users, it can be retrained.

[0031] The typical training of a text classification model begins with a training utterance dataset labeled with a list of predefined intents (or categories or classes). For example, a banking chatbot can be trained using predefined intents such as "open an account," "check balance," "close an account," and "transfer money." These intents are generally considered to belong to the same domain that the chatbot can handle (e.g., the banking domain). Typically, chatbot training is performed using training data that includes multiple utterance examples and the intent associated with each utterance. Once trained, the chatbot can receive new utterances (e.g., in a real-world or production environment) and infer the intent of each utterance from the predefined intents.

[0032] However, the utterances chatbots receive from real-world users in real-world environments (e.g., production environments) can be diverse and noisy. Some of these received utterances may be very different from those used to train the chatbot and may not fall within the intent domain the chatbot was trained to infer and handle. For example, a banking chatbot might receive utterances unrelated to banking, such as "How do I book a trip to Italy?" Such utterances are called out-of-domain (OOD) utterances because they are outside the intent domain trained on the chatbot. It is crucial that the chatbot system can recognize such OOD utterances so that it can take appropriate action. For example, upon detecting an OOD utterance, the chatbot could respond to the user by indicating that the utterance is not something the bot can handle or deal with, rather than selecting the closest matching intent.

[0033] Therefore, different approaches are needed to address these problems. This disclosure describes various embodiments for addressing these problems by identifying out-of-domain utterances. In various embodiments, a combination of clustering-based techniques and metric-based techniques is used for OOD determination. An exemplary technique includes receiving utterances and a target domain of a chatbot; generating sentence embeddings for the utterances; obtaining embedding representations of each cluster of in-domain utterances associated with the target domain; predicting a first probability that the utterance belongs to the target domain using a metric learning model based on the similarity or difference between the sentence embeddings and each embedding representation of each cluster; predicting a second probability that the utterance belongs to the target domain using an outlier detection model based on the distance or density deviation between the determined sentence embeddings and the embedding representations of neighboring clusters; evaluating the first probability and the second probability to determine a final probability that the utterance belongs to the target domain; and classifying the utterance as in-domain or out-of-domain based on the final probability.

[0034] In some embodiments, a method is provided, the method comprising: receiving utterances and a target domain of a chatbot; generating sentence embeddings for the utterances; obtaining embedding representations for each cluster of multiple clusters of intra-domain utterances associated with the target domain, wherein the embedding representation of each cluster is the average of the sentence embeddings of each intra-domain utterance in the cluster; inputting the sentence embeddings of the utterances and the embedding representations of each cluster into a metric learning model having learned model parameters, the metric learning model being configured to provide a first probability about whether the utterance belongs to the target domain; using the metric learning model to determine a similarity or difference between the sentence embeddings of the utterances and each embedding representation of each cluster; and using the metric learning model, based on the determined sentence embeddings of the utterances... The system calculates a first probability that the utterance belongs to the target domain by using similarity or difference between the utterance and each embedding representation of each cluster; inputs the sentence embedding of the utterance and the embedding representation of each cluster into an outlier detection model constructed using a distance or density algorithm for outlier detection; uses the outlier detection model to determine the distance or density deviation between the sentence embedding of the utterance and the embedding representation of neighboring clusters; uses the outlier detection model to predict a second probability that the utterance belongs to the target domain based on the determined distance or density deviation; evaluates the first probability and the second probability to determine a final probability that the utterance belongs to the target domain; and classifies the utterance as in-domain or out-of-domain based on the final probability.

[0035] Robots and Analysis Systems

[0036] A bot (also known as a skill, chatbot, conversational bot, or talkative bot) is a computer program capable of engaging in conversation with an end user. Bots typically respond to natural language messages (e.g., questions or comments) via a messaging application that uses natural language messaging. Businesses can use one or more bot systems to communicate with end users through messaging applications. The messaging application (which may be referred to as a channel) can be a preferred messaging application that the end user has already installed and is familiar with. Therefore, the end user does not need to download and install a new application to chat with the bot system. Messaging applications can include, for example, over-the-top (OTT) messaging channels (such as Facebook Messenger, Facebook WhatsApp, WeChat, Line, Kik, Telegram, Talk, Skype, Slack, or SMS), virtual personal assistants (such as Amazon Dot, Echo, or Show, Google Home, Apple HomePod, etc.), native or hybrid extended mobile and web application extensions / responsive mobile or web applications with chat functionality, or voice-based input (such as devices or applications with Siri, Cortana, Google Voice, or other voice input for interaction).

[0037] In some examples, a robot system may be associated with a Uniform Resource Identifier (URI). A URI can identify a robot system using a string of characters. A URI can be used as a webhook for one or more messaging application systems. A URI may include, for example, a Uniform Resource Locator (URL) or a Uniform Resource Name (URN). The robot system may be designed to receive messages from a messaging application system (e.g., a Hypertext Transfer Protocol (HTTP) POST request message). An HTTP POST request message may involve a URI from a messaging application system. In some embodiments, a message may be different from an HTTP POST request message. For example, the robot system may receive messages from a Short Message Service (SMS). While the discussion herein may refer to communications received by the robot system as messages, it should be understood that a message can be an HTTP POST request message, an SMS message, or any other type of communication between two systems.

[0038] End users can interact with robot systems through conversational interactions (sometimes called conversational user interface (UI)), much like human interactions. In some cases, the interaction may include the end user saying "Hello" to the robot and the robot responding with "Hi" and asking how the robot can help. In other cases, the interaction may be transactional with, for example, a banking robot, such as transferring money from one account to another; informational with, for example, an HR robot, such as checking holiday balances; or with, for example, a retail robot, such as discussing returning purchased goods or seeking technical support.

[0039] In some embodiments, a robot system can intelligently handle end-user interactions without interaction with the robot system's administrator or developer. For example, an end-user can send one or more messages to the robot system to achieve a desired goal. Messages may include some form of content, such as text, emojis, audio, images, video, or other methods of conveying the message. In some embodiments, the robot system can convert the content into a standardized form (e.g., using appropriate parameters for a Representational State Transition (REST) ​​call to an enterprise service) and generate a natural language response. The robot system may also prompt the end-user for additional input parameters or request additional information. In some embodiments, the robot system may also initiate communication with the end-user rather than passively responding to end-user utterances. Various techniques are described herein for identifying explicit calls to a robot system and determining the input of the called robot system. In some embodiments, explicit call analysis is performed by a master robot based on the detected call name in the utterance. In response to the detection of the call name, the utterance can be refined for input to the skill robot associated with the call name.

[0040] A conversation with a chatbot can follow a specific conversation flow that includes multiple states. This flow can be defined based on input to determine what will happen next. In some embodiments, the chatbot system can be implemented using a state machine that includes user-defined states (e.g., end-user intentions) and actions to be taken within or between states. The conversation can take different paths based on end-user input, which can affect the chatbot's decisions regarding the flow. For example, in each state, based on end-user input or utterances, the chatbot can determine the end-user's intention in order to determine the next appropriate action to take. As used herein and in the context of utterances, the term "intention" refers to the intention of the user providing the utterance. For example, a user might intend to involve the chatbot in a conversation about ordering pizza, so the user's intention can be expressed through the utterance "Order pizza." User intentions can relate to specific tasks that the user wants the chatbot to perform on their behalf. Therefore, utterances can be expressed as questions, commands, requests, etc., that reflect the user's intention. Intentions can include goals that the end-user wants to accomplish.

[0041] In the context of chatbot configuration, the term "intent" as used herein refers to configuration information used to map a user's utterances to a specific task / action or a specific type of task / action that the chatbot can perform. To distinguish the intent of the utterances (i.e., the user's intent) from the intent of the chatbot, the latter is sometimes referred to as "bot intent." A bot intent can include one or more utterances associated with the intent. For example, the intent to order pizza can have various permutations of utterances expressing the expectation of placing an order for pizza. These associated utterances can be used to train the chatbot's intent classifier so that the intent classifier can subsequently determine whether an input utterance from the user matches the pizza-ordering intent. A bot intent can be associated with one or more conversational flows used to initiate a conversation with the user in a certain state. For example, the first message for the pizza-ordering intent could be the question "What kind of pizza would you like?". In addition to associated utterances, bot intents can further include named entities related to the intent. For example, the pizza-ordering intent can include variables or parameters for performing the task of ordering pizza, such as topping 1, topping 2, pizza type, pizza size, number of pizzas, etc. The values ​​of the entities are typically obtained through conversation with the user.

[0042] Figure 1This is a simplified block diagram of an environment 100 incorporating a chatbot system according to certain embodiments. Environment 100 includes a Digital Assistant Builder Platform (DABP) 102, which enables users of DABP 102 to create and deploy digital assistant or chatbot systems. DABP 102 can be used to create one or more digital assistant (or DA) or chatbot systems. For example, as... Figure 1 As shown, a user 104 representing a specific business can use DABP 102 to create and deploy a digital assistant 106 for that business's users. For example, DABP 102 can be used by a bank to create one or more digital assistants for its customers. Multiple businesses can use the same DABP 102 platform to create digital assistants. As another example, a restaurant (e.g., a pizzeria) owner can use DABP 102 to create and deploy a digital assistant that enables the restaurant's customers to order food (e.g., order pizza).

[0043] For the purposes of this disclosure, a "digital assistant" is an entity that assists a user in performing various tasks through natural language conversation. A digital assistant can be implemented using software alone (e.g., a digital assistant is a digital entity implemented using programs, code, or instructions executable by one or more processors), hardware, or a combination of hardware and software. Digital assistants can be embodied or implemented in various physical systems or devices such as computers, mobile phones, watches, appliances, and vehicles. Digital assistants are sometimes also referred to as chatbot systems. Therefore, for the purposes of this disclosure, the terms digital assistant and chatbot system are used interchangeably.

[0044] A digital assistant (such as digital assistant 106 built using DABP 102) can be used to perform various tasks via natural language-based conversations between the digital assistant and its user 108. As part of the conversation, the user can provide one or more user inputs 110 to the digital assistant 106 and receive a response 112 from the digital assistant 106. The conversation may include one or more of the inputs 110 and the response 112. Through these conversations, the user can request one or more tasks to be performed by the digital assistant, and in response, the digital assistant is configured to perform the task requested by the user and respond to the user with an appropriate response.

[0045] User input 110 is typically in the form of natural language and is referred to as speech. User speech 110 can be in text form, such as when a user types a sentence, question, text fragment, or even a single word and provides the text as input to the digital assistant 106. In some embodiments, user speech 110 can be in the form of audio input or speech, such as when a user speaks or utters something as input to the digital assistant 106. Speech is typically in the form of the language spoken by user 108. For example, speech can be English or some other language. When speech is in speech form, the speech input is converted into speech in text form in that particular language, and then the text speech is processed by the digital assistant 106. Various speech-to-text processing techniques can be used to convert speech or audio input into text speech, which is then processed by the digital assistant 106. In some embodiments, the speech-to-text conversion can be performed by the digital assistant 106 itself.

[0046] Speech (which can be text or speech) can be fragments, a sentence, multiple sentences, one or more words, one or more questions, combinations of the above types, etc. Digital Assistant 106 is configured to apply Natural Language Understanding (NLU) technology to the speech to understand the meaning of user input. As part of the NLU processing for the speech, Digital Assistant 106 is configured to perform processing for understanding the meaning of the speech, which involves identifying one or more intentions and one or more entities corresponding to the speech. After understanding the meaning of the speech, Digital Assistant 106 can perform one or more actions or operations in response to the understood meaning or intention. For the purposes of this disclosure, it is assumed that these speeches are text speeches directly provided by the user 108 of Digital Assistant 106, or the result of converting input speech speeches into text form. However, this is not intended to limit or constrain in any way.

[0047] For example, user 108 can request to order a pizza by providing a statement such as "I want to order a pizza." Upon receiving such a statement, digital assistant 106 is configured to understand the meaning of the statement and take an appropriate action. The appropriate action may involve responding to the user's input, for example, by requesting information about the type of pizza the user wishes to order, its size, and any toppings. The responses provided by digital assistant 106 can also be in natural language and are typically in the same language as the input statement. As part of generating these responses, digital assistant 106 can perform natural language generation (NLG). To enable the user to order a pizza via a conversation between the user and digital assistant 106, the digital assistant can guide the user to provide all the necessary information for the pizza order and then allow the pizza to be ordered at the end of the conversation. Digital assistant 106 can end the conversation by outputting information to the user indicating that the pizza has been ordered.

[0048] At the conceptual level, the digital assistant 106 performs various processes in response to utterances received from a user. In some embodiments, this processing involves a series of processing steps or a pipeline of processing steps, including, for example, understanding the meaning of the input utterance (sometimes referred to as Natural Language Understanding (NLU), determining the action to be performed in response to the utterance, performing the action where appropriate, generating a response to be output to the user in response to the user's utterance, outputting the response to the user, etc. NLU processing may include performing syntactic analysis on the received input utterance to understand the structure and meaning of the utterance, refining and reformulating the utterance to develop a better comprehensible form (e.g., logical form) or structure of the utterance. Generating a response may include using NLG techniques.

[0049] The NLU processing performed by the digital assistant (such as digital assistant 106) may include various NLP-related processes such as sentence parsing (e.g., tokenization, classification by inflectional form, identification of part-of-speech tags, identification of named entities in sentences, generation of dependency trees to represent sentence structure, segmentation of sentences into clauses, analysis of individual clauses, resolution of pronouns, execution of chunks, etc.). In some embodiments, the NLU processing, or portions thereof, is performed by digital assistant 106 itself. In some other embodiments, digital assistant 106 may use other resources to perform portions of the NLU processing. For example, the syntax and structure of the input utterance sentence can be identified by processing the sentence using a parser, part-of-speech tagger, and / or named entity recognizer. In one implementation, for the English language, a parser, part-of-speech tagger, and named entity recognizer provided by the Stanford Natural Language Processing (NLP) Group are used to analyze sentence structure and syntax. These are provided as part of the Stanford CoreNLP toolkit.

[0050] While the various examples provided in this disclosure illustrate utterances in the English language, this is merely illustrative. In some embodiments, the digital assistant 106 is also capable of processing utterances in languages ​​other than English. The digital assistant 106 may provide subsystems (e.g., components implementing NLU functionality) configured to perform processing for different languages. These subsystems may be implemented as pluggable units that can be invoked from the NLU core server using service calls. This makes NLU processing flexible and scalable for each language, including allowing different processing sequences. Language packs may be provided for individual languages, wherein the language packs may register a list of subsystems that can be served from the NLU core server.

[0051] Digital assistants (such as...) can be enabled through various channels (such as, but not limited to, certain applications, social media platforms, various messaging services and applications, and other applications or channels) Figure 1 The digital assistant 106 depicted is available or accessible to its user 108. A single digital assistant can be configured with several channels, allowing it to run on different services simultaneously and access them through different services.

[0052] Digital assistant or chatbot systems typically incorporate or are associated with one or more skills. In some embodiments, these skills are individual chatbots (called skill-based bots) configured to interact with users and perform specific types of tasks, such as tracking inventory, submitting time cards, creating expense reports, ordering food, checking bank accounts, making appointments, purchasing widgets, etc. For example, for... Figure 1 In the depicted embodiments, the digital assistant or chatbot system 106 includes skills 116-1, 116-2, etc. For the purposes of this disclosure, the terms "skill" and "skills" are used synonymously with the terms "skill bot" and "skill bots," respectively.

[0053] Each skill associated with the digital assistant helps the user complete tasks through a conversation with the user, wherein the conversation may include a combination of text or audio input provided by the user and responses provided by the skill bot. These responses may take the form of text or audio messages to the user and / or simple user interface elements (e.g., a selection list) presented to the user for selection.

[0054] There are various methods to associate or add skills or skill bots to digital assistants. In some instances, a business can develop a skill bot and then add it to a digital assistant using DABP 102. In other instances, a skill bot can be developed and created using DABP 102 and then added to a digital assistant created using DABP 102. In yet another instance, DABP 102 provides an online digital store (called the "Skill Store") that offers multiple skills covering a wide variety of tasks. Skills offered through the Skill Store can also be exposed to various cloud services. To add a skill to a digital assistant created using DABP 102, a DABP 102 user can access the Skill Store via DABP 102, select the desired skill, and instruct the selected skill to be added to the digital assistant created using DABP 102. Skills from the Skills Store can be added to the digital assistant as is or with modifications (for example, a DABP 102 user can select and copy a specific skill bot from the Skills Store, customize or modify the selected skill bot, and then add the modified skill bot to the digital assistant created using the DABP 102).

[0055] Various architectures can be used to implement digital assistant or chatbot systems. For example, in some embodiments, a digital assistant created and deployed using DABP 102 can be implemented using a master bot / secondary (or sub)bot paradigm or architecture. According to this paradigm, the digital assistant is implemented as a master bot that interacts with one or more secondary bots that are skill-based bots. For example, in... Figure 1 In the depicted embodiments, digital assistant 106 includes a master robot 114 and skill robots 116-1, 116-2, etc., which are secondary robots of the master robot 114. In some embodiments, digital assistant 106 itself is considered to act as the master robot.

[0056] Digital assistants implemented using a master-secondary robot architecture enable users to interact with multiple skills through a unified user interface (i.e., via the master robot). When a user engages with the digital assistant, the master robot receives the user input. The master robot then performs processing to determine the meaning of the user's input. It then determines whether the task requested in the user's utterance can be handled by the master robot itself; otherwise, it selects the appropriate skill robot to handle the user's request and routes the session to the selected skill robot. This allows users to interact with the digital assistant through a common, single interface while still having the ability to use several skill robots configured to perform specific tasks. For example, for a digital assistant developed for an enterprise, the master robot can interface with skill robots with specific functionalities, such as a CRM robot for performing functions related to Customer Relationship Management (CRM), an ERP robot for performing functions related to Enterprise Resource Planning (ERP), an HCM robot for performing functions related to Human Capital Management (HCM), etc. In this way, the end user or consumer of the digital assistant only needs to know how to access the digital assistant through the common master robot interface, while multiple skill robots are available in the background to handle user requests.

[0057] In some embodiments, in a master robot / sub-robot infrastructure, a master robot is configured to know a list of available skill robots. The master robot can access metadata identifying the various available skill robots, and for each skill robot, the skill robot's capabilities include tasks that can be performed by the skill robot. Upon receiving a user request in the form of a utterance, the master robot is configured to identify or predict from among the multiple available skill robots that a particular skill robot can best serve or handle the user request. The master robot then routes the utterance (or a portion of the utterance) to that particular skill robot for further handling. Thus, control flows from the master robot to the skill robot. The master robot may support multiple input and output channels. In some embodiments, routing may be performed by means of processing performed by one or more available skill robots. For example, as discussed below, skill robots can be trained to infer the intent of a utterance and determine whether the inferred intent matches an intent configured for the skill robot. Therefore, routing performed by the master robot may involve the skill robot transmitting an instruction to the master robot indicating whether the skill robot is already configured with an intent suitable for handling the utterance.

[0058] Although Figure 1The embodiments shown illustrate a digital assistant 106 including a main robot 114 and skill robots 116-1, 116-2, and 116-3, but this is not intended to be limiting. A digital assistant may include various other components (e.g., other systems and subsystems) that provide the functionality of the digital assistant. These systems and subsystems may be implemented in software only (e.g., code, instructions stored on a computer-readable medium and executable by one or more processors), in hardware only, or in implementations using a combination of software and hardware.

[0059] The DABP 102 provides the infrastructure, services, and features that enable DABP 102 users to create digital assistants (including one or more skill bots associated with the digital assistant). In some instances, skill bots can be created by cloning existing skill bots, for example, cloning skill bots provided by a skill store. As mentioned earlier, the DABP 102 provides a skill store or skill catalog that offers multiple skill bots for performing various tasks. DABP 102 users can clone skill bots from the skill store. The cloned skill bot can be modified or customized as needed. In some other instances, DABP 102 users create skill bots from scratch using the tools and services provided by the DABP 102. As mentioned earlier, the skill store or skill catalog provided by the DABP 102 offers multiple skill bots for performing various tasks.

[0060] In some embodiments, creating or customizing a skill robot at a high level involves the following steps:

[0061] (1) Configure settings for new skill robots

[0062] (2) Configure one or more intentions for the skill robot

[0063] (3) Configure one or more entities for one or more intents.

[0064] (4) Training Skills Robot

[0065] (5) Create a dialogue flow for the skill robot

[0066] (6) Add custom parts to the skill robot as needed.

[0067] (7) Test and deploy skill robots

[0068] Each of the above steps is briefly described below.

[0069] (1) Configure settings for new skill robots—Various settings can be configured for skill robots. For example, skill robot designers can specify one or more invocation names for the skill robot being created. These invocation names can then be used by the digital assistant user to explicitly invoke the skill robot. For example, the user can enter the invocation name in their speech to explicitly invoke the corresponding skill robot.

[0070] (2) Configuring one or more intents and associated example utterances for the skill robot—The skill robot designer assigns one or more intents (also called robot intents) to the skill robot being created. The skill robot is then trained based on these assigned intents. These intents represent categories or classifications that the skill robot is trained to infer in response to input utterances. Upon receiving an utterance, the trained skill robot infers the intent of the utterance, where the inferred intent is selected from a predefined set of intents used to train the skill robot. The skill robot then takes an appropriate action to respond to the utterance based on the intent inferred for that utterance. In some instances, the intents of the skill robot represent tasks that the skill robot can perform for a digital assistant user. Each intent is assigned an intent identifier or intent name. For example, for a skill robot trained for a bank, the intents assigned to the skill robot could include “CheckBalance”, “TransferMoney”, “DepositCheck”, etc.

[0071] For each intent defined for a skill bot, the skill bot designer can also provide one or more example utterances that represent and illustrate the intent. These example utterances are designed to represent the words a user would input to the skill bot in response to that intent. For example, for the CheckBalance intent, example utterances could include “What's my savings account balance?”, “How much is in my checking account?”, “How much money do I have in my account?”, etc. Therefore, various permutations of typical user utterances can be designated as example utterances for the intent.

[0072] These intentions and their associated example utterances are used as training data for training a skill-based robot. Various training techniques can be used. As a result of this training, a predictive model is generated, configured to take utterances as input and output the intentions inferred by the predictive model in response to those utterances. In some instances, the input utterances are provided to an intention analysis engine, which is configured to use the trained model to predict or infer the intentions of the input utterances. The skill-based robot can then take one or more actions based on the inferred intentions.

[0073] (3) Configuring Entities for One or More Intents of a Skill Robot—In some instances, additional context may be needed for the skill robot to respond appropriately to user utterances. For example, there may be cases where user input utterances are parsed into the same intent in the skill robot. For example, in the example above, the utterances “What's my savings account balance?” and “How much is in my checking account?” are both parsed into the same CheckBalance intent, but these utterances are different requests for different things. To clarify such requests, one or more entities are added to the intent. Using the example of a banking skill robot, an entity called AccountType (which defines values ​​called “checking” and “saving”) enables the skill robot to perform syntactic analysis on user requests and respond appropriately. In the example above, although these utterances are parsed into the same intent, the values ​​associated with the AccountType entity for the two utterances are different. This allows the skill robot to perform potentially different actions for the two utterances, even though they are parsed into the same intent. One or more entities can be specified for certain intents configured for the skill robot. Thus, entities are used to add context to the intent itself. Entities help to more fully describe intent and enable skill-based robots to fulfill user requests.

[0074] In some embodiments, there are two types of entities: (a) built-in entities provided by DABP 102; and (2) custom entities that can be specified by the skill bot designer. Built-in entities are general-purpose entities that can be used with various bots. Examples of built-in entities include, but are not limited to, entities relating to time, date, address, number, email address, duration, cycle time, currency, phone number, URL, etc. Custom entities are used for more customized applications. For example, for banking skills, an account type entity can be defined by the skill bot designer to enable various banking transactions by examining keywords entered by the user (such as current account, savings, credit card, etc.).

[0075] (4) Training the Skill Robot—The skill robot is configured to receive user input in utterance form, process the received input through syntactic analysis or otherwise, and identify or select an intent associated with the received user input. As indicated above, this requires training the skill robot. In some embodiments, the skill robot is trained based on intents configured for the skill robot and example utterances associated with those intents (collectively, training data), such that the skill robot can parse user input utterances into one of its configured intents. In some embodiments, the skill robot uses a predictive model trained with the training data and allows the skill robot to discern what the user says (or, in some cases, is attempting to say). DABP 102 provides a variety of different training techniques that skill robot designers can use to train skill robots, including various machine learning-based training techniques, rule-based training techniques, and / or combinations thereof. In some embodiments, a portion of the training data (e.g., 80%) is used to train the skill robot model, and another portion (e.g., the remaining 20%) is used to test or validate the model. Once trained, the trained model (sometimes referred to as the trained skill robot) can be used to process and respond to the user's utterances. In some cases, a user's utterance may be a question that requires only a single answer and no further conversation. To handle this, a Q&A (Question and Answer) intent can be defined for the skill bot. This allows the skill bot to output a response to the user's request without having to update the dialog definition. Q&A intents are created in a similar manner to regular intents. The dialogue flow used for Q&A intents may differ from that used for regular intents.

[0076] (5) Creating a Dialogue Flow for the Skilled Robot—The dialogue flow specified for the skilled robot describes how it reacts as it parses different intentions in response to received user input. The dialogue flow defines the actions or behaviors the skilled robot will take, such as how it responds to user utterances, prompts for user input, and returns data. The dialogue flow is like the process followed by the skilled robot. Figure 1 Skill bot designers specify the dialogue flow using languages ​​such as Markdown. In some embodiments, a version of YAML called OBotML can be used to specify the dialogue flow for the skill bot. The dialogue flow definition for the skill bot acts as a model of the session itself, a model that allows the skill bot designer to carefully orchestrate the interactions between the skill bot and the users it serves.

[0077] In some embodiments, the dialogue flow definition of a skill robot includes the following three parts:

[0078] (a) Context section

[0079] (b) Default Transformation Section

[0080] (c) State section

[0081] The context section—skill bot designers can define variables used in the session flow in the context section. Other variables that can be named in the context section include, but are not limited to: variables for error handling, variables for built-in or custom entities, and user variables that enable the skill bot to recognize and save user preferences.

[0082] The Default Transition section—transitions for the skill bot can be defined either in the dialog flow state section or in the Default Transition section. Transitions defined in the Default Transition section act as fallbacks and are triggered when no applicable transition is defined within the state or when the conditions required to trigger a state transition cannot be met. The Default Transition section can be used to define routes that allow the skill bot to handle unexpected user actions appropriately.

[0083] The state component—the dialogue flow and its associated operations—is defined as a sequence of temporary states that manage the logic within the dialogue flow. Each state node within the dialogue flow definition is named a component, which provides the functionality required at that point in the dialogue. Therefore, states are built around components. States contain component-specific properties and define transitions to other states that are triggered after a component's execution.

[0084] Special scenarios can be handled using the state section. For example, you might sometimes want to provide users with the option to temporarily engage with a first skill while performing actions within a second skill within the digital assistant. For instance, if a user is busy conversing with the shopping skill (e.g., the user has made some purchase choices), the user might want to jump to the banking skill (e.g., the user might want to ensure he / she has enough money for the purchase) and then return to the shopping skill to complete the order. To address this, the action in the first skill can be configured to initiate an interaction with a second, different skill within the same digital assistant and then return to the original stream.

[0085] (6) Adding Custom Parts to the Skill Robot—As described above, the states specified in the skill robot's dialogue flow are named with parts that provide the required functionality corresponding to those states. Parts enable the skill robot to perform functions. In some embodiments, DABP 102 provides a set of pre-configured parts for performing a wide variety of functions. The skill robot designer can select one or more of these pre-configured parts and associate them with states in the skill robot's dialogue flow. The skill robot designer can also use the tools provided by DABP 102 to create custom parts or new parts and associate custom parts with one or more states in the skill robot's dialogue flow.

[0086] (7) Testing and Deploying Skilled Robots – DABP 102 provides several features that enable skilled robot designers to test the skilled robots they are developing. The skilled robots can then be deployed and included in digital assistants.

[0087] While the above description outlines how to create skill bots, similar techniques can be used to create digital assistants (or master bots). At the master bot or digital assistant level, built-in system intents can be configured for the digital assistant. These built-in system intents are used to identify general tasks that the digital assistant (i.e., the master bot) can handle without invoking the skill bots associated with it. Examples of system intents defined for the master bot include: (1) Exit: applicable when a user signals that they want to exit the current session or context within the digital assistant; (2) Help: applicable when a user requests help or direction; and (3) UnresolvedIntent: applicable to user input that doesn't quite match the Exit and Help intents. The digital assistant also stores information about one or more skill bots associated with it. This information enables the master bot to select the specific skill bot to handle utterances.

[0088] At the master robot or digital assistant level, when a user inputs a phrase or utterance into the digital assistant, the digital assistant is configured to perform processing to determine how to route the utterance and related conversation. The digital assistant uses a routing model to determine this, which can be rule-based, AI-based, or a combination thereof. The digital assistant uses the routing model to determine whether the conversation corresponding to the user's input utterance should be routed to a specific skill for handling, handled by the digital assistant or master robot itself according to built-in system intent, or handled as a different state within the current conversation flow.

[0089] In some embodiments, as part of this process, the digital assistant determines whether the user input utterance explicitly identifies a skill bot using its invocation name. If the invocation name is present in the user input, the invocation name is considered an explicit invocation of the skill bot corresponding to the invocation name. In this scenario, the digital assistant can route the user input to the explicitly invoked skill bot for further processing. In some embodiments, if no specific invocation or explicit invocation exists, the digital assistant evaluates the received user input utterance and calculates a confidence score for the system intent and skill bot associated with the digital assistant. The score calculated for the skill bot or system intent indicates how likely the user input represents a task that the skill bot is configured to perform or represents a system intent. Any system intent or skill bot whose associated calculated confidence score exceeds a threshold (e.g., a confidence threshold routing parameter) is selected as a candidate for further evaluation. The digital assistant then selects a specific system intent or skill bot from the identified candidates for further processing of the user input utterance. In some embodiments, after one or more skill bots are identified as candidates, the intents associated with those candidate skills are evaluated (according to the intent model for each skill) and a confidence score is determined for each intent. Any intent with a confidence score exceeding a threshold (e.g., 70%) is generally considered a candidate intent. If a specific skill bot is selected, the user's utterance is routed to that skill bot for further processing. If a system intent is selected, the main bot itself performs one or more actions based on the selected system intent.

[0090] Figure 2 This is a simplified block diagram of a master robot (MB) system 200 according to certain embodiments. The MB system 200 may be implemented in software only, in hardware only, or in a combination of hardware and software. The MB system 200 includes a preprocessing subsystem 210, multiple intention subsystems (MIS) 220, an explicit invocation subsystem (EIS) 230, a skill robot invoker 240, and a data storage 250. Figure 2 The depicted MB system 200 is merely an example of component arrangement in the main robot. Those skilled in the art will recognize many possible variations, alternatives, and modifications. For example, in some embodiments, the MB system 200 may have a more... Figure 2 The systems or components shown may have more or fewer subsystems, may combine two or more subsystems, or may have different subsystem configurations or arrangements.

[0091] The preprocessing subsystem 210 receives the utterance “A” 202 from the user and processes it through a language detector 212 and a language parser 214. As described above, the utterance can be provided in various ways, including audio or text. The utterance 202 can be a sentence fragment, a complete sentence, multiple sentences, etc. The utterance 202 can include punctuation marks. For example, if the utterance 202 is provided as audio, the preprocessing subsystem 210 can use a speech-to-text converter (not shown) that inserts punctuation marks (e.g., commas, semicolons, periods, etc.) into the resulting text to convert the audio into text.

[0092] The language detector 212 detects the language of discourse 202 based on the text of discourse 202. The processing of discourse 202 depends on the language, because each language has its own grammar and semantics. Differences between languages ​​are considered when analyzing the syntax and structure of discourse.

[0093] Language parser 214 performs syntactic analysis on discourse 202 to extract part-of-speech (POS) tags for individual language units (e.g., words) within discourse 202. POS tags include, for example, nouns (NN), pronouns (PN), verbs (VB), etc. Language parser 214 can also tokenize the language units of discourse 202 (e.g., convert each word into a separate token) and categorize words according to inflectional forms. A lexical is the principal form of a group of words as represented in a dictionary (e.g., "run" is a lexical for run, runs, ran, running, etc.). Other types of preprocessing that language parser 214 can perform include chunking compound expressions, for example, combining "credit" and "card" into a single expression "credit card". Language parser 214 can also identify relationships between words in discourse 202. For example, in some embodiments, the language parser 214 generates a dependency tree that indicates which part of the utterance (e.g., a specific noun) is a direct object, which part of the utterance is a preposition, etc. The result of the processing performed by the language parser 214 forms extracted information 205, which, together with the utterance 202 itself, is provided as input to the MIS 220.

[0094] As described above, utterance 202 may include more than one sentence. For the purpose of detecting multiple intents and explicit calls, utterance 202 may be treated as a single unit, even if it comprises multiple sentences. However, in some embodiments, preprocessing may be performed, for example, by preprocessing subsystem 210, to identify individual sentences among multiple sentences for use in multiple intent analysis and explicit call analysis. Generally, whether utterance 202 is processed at the level of a single sentence or as a single unit comprising multiple sentences, the results produced by MIS 220 and EIS 230 are substantially the same.

[0095] MIS 220 determines whether utterance 202 represents multiple intentions. Although MIS 220 can detect the presence of multiple intentions in utterance 202, the processing performed by MIS 220 does not involve determining whether the intention of utterance 202 matches any intention already configured for the robot. Instead, the processing to determine whether the intention of utterance 202 matches a robot intention can be performed by the intention classifier 242 of MB system 200 or (e.g., as...) Figure 3 The intent classifier of the skill-based chatbot (as shown in the embodiment) is executed. The processing performed by MIS 220 assumes the existence of a bot that can handle utterance 202 (e.g., a specific skill-based bot or the main bot itself). Therefore, the processing performed by MIS 220 does not need to know which bots are in the chatbot system (e.g., the identities of skill-based bots registered with the main bot) or what intents have been configured for a specific bot.

[0096] To determine that utterance 202 includes multiple intentions, MIS 220 applies one or more rules from a set of rules 252 in data storage 250. The rules applied to utterance 202 depend on the language of utterance 202 and may include sentence patterns indicating the presence of multiple intentions. For example, a sentence pattern may include co-conjunctions connecting two parts of a sentence (e.g., conjunctions) that correspond to different intentions. If utterance 202 matches a sentence pattern, it can be inferred that utterance 202 represents multiple intentions. It should be noted that utterances with multiple intentions do not necessarily have different intentions (e.g., intentions involving different robots or different intentions within the same robot). Instead, utterances can have different instances of the same intention, such as “Place a pizza order using payment account X, then place a pizza order using payment account Y”.

[0097] As part of determining that discourse 202 represents multiple intentions, MIS 220 also determines which parts of discourse 202 are associated with each intention. MIS 220 constructs a new discourse for each intention represented in a discourse containing multiple intentions, replacing the original discourse for individual processing, for example, such as... Figure 2The original utterance 202 can be divided into two or more separate utterances, processed one at a time, as depicted in the diagram. The MIS 220 uses the extracted information 205 and / or analysis of the utterance 202 itself to determine which of the two or more utterances should be processed first. For example, the MIS 220 may determine that utterance 202 contains markers indicating a specific intention that should be processed first. The newly formed utterance corresponding to that specific intention (e.g., one of utterances 206 or 208) will be sent first for further processing by the EIS 230. After the session triggered by the first utterance has ended (or been temporarily suspended), the next highest priority utterance (e.g., another utterance of utterances 206 or 208) can then be sent to the EIS 230 for processing.

[0098] EIS 230 determines whether a received utterance (e.g., utterance 206 or utterance 208) contains a call name for a skill bot. In some embodiments, each skill bot in the chatbot system is assigned a unique call name that distinguishes it from other skill bots in the chatbot system. A list of call names may be stored in data storage 250 as part of skill bot information 254. When a utterance contains a word that matches a call name, the utterance is considered an explicit call. If a bot is not explicitly called, the utterance received by EIS 230 is considered a non-explicit call utterance 234 and is fed into the main bot's intent classifier (e.g., intent classifier 242) to determine which bot to use to handle the utterance. In some instances, intent classifier 242 will determine that the main bot should handle the non-explicit call utterance. In other instances, intent classifier 242 will determine which skill bot to route the utterance to for handling.

[0099] The explicit invocation functionality provided by EIS 230 offers several advantages. It reduces the amount of processing the master robot must perform. For example, when explicit invocation is present, the master robot may not need to perform any intent classification analysis (e.g., using intent classifier 242), or may need to perform a simplified intent classification analysis to select the skill robot. Therefore, explicit invocation analysis can achieve the selection of a specific skill robot without relying on intent classification analysis.

[0100] Furthermore, there may be functional overlap between multiple skill robots. This can occur, for example, if two skill robots handle overlapping or very similar intentions. In such cases, the master robot may struggle to identify which of the multiple skill robots to select based solely on intention classification analysis. In this scenario, explicit invocation ensures unambiguity regarding the specific skill robot to be used.

[0101] In addition to determining whether a utterance is an explicit invocation, EIS 230 is also responsible for determining whether any part of the utterance should be used as input to the explicitly invoked skill robot. Specifically, EIS 230 can determine whether a portion of the utterance is irrelevant to the invocation. EIS 230 can perform this determination by analyzing the utterance and / or analyzing the extracted information 205. EIS 230 may send the irrelevant portion of the utterance to the invoked skill robot instead of sending the entire utterance received by EIS 230. In some instances, the input to the invoked skill robot is simply formed by removing any utterance portion associated with the invocation. For example, “I want to order pizza using Pizza Bot” can be shortened to “I want to order pizza” because “using Pizza Bot” is associated with the invocation of the Pizza Bot but not with any processing to be performed by the Pizza Bot. In some instances, EIS 230 can reformat the portion to be sent to the invoked robot, for example, to form a complete sentence. Therefore, EIS 230 not only determines that an explicit call exists, but also, when an explicit call exists, determines what to send to the skill robot. In some instances, there may be no text to be input into the called robot. For example, if the utterance is "Pizza Bot," EIS 230 can determine that the pizza robot is being called, but there is no text to be processed by the pizza robot. In such a scenario, EIS 230 can indicate to the skill robot caller 240 that there is no content to send.

[0102] Skill robot invoker 240 invokes skill robots in various ways. For example, skill robot invoker 240 may invoke a robot in response to receiving an instruction 235 indicating that a specific skill robot has been selected as a result of an explicit invocation. Instruction 235 may be sent by EIS 230 along with the input of the explicitly invoked skill robot. In this scenario, skill robot invoker 240 relinquishes control of the session to the explicitly invoked skill robot. The explicitly invoked skill robot will determine the appropriate response to the input from EIS 230 by treating the input as an independent utterance. For example, the response may be to perform a specific action or to start a new session in a specific state, where the initial state of the new session depends on the input sent from EIS 230.

[0103] Another way the skill robot invoker 240 can invoke a skill robot is through implicit invocation using an intent classifier 242. The intent classifier 242 can be trained using machine learning and / or rule-based training techniques to determine the likelihood that a utterance represents a task that a particular skill robot is configured to perform. The intent classifier 242 is trained on different categories, one category per skill robot. For example, whenever a new skill robot is registered with the master robot, a list of example utterances associated with the new skill robot can be used to train the intent classifier 242 to determine the likelihood that a particular utterance represents a task that the new skill robot can perform. The parameters produced as a result of this training (e.g., a set of parameter values ​​for a machine learning model) can be stored as part of the skill robot information 254.

[0104] In some embodiments, the intent classifier 242 is implemented using a machine learning model, as described in further detail herein. Training the machine learning model may involve taking input from at least a subset of example utterances associated with various skill robots to generate an inference as the output of the machine learning model about which robot is the correct robot for disposing of any particular training utterance. For each training utterance, an indication of the correct robot for the training utterance can be provided as base fact information. The behavior of the machine learning model can then be adapted (e.g., via backpropagation) to minimize the discrepancy between the generated inference and the base fact information.

[0105] In some embodiments, the intent classifier 242 determines a confidence score for each skill robot registered with the master robot, indicating the likelihood that the skill robot can handle a utterance (e.g., a non-explicit invocation utterance 234 received from EIS 230). The intent classifier 242 may also determine a confidence score for each configured system-level intent (e.g., help, exit). If a particular confidence score meets one or more conditions, the skill robot invoker 240 invokes the robot associated with that particular confidence score. For example, a threshold confidence score value may need to be met. Therefore, the output 245 of the intent classifier 242 is an identification of a system intent or an identification of a particular skill robot. In some embodiments, in addition to meeting a threshold confidence score value, the confidence score must exceed the next highest confidence score by a certain margin. Imposing such a condition when the confidence scores of multiple skill robots all exceed the threshold confidence score value will enable routing to a particular skill robot.

[0106] After identifying a robot based on a confidence score assessment, the skill robot invoker 240 hands over processing to the identified robot. In the case of system intent, the identified robot is the master robot. Otherwise, the identified robot is a skill robot. Further, the skill robot invoker 240 determines what to provide as input 247 to the identified robot. As described above, in the case of explicit invoking, input 247 can be based on a utterance portion unrelated to the invoking, or input 247 can be nothing (e.g., an empty string). In the case of implicit invoking, input 247 can be the entire utterance.

[0107] Data storage 250 includes one or more computing devices that store data used by various subsystems of the main robot system 200. As explained above, data storage 250 includes rules 252 and skill robot information 254. Rules 252 include, for example, rules for determining, by MIS 220, when a utterance represents multiple intents and how to segment the utterance representing multiple intents. Rules 252 further include rules for determining, by EIS 230, which parts of the utterance explicitly invoking the skill robot should be sent to the skill robot. Skill robot information 254 includes the invoking names of skill robots in the chatbot system, for example, a list of invoking names of all skill robots registered with a particular main robot. Skill robot information 254 may also include information used by intent classifier 242 to determine the confidence score of each skill robot in the chatbot system, such as parameters of a machine learning model.

[0108] Figure 3 This is a simplified block diagram of a skill robot system 300 according to certain embodiments. The skill robot system 300 is a computing system that can be implemented solely in software, solely in hardware, or a combination of hardware and software. In some embodiments, such as Figure 1 In the depicted embodiments, the skill robot system 300 can be used to implement one or more skill robots within a digital assistant.

[0109] The skill robot system 300 includes a MIS 310, an intent classifier 320, and a session manager 330. The MIS 310 is similar to... Figure 2 MIS 220 provides similar functionality, including operablely using rules 352 in data storage 350 to determine: (1) whether a utterance represents multiple intents, and if so, (2) how to split the utterance into separate utterances for each of the multiple intents. In some embodiments, the rules applied by MIS 310 for detecting multiple intents and for segmenting utterances are the same as those applied by MIS 220. MIS 310 receives utterance 302 and extracted information 304. The extracted information 304 is similar to... Figure 1The information extracted from 205 can be generated using a language parser 214 or a language parser native to the skill robot system 300.

[0110] The intent classifier 320 can be combined with the above. Figure 2 The intent classifier 242 discussed in the embodiments is trained in a similar manner and is described in further detail herein. For example, in some embodiments, the intent classifier 320 is implemented using a machine learning model. For a specific skill robot, the machine learning model of the intent classifier 320 is trained using at least a subset of example utterances associated with that specific skill robot as training utterances. The underlying fact of each training utterance will be the specific robot intent associated with the training utterance.

[0111] Phras 302 can be received directly from the user or provided through the main bot. When phrasing 302 is provided through the main bot, for example, as via... Figure 2 The results processed by MIS 220 and EIS 230 in the embodiments depicted can bypass MIS 310 to avoid repeating the processing already performed by MIS 220. However, if utterance 302 is received directly from the user, for example, during a session after routing to a skill bot, MIS 310 can process utterance 302 to determine whether utterance 302 represents multiple intents. If so, MIS 310 applies one or more rules to split utterance 302 into separate utterances for each intent, such as utterance “D” 306 and utterance “E” 308. If utterance 302 does not represent multiple intents, MIS 310 forwards utterance 302 to intent classifier 320 for intent classification without splitting utterance 302.

[0112] Intent classifier 320 is configured to match received utterances (e.g., utterances 306 or 308) with intents associated with the skill robot system 300. As explained above, the skill robot may be configured with one or more intents, each intent including at least one example utterance associated with that intent and used to train the classifier. Figure 2In one embodiment, the intent classifier 242 of the main robot system 200 is trained to determine confidence scores for each skill robot and confidence scores for system intents. Similarly, an intent classifier 320 can be trained to determine confidence scores for each intent associated with the skill robot system 300. The classification performed by the intent classifier 242 is at the robot level, while the classification performed by the intent classifier 320 is at the intent level and therefore more granular. The intent classifier 320 can access intent information 354. For each intent associated with the skill robot system 300, the intent information 354 includes a list of utterances that represent the intent and explain its meaning, and are generally associated with a task that can be performed by that intent. The intent information 354 may further include parameters generated as a result of training on this list of utterances.

[0113] The session manager 330 receives a specific intent indication 322 as the output of the intent classifier 320, which identifies the indication as the best match for the utterance input to the intent classifier 320. In some instances, the intent classifier 320 may not determine any match. For example, if the utterance relates to a system intent or the intent of a different skill robot, the confidence score calculated by the intent classifier 320 may be below a threshold confidence score value. When this occurs, the skill robot system 300 may submit the utterance to the master robot for processing, e.g., to route it to a different skill robot. However, if the intent classifier 320 successfully identifies an intent within a skill robot, the session manager 330 will initiate a session with the user.

[0114] A session initiated by session manager 330 is a session specific to an intent identified by intent classifier 320. For example, session manager 330 may be implemented using a state machine configured to perform a dialogue flow in response to an identified intent. The state machine may include a default starting state (e.g., when an intent is invoked without any additional input) and one or more additional states, each associated with an action to be performed by the skill robot (e.g., performing a purchase transaction) and / or a dialogue to be presented to the user (e.g., a question, a response). Thus, session manager 330 may determine an action / dialogue 335 upon receiving an indication 322 that an intent has been identified, and may determine additional actions or dialogues in response to subsequent utterances received during the session.

[0115] Data storage 350 includes one or more computing devices that store data used by various subsystems of the skill robot system 300. For example... Figure 3 As depicted, data storage 350 may include rules 352 and intent information 354. In some embodiments, data storage 350 may be integrated into the data storage of the main robot or digital assistant, for example... Figure 2The data storage in the middle is 250.

[0116] OOD detection system and architecture

[0117] When a chatbot receives a utterance, it must accurately determine whether the utterance is in-domain or out-of-domain. It has been found that models used to classify utterances as intentions can be overconfident and provide poor results on text with out-of-domain intent. To overcome this problem, various embodiments involve techniques that use clustering-based and metric-based methods to compute probabilities about whether a utterance belongs to a target domain (e.g., a given skill bot). The probabilities computed from the clustering-based and metric-based methods are then combined in an ensemble method to obtain the best result from both approaches. The ensemble method ultimately classifies the utterance as in-domain or out-of-domain based on the final combined probabilities.

[0118] Figure 4 The illustration shows a configuration for training and utilizing a classifier based on text data 405 (e.g., about...). Figure 2 and Figure 3 A block diagram of aspects of a chatbot system 400 (describing an intent classifier 242 or 320). Figure 4 As shown, the text classification performed by the chatbot system 400 in this example includes the following phases: a prediction model training phase 410; a skill bot invocation phase 415 for determining the probability that a utterance represents a task (e.g., in-domain or out-of-domain) that a particular skill bot is configured to perform; and an intent prediction phase 420 for classifying the utterance into one or more intents. The prediction model training phase 410 establishes and trains one or more prediction models 425a-425n ('n' represents any natural number) for use in the other phases (the prediction models may be individually referred to as prediction model 425 or collectively as prediction model 425 herein). For example, prediction model 425 may include one or more models (or an ensemble of models) for determining the probability that a utterance represents a task that a particular skill bot is configured to perform (e.g., calculating the probability of whether the utterance belongs to a target domain), another model for predicting intents based on utterances from a first type of skill bot, and another model for predicting intents based on utterances from a second type of skill bot. Other types of prediction models may be implemented in other examples according to this disclosure.

[0119] Predictive model 425 may be a machine learning (“ML”) model, such as a convolutional neural network (“CNN”), for example, an initial neural network, a residual neural network (“ResNet”); or a recurrent neural network, such as a long short-term memory (“LSTM”) model or a gated recurrent unit (“GRU”) model; or other variants of deep neural networks (“DNN”) (e.g., a stacked highway network, a wide and deep learning network with a linear model and a deep neural network, a multi-label n-class DNN classifier for single-intent classification, or a multi-class DNN classifier). Predictive model 425 may also be any other suitable ML model trained for natural language processing, such as a Naive Bayes classifier, a linear classifier, a support vector machine, a bagging model such as a random forest model, a boosting model, a shallow neural network, or a combination of one or more of these techniques—for example, CNN-HMM or MCNN (multi-scale convolutional neural network). Chatbot system 400 can employ the same type of prediction model or different types of prediction models to determine the probability that utterances represent a task configured to be performed by a robot with a specific skill level, predicting intent based on utterances from a first-type skill robot, and predicting intent based on utterances from a second-type skill robot. Other types of prediction models can be implemented in other examples according to this disclosure.

[0120] To train various prediction models 425, the training phase 410 consists of three main parts: dataset preparation 430, feature engineering 435, and model training 440. Dataset preparation 430 includes loading data assets 445, splitting the data assets 445 into training and validation sets 445a-n to enable the system to train and test the prediction models 425, and performing basic preprocessing. Data assets 445 may include at least a subset of utterances from example utterances associated with various skill-based chatbots. As mentioned above, utterances can be provided in various ways, including audio or text. utterances can be sentence fragments, complete sentences, multiple sentences, etc. For example, if utterances are provided as audio, data preparation 430 can use a speech-to-text converter (not shown) to insert punctuation marks (e.g., commas, semicolons, periods, etc.) into the resulting text to convert the audio to text. In some instances, example utterances are provided by the client or customer. In other instances, example utterances are automatically generated from a previous utterance library (e.g., utterances specific to the skills the chatbot is to learn are identified from the library). The data assets 445 of the prediction model 425 may include input text or audio (or input features of text or audio frames) and labels 450 corresponding to the input text or audio (or input features) as matrices or value tables. For example, for each training utterance, the correct robot instruction for the training utterance can be provided as basic fact information for the label 450. The behavior of the prediction model 425 can then be adapted (e.g., through backpropagation) to minimize the discrepancy between the generated inference and the basic fact information. Alternatively, for a specific skill robot, the prediction model 425 can be trained using at least a subset of example utterances associated with that specific skill robot as training utterances. The basic fact information for the label 450 of each training utterance will be the specific robot intent associated with the training utterance.

[0121] In various embodiments, data preparation 430 includes OOD data augmentation 455, with data assets 445 including OOD utterance examples from various contexts to make the predictive model 425 more resilient to OOD utterances. By augmenting data assets 445 with OOD examples from various contexts, the predictive model 425 becomes better refined on the most important parts of the examples and contexts to correlate them with their classifications, including OOD classifications. Augmentation 455 can be implemented using OOD augmentation techniques to combine OOD utterances from various contexts with the original utterances of data assets 445. OOD augmentation techniques typically include four operations: (i) generating a dataset containing multiple OOD examples, (ii) filtering out OOD examples whose contexts are too similar to the original utterances, and (iii) feeding OOD examples into the model in batches during training to balance OOD examples and in-domain examples, since the number of OOD examples can be much larger than that of in-domain utterances, and wherein the batch feeding starts with batches containing simpler OOD examples and continues with batches containing more difficult OOD examples.

[0122] In some instances, additional augmentations can be applied to data assets 445 (using OOD augmentation). For example, simple data augmentation (EDA) techniques can be used to improve the performance of text classification tasks. EDA includes four operations: synonym replacement, ransom insertion, random swapping, and random deletion, which prevent overfitting and help train more robust models. Note that, compared to OOD augmentation, EDA operations typically: (i) obtain words from the original text, and (ii) incorporate these words into each data asset 445 relative to the original text. For example, the synonym replacement operation involves randomly selecting n non-stop words from the original sentence (e.g., utterance) and replacing each of these words with one of its randomly selected synonyms. The random insertion operation involves—n times—finding a random synonym of a random word that is not a stop word in the original sentence and inserting that synonym at a random position in the sentence. The random swapping operation involves—n times—randomly selecting two words in the sentence and swapping their positions. The random deletion operation involves randomly deleting each word in the sentence with probability p.

[0123] In various embodiments, feature engineering 435 includes transforming data asset 445 into feature vectors and / or creating new features that will be created using data asset 445, using an encoding model such as a Multilingual Universal Sentence Encoder (MUSE). An encoding model is a model capable of mapping natural language elements such as sentences, words, and n-grams (a set of n letters / words) to arrays of numbers. In this way, each natural language element can be represented as a single point in a vector space. The aim is to obtain representations of sentences, words, and n-grams that can be used by computing devices for data processing without losing too much information. Feature vectors can include count vectors as features, term frequency-inverse document frequency (TF-IDF) vectors as features (e.g., word-level, n-gram-level, or character-level), word embeddings as features, text / NLP as features, topic models as features, or combinations thereof. Count vectors are matrix notations of data asset 445, where each row represents a utterance, each column represents a word from the utterance, and each cell represents the frequency count of a particular word in the utterance. The TF-IDF score represents the relative importance of a word in the utterance. Word embeddings are a form of representation that uses dense vector representations to represent words and utterances. The positions of words in the vector space are learned from the text and are based on the words surrounding that word when it is used. Text-based / NLP features can include the number of words in the utterance, the number of characters in the utterance, the average word density, the number of punctuation marks, the number of uppercase letters, the number of title words, the frequency distribution of part-of-speech tags (e.g., nouns and verbs), or any combination thereof. Topic modeling is a technique for identifying phrases (called topics) from a set of utterances containing the best information from the set.

[0124] In various embodiments, model training 440 includes training a predictive model 425 using sentence embeddings and feature vectors and / or new features created in feature engineering 435. In some instances, the training process includes iterative operations to find a set of parameters for the predictive model 425 that minimizes the loss or error function of the predictive model 425. Each iteration may involve finding a set of parameters for the predictive model 425 such that the value of the loss or error function using that set of parameters is less than the value of the loss or error function using another set of parameters in a previous iteration. The loss or error function can be constructed to measure the difference between the output predicted using the predictive model 425 and the labels 450 contained in the data asset 445. Once this set of parameters is identified, the predictive model 425 is trained and can be used for prediction as designed.

[0125] In addition to data assets 445, labels 450, feature vectors, and / or new features, other techniques and information can be employed to refine the training process of the predictive model 425. For example, feature vectors and / or new features can be combined to help improve the accuracy of the classifier or model. Alternatively or additionally, hyperparameters can be tuned or optimized; for example, multiple parameters such as tree length, leaves, and network parameters can be fine-tuned to obtain the best-fit model. Although the training mechanisms described in this paper primarily focus on training the predictive model 425, these training mechanisms can also be used to fine-tune existing predictive models 425 trained from other data assets. For example, in some cases, the predictive model 425 may have been pre-trained using utterances specific to another skill-based robot. In these cases, the predictive model 425 can be retrained using data assets 445 (e.g., using OOD augmentation).

[0126] The prediction model training phase 410 outputs a trained prediction model 425, which includes a task prediction model 460 and an intent prediction model 465. The task prediction model 460 can be used in the skill robot invocation phase 415 to determine the probability that a utterance represents a particular skill robot configured to perform a task 470, and the intent prediction model 465 can be used in the intent prediction phase 420 to classify the utterance into one or more intents 475. In some instances, the skill robot invocation phase 415 and the intent prediction phase 420 can be performed independently using separate models. For example, the trained intent prediction model 465 can be used in the intent prediction phase 420 to predict the intent of a skill robot without first identifying the skill robot in the skill robot invocation phase 415. Similarly, the task prediction model 460 can be used in the skill robot invocation phase 415 to predict the task or skill robot to be used for a utterance without identifying the intent of the utterance in the intent prediction phase 420.

[0127] Alternatively, the skill robot invocation phase 415 and the intent prediction phase 420 can proceed sequentially, with one phase using the output of the other as input, or, for a specific skill robot, one phase being invoked in a specific manner based on the output of the other. For example, given text data 405, the skill robot invoker can invoke the skill robot implicitly using the skill robot invocation phase 415 and the task prediction model 460. The task prediction model 460 can be trained using machine learning and / or rule-based training techniques to determine the probability that a utterance represents a task configured to be performed by a specific skill robot 470. Then, for the identified or invoked skill robot and the given text data 405, the intent prediction phase 420 and the intent prediction model 465 can be used to match received utterances (e.g., utterances within a given data asset 445) with intents 475 associated with the skill robot. As explained herein, a skill robot can be configured with one or more intents, each intent including at least one example utterance associated with said intent and used to train a classifier. In some embodiments, the skill robot invocation phase 415 and the task prediction model 460 used in the main robot system are trained to determine the confidence scores of each skill robot and the system intent. Similarly, the intent prediction phase 420 and the intent prediction model 465 can be trained to determine the confidence score of each intent associated with the skill robot system. The classification performed by the skill robot invocation phase 415 and the task prediction model 460 is at the robot level, while the classification performed by the intent prediction phase 420 and the intent prediction model 465 is at the intent level and is therefore more granular.

[0128] Figure 5 A block diagram illustrating various aspects of model architecture 500 is shown, which provides clustering-based and metric-based methods to calculate the probability of whether a discourse belongs to a target domain (e.g., regarding...). Figure 4The described skill robot invocation phase 415). Model architecture 500 includes a clustering component 505, a classification component 510, and an ensemble component 515. Clustering component 505 includes two phases: (i) an unsupervised clustering model 520 and (ii) an outlier detection model 525. Unsupervised clustering model 520 is shared between clustering-based and metric-based methods for determining clusters 530 within domain data 535, calculating centroids 540, and generating embedding representations 545 for said clusters. Outlier detection model 525 is constructed using distance or density algorithms (e.g., Z-score, K-means, DBSCAN, Local Outlier Detection (LOF), Isolation Forest, etc.) to provide a probability 550 (e.g., a second probability) about whether an input utterance 555 belongs to the target domain. Classification component 510 includes two phases: (i) an unsupervised clustering model 520 and (ii) a metric learning model 560. A metric learning model 560 calculates the absolute difference 565 between the sentence embedding 570 of the input utterance 555 and the clustered embedding representation 545, and is constructed by a deep learning network 575 with learned model parameters, the deep learning network being configured to provide a probability 580 (e.g., a first probability) regarding whether the input utterance 555 belongs to the target domain. An integration component 515 is configured to evaluate probabilities 580 and 550 to determine a final probability 585 regarding whether the input utterance 555 belongs to the target domain, and classifies the input utterance 555 as within or outside the chatbot's domain based on the final probability 585.

[0129] Regarding the clustering-based method implemented by clustering component 505, the intra-domain data 535 used to train the unsupervised clustering algorithm includes intra-domain utterances associated with a specific domain or skill robot (e.g., pizza-only ordering training data). An embedding model 590 (e.g., MUSE) can be used to generate sentence embeddings for each intra-domain utterance by mapping natural language elements, including sentences, words, and n-grams, to arrays of numbers. Each natural language element is represented as a single point in a vector space. Therefore, each sentence embedding is a vector representing the values ​​of the natural language element. Unsupervised clustering algorithms (e.g., K-means, affine propagation, agglomerative clustering, equilibrium iterative reduction and clustering (BIRCH), DBSCAN, mean shift, ordering points to identify cluster structures (OPTICS), etc.) take data points (i.e., sentence embeddings for each intra-domain utterance) as input and group them into clusters. This grouping process is the training phase of the unsupervised clustering algorithm. The result will be an unsupervised clustering model 520 that takes a data sample (e.g., sentence embeddings of s new in-domain utterances) as input and returns the cluster to which the new data point belongs based on the training experienced by the unsupervised clustering model 520. Once clusters are determined for the in-domain data 535, an embedding representation 545 is generated for each cluster. The embedding representation 545 is the average of the sentence embeddings of each in-domain utterance in the cluster. The clustering process reduces the in-domain data 535 to embedding representations 545 of a more manageable size for the clusters (e.g., equal to or less than 1000 embedding representations, equal to or less than 500 embedding representations, or equal to or less than 250 embedding representations).

[0130] The outlier detection model 525 optionally includes an unsupervised clustering algorithm (e.g., K-means, affine propagation, agglomerative clustering, BIRCH, DBSCAN, mean shift, OPTICS, etc.) that takes data points (i.e., the embedding representations 545 and centroid calculations of clusters determined by centroid calculation 535 and cluster detection 530) as input and further groups them into refined clusters. This grouping process is the training phase of the unsupervised clustering algorithm. The result will be an unsupervised clustering model that takes data samples (e.g., the embedding representations and centroid calculations of clusters) as input and returns the refined clusters to which new data points belong based on the training undergone by the unsupervised clustering model. Once refined clusters are determined for the embedding representations 545, a refined embedding representation is generated for each refined cluster. The refined embedding representation is the average of the embedding representations 545 for each refined cluster. An outlier detection model 525 includes a distance or density algorithm (e.g., Z-score, K-means, DBSCAN, Local Outlier Detection (LOF), Isolated Forest, etc.) configured to determine the distance or density deviation between the sentence embedding 570 of the input utterance 555 and the embedding representations (or modified embedding representations) of neighboring clusters. The outlier detection model 525 predicts a probability 525 regarding whether the input utterance 555 belongs to the target domain based on the determined distance or density deviation. For example, the outlier detection model 525 may consider an input utterance 555 that has a large distance from any neighbor or a density significantly lower than any neighbor as an outlier, which can then be used to provide a probability 525 regarding whether the input utterance 555 belongs to the target domain.

[0131] Regarding the metric-based method implemented by the classification component 510, a training dataset can be used to train the deep learning network 575, which includes (i) sentence embeddings of the input utterances, (ii) embedding representations of each cluster consisting of sentence embeddings from the intra-domain data, and (iii) the absolute difference between the sentence embeddings of the input utterances and each embedding representation of each cluster consisting of sentence embeddings from the intra-domain data. The intra-domain data 535 used to train the deep learning network 575 includes intra-domain utterances associated with various domains or skill robots (e.g., not only pizza ordering training data, but also training data from any other available domain or robot, such as payroll robots, weather robots, bank account robots, etc.).

[0132] In some embodiments, the deep learning network 575 is a stacked highway network with a nonlinear transformation as part of a gating function. The model parameters of the stacked highway network can be learned using a training dataset. During the training of the metric learning model 560 using the training dataset, the high-dimensional features of the sentence embeddings and the embedding representations of each cluster are transformed into lower-dimensional vectors, which are then concatenated with the features of the in-domain utterances and fed into the hidden layers of the deep neural network. The values ​​of the lower-dimensional vectors are randomly initialized and learned along with the model parameters to minimize the loss function.

[0133] Once trained, the stacked highway network can determine the similarity or difference between the sentence embedding 570 of the input utterance 555 and each embedding representation 545 of each cluster. An embedding model 595 (e.g., MUSE) can be used to generate sentence embeddings 570 for the input utterance 555 by mapping natural language elements, including sentences, words, and n-grams, to arrays of numbers. Each natural language element is represented as a single point in a vector space. Therefore, a sentence embedding is a vector representing the values ​​of a natural language element.

[0134] A stacked highway network can be described as follows:

[0135] ·Probability(SAME_DOMAIN or NOT_SAME_DOMAIN|x, target_domain)~Probability(SAME_DOMAIN or NOT_SAME_DOMAIN|x, centroid(target_domain))

[0136] ·Probability(SAME_DOMAIN or NOT_SAME_DOMAIN|x, centroid(target_domain))=softmax(Φ(x, centroid(target_domain)))

[0137] ·Φ(x, centroid(target_domain))=highway_networkΩ(W*f(x, centroid(target_domain))+b), where W and b are the learnable weights and biases, respectively; Ω refers to the learnable model parameters of f(x, centroid(target_domain))=concatenation(MUSE(x);

[0138] MUSE(centroid(target_domain));|MUSE(x)-

[0139] ·MUSE(centroid(target_domain))|;MUSE(x)*MUSE(centroid(target_domain))). In this instance, MUSE is used to form a representation of (u, v, |uv|, u*v) for pairwise ranking.

[0140] The similarity or difference between the sentence embedding 570 of the input utterance 555 and each embedding representation 545 of each cluster can be determined by: (i) calculating the absolute difference 565 between the sentence embedding of the utterance and each embedding representation of each cluster; (ii) inputting the absolute difference 565, the sentence embedding 570 of the input utterance, and the embedding representation 545 of each cluster into a stacked highway network; and (iii) using the stacked highway network, the absolute difference 565, the sentence embedding 570 of the input utterance 555, and the embedding representation 545 of each cluster to determine the similarity or difference between the sentence embedding 570 of the input utterance 555 and each embedding representation 545 of each cluster. Figure 5 As shown, the absolute difference between the sentence embedding 570 of the input utterance 555 and each embedding representation 545 of each cluster is calculated by taking the absolute value of the difference between the vector value of the sentence embedding 570 of the input utterance 555 (e.g., V = [0.1, 0.4, -0.5]) and the vector value of each embedding representation 545 of each cluster composed of the sentence embeddings of the domain data 535 (e.g., U = [0.3, 0.1, 0.4]) (|UV| = [0.2, 0.3, 0.9]|). The probability 580 of whether the input utterance 555 belongs to the target domain can be predicted by the stacked highway network based on the similarity or difference between the determined sentence embedding 570 of the input utterance 555 and each embedding representation 545 of each cluster.

[0141] In other embodiments, the deep learning network 575 is a wide and deep learning network with a linear model and a deep neural network. The linear model includes model parameters trained using a training dataset. The training dataset includes the absolute difference between sentence embeddings of utterances and each embedding representation of each cluster of intra-domain utterances from multiple domains. During training the linear model using the training dataset, a hypothesis function is used to learn a linear relationship between the sentence embeddings of utterances and each embedding representation of each cluster. During learning the linear relationship, multiple model parameters are learned to minimize a loss function. The deep learning network includes model parameters trained using a training dataset. The training dataset includes sentence embeddings of intra-domain utterances from multiple domains. During training the deep learning network using the training dataset, high-dimensional features of the sentence embeddings of intra-domain utterances are transformed into lower-dimensional vectors, which are then concatenated with the features of the intra-domain utterances and fed into the hidden layers of the deep neural network. The values ​​of the lower-dimensional vectors are randomly initialized and learned along with multiple model parameters to minimize the loss function.

[0142] Once trained, a wide and deep learning network can determine the similarity or difference between the sentence embedding 570 of the input utterance 555 and each embedding representation 545 of each cluster. An embedding model 595 (e.g., MUSE) can be used to generate sentence embeddings 570 for the input utterance 555 by mapping natural language elements, including sentences, words, and n-grams, to arrays of numbers. Each natural language element is represented as a single point in a vector space. Therefore, a sentence embedding is a vector representing the values ​​of a natural language element.

[0143] Determining the similarity or difference between the sentence embedding 570 of the input utterance 555 and each embedding representation 545 of each cluster may include: (i) calculating the absolute difference 565 between the sentence embedding 570 of the input utterance 555 and each embedding representation 545 of each cluster; (ii) feeding the absolute difference 565, the sentence embedding 570 of the input utterance 565, and the embedding representation 545 of each cluster into a wide and deep learning network; (iii) using a linear model and the absolute difference 565 to predict a wide-base probability regarding whether the input utterance 555 belongs to the target domain; and (iv) using a deep neural network, the sentence embedding 570 of the input utterance 555, and the embedding representation 545 of each cluster to determine the similarity or difference between the sentence embedding 570 of the input utterance 555 and each embedding representation 545 of each cluster. The probability 580 of whether the input utterance 555 belongs to the target domain can be predicted by evaluating the width probability using the final layer of a wide and deep learning network, as well as the similarity or difference between the sentence embedding 570 of the input utterance 555 and each embedding representation 545 of each cluster.

[0144] The integration component 515 evaluates probabilities 580 and 550 to determine a final probability 585 regarding whether the input utterance 555 belongs to the target domain, and classifies the input utterance 555 as either in-domain or out-of-domain based on the final probability 585. In some instances, the integration component 515 uses the following in_domain_prob function (in_domain_prob(ensemble,x) = max(in_domain_prob(cluster-based,x),in_domain_prob(metric-based,x)), where the in_domain_prob function returns the in-domain probability of utterance x based on both the cluster-based and metric-based methods. Essentially, if either method indicates that x is in the domain, the utterance is in the target domain; and if both methods indicate that x is out of the domain (an error occurs on the side where utterance x is in the domain), the utterance is out of the target domain.

[0145] Techniques for OOD determination

[0146] Figure 6 This is a flowchart illustrating a process 600 for identifying OOD discourse according to certain embodiments. Figure 6 The described process can be implemented as software (e.g., code, instructions, programs), hardware, or a combination thereof, executed by one or more processing units (e.g., processors, cores) of a corresponding system. The software can be stored on a non-transitory storage medium (e.g., a memory device). Figure 6 The methods presented and described below are intended to be illustrative rather than restrictive. Although Figure 6 The individual processing steps, occurring in a specific sequence or order, are described, but this is not intended to be limiting. In some alternative embodiments, the steps may be performed in a different order, or some steps may be performed in parallel. In some embodiments, such as in... Figures 1 to 5 In the embodiments depicted, Figure 6 The processing described herein can be performed by a trained model architecture (e.g., model architecture 500) to identify OOD utterances.

[0147] At point 605, receive conversations and chatbot messages (e.g., about...). Figure 1 , Figure 2 and Figure 3 The target domain (described by the skills or chatbot). The target domain is defined for chatbots that focus on specific types of tasks (such as tracking inventory, submitting attendance cards, and creating expense reports).

[0148] At position 610, sentence embeddings are generated for the utterance. Sentence embeddings for the utterance can be generated using an embedding model that maps natural language elements, including sentences, words, and n-grams, to an array of numbers. Each natural language element is represented as a single point in a vector space. Therefore, each sentence embedding is a vector representing the values ​​of the natural language elements.

[0149] At point 615, an embedding representation is obtained for each of multiple clusters of intra-domain utterances associated with the target domain. The embedding representation for each cluster is the average of the sentence embeddings of each intra-domain utterance within that cluster. Obtaining the embedding representation for each cluster may include: obtaining intra-domain utterances based on the target domain (e.g., if the target domain is ordering pizza, then all intra-domain utterances will involve utterances associated with ordering pizza, such as "I would like to order a cheese pizza"); generating sentence embeddings for each intra-domain utterance; inputting the sentence embeddings of each intra-domain utterance into an unsupervised clustering model configured to interpret the intra-domain utterances and identify multiple clusters within the feature space of the intra-domain utterances; using the unsupervised clustering model, classifying the sentence embeddings of each intra-domain utterance into one of the multiple clusters based on the similarity and difference between the features of the sentence embeddings and the features of the sentence embeddings within each cluster; calculating the centroid of each of the multiple clusters; and outputting the embedding representation and centroid of each of the multiple clusters. Unsupervised clustering models can include K-means, affine propagation, agglomerative clustering, BIRCH, DBSCAN, mean shift, OPTICS, etc.

[0150] Sentence embeddings for each domain-specific utterance can be generated using an embedding model that maps natural language elements, including sentences, words, and n-grams, to an array of numbers. Each natural language element is represented as a single point in a vector space. Therefore, each sentence embedding is a vector representing the values ​​of the natural language elements.

[0151] At 620, the sentence embedding of the utterance and the embedding representation of each cluster are input into a metric learning model with learned model parameters, the metric learning model being configured to provide a first probability about whether the utterance belongs to the target domain. At 625, the metric learning model is used to determine the similarity or difference between the sentence embedding of the utterance and each embedding representation of each cluster. At 630, the metric learning model is used to predict a first probability about whether the utterance belongs to the target domain based on the determined similarity or difference between the sentence embedding of the utterance and each embedding representation of each cluster.

[0152] In some embodiments, the metric learning model includes a stacked highway network with a nonlinear transformation as part of a gating function. Determining the similarity or difference between the sentence embeddings of the utterance and each embedding representation of each cluster may include: (i) calculating the absolute difference between the sentence embeddings of the utterance and each embedding representation of each cluster; (ii) inputting the absolute difference, the sentence embeddings of the utterance, and the embedding representations of each cluster into the stacked highway network; and (iii) using the stacked highway network, the absolute difference, the sentence embeddings of the utterance, and the embedding representations of each cluster to determine the similarity or difference between the sentence embeddings of the utterance and each embedding representation of each cluster.

[0153] The model parameters of a stacked highway network can be learned using a training dataset comprising: (i) sentence embeddings of utterances, (ii) embedding representations of each cluster consisting of sentence embeddings of intra-domain utterances from multiple domains, and (iii) the absolute difference between the sentence embeddings of the utterances and each embedding representation of each cluster consisting of sentence embeddings of intra-domain utterances. During training the metric learning model using the training dataset, the high-dimensional features of the sentence embeddings and the embedding representations of each cluster are transformed into lower-dimensional vectors, which are then concatenated with the features of the intra-domain utterances and fed into the hidden layers of the deep neural network. The values ​​of the lower-dimensional vectors are randomly initialized and learned along with the model parameters to minimize the loss function.

[0154] In other embodiments, the metric learning model includes a wide and deep learning network with a linear model and a deep neural network. Determining the similarity or difference between the sentence embedding of the utterance and each embedding representation of each cluster includes: (i) calculating the absolute difference between the sentence embedding of the utterance and each embedding representation of each cluster; (ii) inputting the absolute difference, the sentence embedding of the utterance, and the embedding representation of each cluster into the wide and deep learning network; (iii) using the linear model and the absolute difference to predict a wide-base probability regarding whether the utterance belongs to a target domain; and (iv) using the deep neural network, the sentence embedding of the utterance, and the embedding representation of each cluster to determine the similarity or difference between the sentence embedding of the utterance and each embedding representation of each cluster. Predicting a first probability includes: using the final layer of the wide and deep learning network to evaluate the width probability and the similarity or difference between the sentence embedding of the utterance and each embedding representation of each cluster.

[0155] The linear model comprises model parameters trained using a training dataset. The training dataset consists of the absolute differences between sentence embeddings of utterances and the absolute differences between each embedding representation of each cluster of in-domain utterances from multiple domains. During training the linear model using the training dataset, a hypothesis function is used to learn the linear relationship between the sentence embeddings of utterances and the absolute differences between each embedding representation of each cluster. During the learning of this linear relationship, multiple model parameters are learned to minimize the loss function.

[0156] A deep learning network comprises model parameters trained using a training dataset. The training dataset includes sentence embeddings of intra-domain utterances from multiple domains. During training of the deep learning network using the training dataset, high-dimensional features of the intra-domain utterance sentence embeddings are transformed into lower-dimensional vectors. These lower-dimensional vectors are then concatenated with the intra-domain utterance features and fed into the hidden layers of the deep neural network. The values ​​of these lower-dimensional vectors are randomly initialized and learned along with multiple model parameters to minimize a loss function.

[0157] At position 635, the sentence embeddings of the utterance and the embedding representations of each cluster are input into an outlier detection model constructed using a distance or density algorithm for outlier detection. The distance or density algorithm can be Z-score, K-means, DBSCAN, Local Outlier Detection (LOF), Isolation Forest, etc.

[0158] At 640, an outlier detection model is used to determine the distance or density deviation between the sentence embedding of the utterance and the embedding representations of neighboring clusters. At 645, an outlier detection model is used to predict a second probability about whether the utterance belongs to the target domain, based on the determined distance or density deviation. The prediction may include calculating the z-score of the utterance based on the distance or density deviation between the sentence embedding of the utterance and the embedding representations of neighboring clusters, and determining the second probability about whether the utterance belongs to the target domain by applying a sigmoid function to the z-score.

[0159] At 650, the first and second probabilities are evaluated to determine the final probability of whether the utterance belongs to the target domain. At 655, the utterance is classified as either in-domain or out-of-domain based on the final probability. The probabilities calculated from the cluster-based and metric-based methods are combined into an ensemble method to obtain the best results from both. In some instances, the ensemble method includes (in_domain_prob(ensemble,x) = max(in_domain_prob(cluster-based,x),in_domain_prob(metric-based,x)), where the in_domain_prob function returns the in-domain probability of utterance x based on both the cluster-based and metric-based methods. Essentially, if either method indicates that x is in the domain, the utterance is in the target domain; and if both methods indicate that x is out of the domain (an error occurs on the side where utterance x is in the domain), the utterance is out of the target domain.

[0160] Explanatory System

[0161] Figure 7A simplified diagram of a distributed system 700 is depicted. In the illustrated example, the distributed system 700 includes one or more client computing devices 702, 704, 706, and 708 coupled to a server 712 via one or more communication networks 710. The client computing devices 702, 704, 706, and 708 can be configured to execute one or more applications.

[0162] In various examples, server 712 may be adapted to run one or more services or software applications implementing one or more embodiments described in this disclosure. In some examples, server 712 may also provide other services or software applications that may include non-virtual and virtual environments. In some examples, these services may be provided as web-based services or cloud services (such as under a Software as a Service (SaaS) model) to users of client computing devices 702, 704, 706, and / or 708. Users operating client computing devices 702, 704, 706, and / or 708 may then use one or more client applications to interact with server 712 to utilize the services provided by these components.

[0163] exist Figure 7 In the depicted configuration, server 712 may include one or more components 718, 720, and 722 that implement the functions performed by server 712. These components may include software components that can be executed by one or more processors, hardware components, or a combination thereof. It should be understood that various different system configurations, different from distributed system 700, are possible. Therefore, Figure 7 The example shown is an example of a distributed system for implementing the example system and is not intended to be restrictive.

[0164] Users can use client computing devices 702, 704, 706, and / or 708 to execute one or more applications, models, or chatbots, which can generate one or more events or models that can then be implemented or serviced according to the teachings of this disclosure. The client device can provide an interface that enables users of the client device to interact with it. The client device can also output information to the user via this interface. Although Figure 7 It describes only four client computing devices, but can support any number of client computing devices.

[0165] Client devices can include various types of computing systems, such as portable handheld devices, general-purpose computers like personal computers and laptops, workstation computers, wearable devices, gaming systems, thin clients, various messaging devices, sensors, or other sensing devices. These computing devices can run various types and versions of software applications and operating systems (e.g., Microsoft). Apple Or a UNIX-like operating system, Linux, or a Linux-like operating system (such as Google Chrome). TM OS), including various mobile operating systems (e.g., Microsoft Windows). Windows Android TM , Palm Portable handheld devices can include cellular phones, smartphones (e.g., ), tablet computers (e.g., Wearable devices include Google's personal digital assistants (PDAs), personal digital assistants (PDAs), and more. Head-mounted displays and other devices. Gaming systems can include various handheld gaming devices, internet-enabled gaming devices (e.g., with or without internet access). Microsoft gesture input devices Game console, Sony System, by Various game systems and others are provided. Client devices can run a variety of different applications, such as various Internet-related applications, communication applications (e.g., email applications, short message service (SMS) applications), and can use various communication protocols.

[0166] (Multiple) networks 710 can be any type of network familiar to those skilled in the art that supports data communication using any of the various available protocols, including but not limited to TCP / IP (Transmission Control Protocol / Internet Protocol), SNA (System Network Architecture), and IPX (Internet Packet Switching). And so on. By way of example only, the (multiple) networks 710 can be local area networks (LANs), Ethernet-based networks, token ring networks, wide area networks (WANs), the Internet, virtual networks, virtual private networks (VPNs), intranets, extranets, public switched telephone networks (PSTNs), infrared networks, wireless networks (e.g., according to the IEEE 1002.11 protocol suite), etc. (and / or any other wireless protocol operating on a network) and / or any combination of these networks and / or other networks.

[0167] Server 712 may consist of: one or more general-purpose computers, dedicated server computers (including PC (personal computer) servers by way of example), Servers (including servers, mid-range servers, mainframe computers, rack servers, etc.), server groups, server clusters, or any other suitable arrangement and / or combination. Server 712 may include one or more virtual machines running a virtual operating system or other computing architectures involving virtualization, such as logical storage devices that can be virtualized to maintain one or more flexible pools of virtual storage devices for the server. In various examples, server 712 may be adapted to run one or more services or software applications that provide the functionality described in the foregoing disclosure.

[0168] The computing system in server 712 can run one or more operating systems, including any of the operating systems discussed above and any commercially available server operating system. Server 712 can also run any of a variety of additional server applications and / or middleware applications, including HTTP (Hypertext Transfer Protocol) servers, FTP (File Transfer Protocol) servers, CGI (Common Gateway Interface) servers, etc. Servers, database servers, etc. Exemplary database servers include, but are not limited to, those that can be accessed from... Those database servers acquired through purchases by companies such as IBM.

[0169] In some implementations, server 712 may include one or more applications to analyze and merge data feeds and / or event updates received from users of client computing devices 702, 704, 706, and 708. As an example, data feeds and / or event updates may include, but are not limited to, those provided in the original text. feed, The server 712 may receive real-time updates from one or more third-party information sources and continuous data streams. These real-time updates may include real-time events related to sensor data applications, financial reporting machines, network performance measurement tools (e.g., network monitoring and traffic management applications), clickstream analysis tools, vehicle traffic monitoring, and the like. The server 712 may also include one or more applications to display data feeds and / or real-time events via one or more display devices of client computing devices 702, 704, 706, and 708.

[0170] The distributed system 700 may also include one or more data repositories 714, 716. In some examples, these data repositories may be used to store data and other information. For example, one or more of data repositories 714, 716 may be used to store information (such as information related to chatbot performance or generated models) for use by the chatbot, which is used by server 712 in performing various functions according to various embodiments. Data repositories 714, 716 may reside in various locations. For example, the data repository used by server 712 may be local to server 712 or may be located remotely to server 712 and communicate with server 712 via a network-based or dedicated connection. Data repositories 714, 716 may be of different types. In some examples, the data repository used by server 712 may be a database, such as a relational database, such as one built by Oracle. Databases from other vendors. One or more of these databases can be adapted to respond to SQL-formatted commands to store, update, and retrieve data from the database.

[0171] In some examples, one or more of the data stores 714, 716 can also be used by the application to store application data. The data store used by the application can be of different types, such as key-value stores, object stores, or general-purpose stores supported by the file system.

[0172] In some examples, the functionality described in this disclosure can be provided as a service through a cloud environment. Figure 8 This is a simplified block diagram of a cloud-based system environment in which various services, based on certain examples, can be provided as cloud services. Figure 8 In the depicted example, cloud infrastructure system 802 can provide one or more cloud services that can be requested by a user using one or more client computing devices 804, 806, and 808. Cloud infrastructure system 802 may include one or more computers and / or servers, which may include those computers and / or servers described above with respect to server 812. The computers in cloud infrastructure system 802 may be organized as general-purpose computers, dedicated server computers, server clusters, server groups, or any other suitable arrangement and / or combination.

[0173] Multiple networks 810 can facilitate data communication and exchange between clients 804, 806, and 808 and cloud infrastructure system 802. Multiple networks 810 may include one or more networks. The networks may be of the same or different types. Multiple networks 810 may support one or more communication protocols (including wired and / or wireless protocols) to facilitate communication.

[0174] Figure 8 The example depicted is merely one example of a cloud infrastructure system and is not intended to be limiting. It should be understood that in some other examples, the cloud infrastructure system 802 may have more... Figure 8 The depicted components may have more or fewer components, may combine two or more components, or may have different component configurations or arrangements. For example, although... Figure 8 Three client computing devices are depicted, but in alternative examples, any number of client computing devices can be supported.

[0175] The term cloud service is generally used to refer to services that become available to users on demand through a service provider's systems (e.g., cloud infrastructure systems 802) and via communication networks such as the Internet. Typically, in a public cloud environment, the servers and systems that make up the cloud service provider's systems differ from the customer's own on-premises servers and systems. The cloud service provider's systems are managed by the cloud service provider. Therefore, customers can utilize cloud services provided by the cloud service provider without having to purchase separate licenses, support, or hardware and software resources for the services. For example, the cloud service provider's systems can host applications, and users can subscribe to and use the applications on demand via the Internet without having to purchase the infrastructure resources to run the applications. Cloud services are designed to provide easy, scalable access to applications, resources, and services. Several providers offer cloud services. For example, Oracle, based in Redwood Shores, California. It provides several cloud services, such as middleware services, database services, Java cloud services, and other services.

[0176] In some examples, cloud infrastructure system 802 may provide one or more cloud services using different models, such as Software as a Service (SaaS), Platform as a Service (PaaS), Infrastructure as a Service (IaaS), and other models (including hybrid service models). Cloud infrastructure system 802 may include a set of applications, middleware, databases, and other resources that enable the provisioning of various cloud services.

[0177] The SaaS model enables applications or software to be delivered as a service to customers via communication networks such as the Internet, without the customer having to purchase the underlying application's hardware or software. For example, the SaaS model can be used to provide customers with access to on-demand applications hosted by cloud infrastructure systems. Oracle Examples of the SaaS services offered include, but are not limited to, various services for human resources / capital management, customer relationship management (CRM), enterprise resource planning (ERP), supply chain management (SCM), enterprise performance management (EPM), analytics services, social applications, and others.

[0178] The IaaS model is typically used to provide customers with infrastructure resources (such as servers, storage, hardware, and networking resources) as cloud services to offer elastic computing and storage capabilities. (Produced by Oracle) It provides a variety of IaaS services.

[0179] The PaaS model is typically used to provide a platform and environment resources as a service that enable customers to develop, run, and manage applications and services without requiring them to purchase, build, or maintain such resources. (Provided by Oracle) Examples of PaaS services provided include, but are not limited to, Oracle Java Cloud Service (JCS), Oracle Database Cloud Service (DBCS), Data Management Cloud Service, various application development solutions services, and other services.

[0180] Cloud services are typically delivered in an on-demand, self-service, subscription-based, elastically scalable, reliable, highly available, and secure manner. For example, a customer may subscribe to one or more services provided by cloud infrastructure system 802 via a subscription order. Cloud infrastructure system 802 then performs processing to provide the service requested in the customer's subscription order. For instance, a user may use utterances to request the cloud infrastructure system to take an action (e.g., an intent) as described above and / or to provide services for a chatbot system as described herein. Cloud infrastructure system 802 can be configured to provide one or more cloud services.

[0181] Cloud infrastructure system 802 can provide cloud services through different deployment models. In a public cloud model, cloud infrastructure system 802 can be owned by a third-party cloud service provider, and cloud services are provided to any general public customer, which can be an individual or a business. In some other examples, under a private cloud model, cloud infrastructure system 802 can operate within an organization (e.g., within a business organization), and services are provided to customers within the organization. For example, customers can be various departments within the organization, such as human resources or payroll departments, or even individuals within the organization. In some other examples, under a community cloud model, cloud infrastructure system 802 and the services provided can be shared by several organizations in the relevant community. Various other models, such as hybrids of the models mentioned above, can also be used.

[0182] Client computing devices 804, 806, and 808 can be of different types (e.g., Figure 7 The depicted client computing devices 702, 704, 706, and 708 may also be capable of operating one or more client applications. Users can use the client devices to interact with the cloud infrastructure system 802, such as requesting services provided by the cloud infrastructure system 802. For example, a user can use the client device to request information or actions from a chatbot as described in this disclosure.

[0183] In some examples, the processing performed by cloud infrastructure system 802 to provide services may involve model training and deployment. This analysis may involve using, analyzing, and manipulating datasets to train and deploy one or more models. This analysis may be performed by one or more processors, potentially processing data in parallel, performing simulations using the data, etc. For example, big data analysis may be performed by cloud infrastructure system 802 to generate and train one or more models for a chatbot system. The data used for this analysis may include structured data (e.g., data stored in a database or structured according to a structured model) and / or unstructured data (e.g., data blocks (binary large objects)).

[0184] like Figure 8 As illustrated in the examples, cloud infrastructure system 802 may include infrastructure resources 830 used to facilitate the provision of various cloud services offered by cloud infrastructure system 802. Infrastructure resources 830 may include, for example, processing resources, storage or memory resources, networking resources, etc. In some examples, a storage virtual machine that can be used to service storage requested from an application may be part of cloud infrastructure system 802. In other examples, the storage virtual machine may be part of a different system.

[0185] In some examples, to facilitate the efficient provisioning of these resources to support the various cloud services offered by the cloud infrastructure system 802 to different customers, resources can be bound to resource groups or resource modules (also known as "pods"). Each resource module or pod may include a pre-integrated and optimized combination of one or more types of resources. In some examples, different pods may be pre-provisioned for different types of cloud services. For example, a first pod may be provisioned for a database service, and a second pod may be provisioned for a Java service (the second pod may include a different combination of resources than the pods in the first pod), and so on. For some services, resources allocated for provisioning the service can be shared between services.

[0186] The cloud infrastructure system 802 itself can internally use services 832 shared by different components of the cloud infrastructure system 802 and that facilitate the provision of services by the cloud infrastructure system 802. These internal shared services may include, but are not limited to, security and identity services, integration services, enterprise repository services, enterprise manager services, virus scanning and whitelisting services, high availability, backup and recovery services, services for enabling cloud support, email services, notification services, file transfer services, etc.

[0187] Cloud infrastructure system 802 may include multiple subsystems. These subsystems may be implemented in software or hardware, or a combination thereof. Figure 8 The depicted subsystem may include a user interface subsystem 812 that enables users or customers of the cloud infrastructure system 802 to interact with the cloud infrastructure system 802. The user interface subsystem 812 may include various interfaces, such as a web interface 814, an online store interface 816 (where advertisements are displayed and customers can purchase cloud services provided by the cloud infrastructure system 802), and other interfaces 818. For example, a customer may use a client device to request (service request 834) one or more services provided by the cloud infrastructure system 802 using one or more of the interfaces 814, 816, and 818. For example, a customer may access an online store, browse cloud services provided by the cloud infrastructure system 802, and place a subscription order for one or more services provided by the cloud infrastructure system 802 that the customer wishes to subscribe to. A service request may include information identifying the customer and the one or more services the customer wishes to subscribe to. For example, a customer may place a subscription order for services provided by the cloud infrastructure system 802. As part of the order, the customer may provide information identifying the service to be provided to a chatbot system and optionally provide one or more credentials for the chatbot system.

[0188] In some examples (such as) Figure 8 In the illustrated example, cloud infrastructure system 802 may include an order management subsystem (OMS) 820 configured to process new orders. As part of this process, OMS 820 may be configured to: create an account for a customer (if not already created); receive billing and / or billing information from the customer to be used to bill the customer for the requested services; verify customer information; place an order for the customer after verification; and schedule various workflows to prepare the order for delivery.

[0189] Once correctly verified, OMS 820 can invoke the Order Provisioning Subsystem (OPS) 824, configured as order provisioning resources (including processing resources, storage resources, and networking resources). Provisioning may include allocating resources for an order and configuring resources to facilitate the service requested by the customer's order. The manner in which resources are provisioned for an order and the type of resources provided may depend on the type of cloud service the customer has subscribed to. For example, according to a workflow, OPS 824 may be configured to determine the specific cloud service being requested and identify the number of groups that may have been pre-configured for that specific cloud service. The number of groups allocated to an order may depend on the size / volume / tier / scope of the requested service. For example, the number of groups to be allocated may be determined based on the number of users the service is to support, the duration of the requested service, etc. The allocated groups can then be customized for a specific requesting customer to provide the requested service.

[0190] In some examples, the setup phase processing described above can be performed by the cloud infrastructure system 802 as part of the provisioning process. The cloud infrastructure system 802 can generate application IDs and select storage virtual machines for the application from storage virtual machines provided by the cloud infrastructure system 802 itself or from storage virtual machines provided by other systems besides the cloud infrastructure system 802.

[0191] The cloud infrastructure system 802 may send a response or notification 844 to the requesting client to indicate when the requested service is now ready for use. In some instances, information (e.g., a link) enabling the client to begin using and taking advantage of the benefits of the requested service may be sent. In some examples, for the client requesting the service, the response may include a chatbot system ID generated by the cloud infrastructure system 802 and information identifying the chatbot system selected by the cloud infrastructure system 802 corresponding to the chatbot system with the chatbot system ID.

[0192] Cloud infrastructure system 802 can provide services to multiple customers. For each customer, cloud infrastructure system 802 is responsible for managing information related to one or more subscription orders received from the customer, maintaining customer data related to the orders, and providing the requested services to the customer. Cloud infrastructure system 802 can also collect usage statistics about customers' use of subscribed services. For example, it can collect statistics such as storage usage, data transfer volume, number of users, system uptime, and system downtime. This usage information can be used to issue bills to customers. Billing can be done, for example, on a monthly basis.

[0193] Cloud infrastructure system 802 can provide services to multiple customers in parallel. Cloud infrastructure system 802 can store information about these customers (potentially including proprietary information). In some examples, cloud infrastructure system 802 includes an Identity Management Subsystem (IMS) 828 configured to manage customer information and provide separation of the managed information so that information related to one customer cannot be accessed by another customer. IMS 828 can be configured to provide various security-related services, such as identity services, information access management, authentication and authorization services, services for managing customer identities and roles, and related functions.

[0194] Figure 9 An example of computer system 900 is illustrated. In some examples, computer system 900 can be used to implement any digital assistant or chatbot system within a distributed environment, as well as the various servers and computer systems described above. Figure 9 As shown, the computer system 900 includes various subsystems, including a processing subsystem 904 that communicates with multiple other subsystems via a bus subsystem 902. These other subsystems may include a processing acceleration unit 906, an I / O subsystem 908, a storage subsystem 918, and a communication subsystem 924. The storage subsystem 918 may include non-transitory computer-readable storage media, including storage medium 922 and system memory 910.

[0195] Bus subsystem 902 provides a mechanism for allowing the various components and subsystems of computer system 900 to communicate with each other as intended. While bus subsystem 902 is schematically shown as a single bus, alternative examples of bus subsystems may utilize multiple buses. Bus subsystem 902 can be any of several types of bus structures, including a memory bus or memory controller, a peripheral bus, a local bus using any of a variety of bus architectures, etc. For example, such architectures may include an Industry Standard Architecture (ISA) bus, a Micro Channel Architecture (MCA) bus, an Enhanced ISA (EISA) bus, a Video Electronics Standards Association (VESA) local bus, and a Peripheral Component Interconnect (PCI) bus (which may be implemented as a mezzanine bus manufactured to the IEEE P1386.1 standard), etc.

[0196] Processing subsystem 904 controls the operation of computer system 900 and may include one or more processors, application-specific integrated circuits (ASICs), or field-programmable gate arrays (FPGAs). Processors may include single-core or multi-core processors. The processing resources of computer system 900 may be organized into one or more processing units 932, 934, etc. A processing unit may include one or more processors, one or more cores from the same or different processors, a combination of cores and processors, or other combinations of cores and processors. In some examples, processing subsystem 904 may include one or more dedicated coprocessors such as graphics processors or digital signal processors (DSPs). In some examples, some or all of the processing units of processing subsystem 904 may be implemented using custom circuitry such as ASICs or FPGAs.

[0197] In some examples, the processing units in processing subsystem 904 can execute instructions stored in system memory 910 or on computer-readable storage medium 922. In various examples, the processing units can execute various program or code instructions and can maintain multiple concurrently executing programs or processes. At any given time, some or all of the program code to be executed can reside in system memory 910 and / or on computer-readable storage medium 922 (potentially including residing on one or more storage devices). With appropriate programming, processing subsystem 904 can provide the various functions described above. In an instance where computer system 900 is executing one or more virtual machines, one or more processing units can be assigned to each virtual machine.

[0198] In some examples, a processing acceleration unit 906 may optionally be provided for performing custom processing or for offloading some of the processing performed by the processing subsystem 904, thereby accelerating the overall processing performed by the computer system 900.

[0199] I / O subsystem 908 may include devices and mechanisms for inputting information to and / or outputting information from or via computer system 900. Generally, the term input device is intended to include all possible types of devices and mechanisms for inputting information to computer system 900. User interface input devices may include, for example, keyboards, pointing devices such as mice or trackballs, touchpads or touchscreens integrated into displays, scroll wheels, click wheels, dial pads, buttons, switches, keypads, audio input devices with voice command recognition systems, microphones, and other types of input devices. User interface input devices may also include motion sensing and / or gesture recognition devices, such as Microsoft's [unclear - possibly a specific technology or feature], which enables users to control and interact with input devices. Motion sensor, Microsoft The 360 ​​game controller provides an interface for receiving input using gestures and spoken commands. The user interface input device may also include eye gesture recognition devices, such as detecting eye movements from the user (e.g., "blinking" when taking a photo and / or making menu selections) and translating the eye gestures into the input device (such as Google). Google input Blink detector. Additionally, the user interface input device may include enabling the user to interact with a voice recognition system (e.g., ...) via voice commands. A voice recognition sensing device for interaction with navigators.

[0200] Other examples of user interface input devices include, but are not limited to, 3D mice, joysticks or pointing sticks, game controllers and graphics tablets, and audio / visual devices such as speakers, digital cameras, digital camcorders, portable media players, webcams, image scanners, fingerprint scanners, barcode readers, 3D scanners, 3D printers, laser rangefinders, and eye-tracking devices. Additionally, user interface input devices may include, for example, medical imaging input devices such as computed tomography (CT), magnetic resonance imaging (MRI), positron emission tomography (PET), and medical ultrasound examination equipment. User interface input devices may also include, for example, audio input devices such as MIDI keyboards, digital musical instruments, etc.

[0201] Generally, the term "output device" is intended to encompass all possible types of devices and mechanisms for outputting information from a computer system 900 to a user or other computer. User interface output devices may include display subsystems, indicator lights, or non-visual displays such as audio output devices. Display subsystems may be cathode ray tube (CRT), flat panel devices (such as those using liquid crystal displays (LCDs) or plasma displays), projection devices, touchscreens, etc. For example, user interface output devices may include, but are not limited to, various display devices that visually convey text, graphics, and audio / video information, such as monitors, printers, speakers, headsets, car navigation systems, plotters, voice output devices, and modems.

[0202] Storage subsystem 918 provides a repository or data store for storing information and data used by computer system 900. Storage subsystem 918 provides a tangible, non-transitory, computer-readable storage medium for storing basic programming and data constructs that provide some example functionality. Storage subsystem 918 may store software (e.g., programs, code modules, instructions) that provides the functionality described above when executed by processing subsystem 904. The software may be executed by one or more processing units of processing subsystem 904. Storage subsystem 918 may also provide certification according to the teachings of this disclosure.

[0203] The storage subsystem 918 may include one or more non-transitory memory devices, which may include volatile memory devices and non-volatile memory devices. For example... Figure 9 As shown, the storage subsystem 918 includes system memory 910 and computer-readable storage medium 922. System memory 910 may include multiple memories, including volatile main random access memory (RAM) for storing instructions and data during program execution and non-volatile read-only memory (ROM) or flash memory where fixed instructions are stored. In some embodiments, a basic input / output system (BIOS) containing basic routines that help transfer information between elements within the computer system 900 during startup may typically be stored in ROM. RAM typically contains data and / or program modules currently operated and executed by processing subsystem 904. In some embodiments, system memory 910 may include various different types of memory such as static random access memory (SRAM), dynamic random access memory (DRAM), etc.

[0204] By using examples rather than restrictions, such as Figure 9 As depicted, system memory 910 can load an executing application 912 (which may include various applications such as web browsers, middleware applications, relational database management systems (RDBMS), etc.), program data 914, and operating system 916. By way of example, operating system 916 may include various versions of Microsoft... Apple and / or Linux operating system, and various commercially available... Or a UNIX-like operating system (including but not limited to various GNU / Linux operating systems, Google...) OS, etc.) and / or such as iOS, Telephone, OS OS OS operating system and other mobile operating systems.

[0205] Computer-readable storage medium 922 can store programming and data structures that provide some example functionality. Computer-readable storage medium 922 can provide storage for computer-readable instructions, data structures, program modules, and other data for computer system 900. Software (programs, code modules, instructions) that provides the functions described above, when executed by processing subsystem 904, can be stored in storage subsystem 918. By way of example, computer-readable storage medium 922 may include, for example, hard disk drives, disk drives, optical disc drives (such as CD-ROMs, DVDs, etc.). Non-volatile memory such as disks or other optical media. Computer-readable storage medium 922 may include, but is not limited to, [other types of storage media]. Drives, flash memory cards, Universal Serial Bus (USB) flash memory drives, Secure Digital (SD) cards, DVD discs, digital videotapes, etc. Computer-readable storage media 922 may also include SSDs based on flash memory, such as flash memory-based solid-state drives (SSDs), enterprise-class flash memory drives, SSDs based on non-volatile memory such as solid-state ROM, SSDs based on volatile memory such as solid-state RAM, dynamic RAM, static RAM, DRAM-based SSDs, magnetoresistive RAM (MRAM) SSDs, and hybrid SSDs using a combination of DRAM and flash memory-based SSDs.

[0206] In some examples, the storage subsystem 918 may also include a computer-readable storage medium reader 920 that can be further connected to a computer-readable storage medium 922. The reader 920 may receive data from a storage device such as a disk, flash memory drive, etc., and is configured to read data from said storage device.

[0207] In some examples, computer system 900 may support virtualization technologies, including but not limited to the virtualization of processing and memory resources. For example, computer system 900 may provide support for executing one or more virtual machines. In some examples, computer system 900 may execute programs such as hypervisors that facilitate the configuration and management of virtual machines. Each virtual machine may be allocated memory, computing (e.g., processor, cores), I / O, and networking resources. Each virtual machine typically runs independently of other virtual machines. Virtual machines typically run their own operating system, which may be the same as or different from the operating systems executed by other virtual machines executed by computer system 900. Therefore, multiple operating systems may potentially be run simultaneously by computer system 900.

[0208] The communication subsystem 924 provides interfaces to other computer systems and networks. The communication subsystem 924 serves as an interface for receiving data from other systems and transmitting data from computer system 900 to other systems. For example, the communication subsystem 924 enables computer system 900 to establish communication channels to one or more client devices via the Internet for receiving and sending information to client devices. For example, when computer system 900 is used to implement... Figure 1 When the robot system 120 is described, the communication subsystem can be used to communicate with a chatbot system selected for the application.

[0209] The communication subsystem 924 may support both wired and / or wireless communication protocols. In some examples, the communication subsystem 924 may include radio frequency (RF) transceiver components (e.g., advanced data network technologies such as cellular telephone technology, 3G, 4G, or EDGE (Global Evolution Enhanced Data Rate), WiFi (IEEE 802.XX Home Standard, or other mobile communication technologies, or any combination thereof), GPS receiver components, and / or other components for accessing wireless voice and / or data networks. In some examples, in addition to or as an alternative to a wireless interface, the communication subsystem 924 may provide wired network connectivity (e.g., Ethernet).

[0210] The communication subsystem 924 can receive and transmit data in various forms. In some examples, among others, the communication subsystem 924 can also receive input communications in the form of structured and / or unstructured data feeds 926, event streams 928, event updates 930, etc. For example, the communication subsystem 924 can be configured to receive (or send) data feeds 926 in real time from users of social media networks and / or other communication services, such as... feed, Updates, web feeds (such as rich site summary (RSS) feeds) and / or real-time updates from one or more third-party information sources.

[0211] In some examples, the communication subsystem 924 may be configured to receive data in the form of a continuous data stream, which may include an event stream 928 and / or event updates 930 of real-time events (which may be inherently continuous or unbounded and may not have an explicit end). Examples of applications that generate continuous data may include, for example, sensor data applications, financial reporting machines, network performance measurement tools (e.g., network monitoring and traffic management applications), clickstream analysis tools, vehicle traffic monitoring, etc.

[0212] The communication subsystem 924 can also be configured to transmit data from computer system 900 to other computer systems or networks. Data can be transmitted in various forms, such as structured and / or unstructured data feeds 926, event streams 928, event updates 930, etc., to one or more databases that can communicate with one or more streaming data source computers coupled to computer system 900.

[0213] The computer system 900 can be of a variety of types, including handheld portable devices (e.g., Cellular phone Computing tablets, PDAs), wearable devices (e.g., Google...) Head-mounted displays, personal computers, workstations, mainframes, self-service kiosks, server racks, or any other data processing systems. Due to the constantly evolving nature of computers and networks, [the following is relevant:] Figure 9 The description of the computer system 900 is intended only as a concrete example. It has a higher... Figure 9 Many other configurations with more or fewer components are possible for the system depicted. Based on this disclosure and the teachings provided herein, it should be understood that there are other ways and / or methods to implement the various examples.

[0214] While specific examples have been described, various modifications, alterations, alternative constructions, and equivalents are possible. The examples are not limited to operations in a particular data processing environment but are free to operate in multiple data processing environments. Furthermore, although certain examples have been described using specific series of transactions and steps, it will be apparent to those skilled in the art that this is not intended to be restrictive. While some flowcharts describe operations as sequential processes, many operations can be performed in parallel or simultaneously. Additionally, the order of operations can be rearranged. Processes may have additional steps not included in the diagrams. Various features and aspects of the examples described above can be used individually or in combination.

[0215] Furthermore, while certain examples have been described using specific combinations of hardware and software, it should be recognized that other combinations of hardware and software are also possible. Some examples may be implemented using only hardware, only software, or a combination thereof. The various processes described herein can be implemented on the same or different processors in any combination.

[0216] When a device, system, component, or module is described as being configured to perform certain operations or functions, this configuration can be accomplished, for example, by designing electronic circuitry to perform operations, by programming programmable electronic circuitry (such as a microprocessor) to perform operations (such as by executing computer instructions or code), or by a processor or core programmed to execute code or instructions stored on a non-transitory memory medium, or any combination thereof. Processes can communicate using a variety of techniques, including but not limited to conventional techniques for inter-process communication, and different pairs of processes may use different techniques, or the same pair of processes may use different techniques at different times.

[0217] Specific details are set forth in this disclosure to provide a thorough understanding of the examples. However, the examples can be practiced without these specific details. For example, well-known circuits, processes, algorithms, structures, and techniques have been shown without unnecessary details to avoid obscuring the examples. This description provides only examples and is not intended to limit the scope, applicability, or configuration of other examples. Rather, the foregoing description of the examples will provide those skilled in the art with enabling descriptions for implementing the various examples. Various changes can be made to the function and arrangement of the elements.

[0218] Therefore, this specification and drawings should be viewed in an illustrative rather than restrictive sense. However, it will be apparent that additions, omissions, deletions, and other modifications and changes may be made without departing from the broader spirit and scope set forth in the claims. Thus, while specific examples have been described, they are not intended to be restrictive. Various modifications and equivalents are within the scope of the following claims.

[0219] In the foregoing description, various aspects of this disclosure have been described with reference to specific examples; however, those skilled in the art will recognize that this disclosure is not limited thereto. Various features and aspects of the disclosure described above may be used individually or in combination. Furthermore, the examples may be utilized in any number of environments and applications other than those described herein without departing from the broader spirit and scope of the specification. Therefore, the specification and drawings should be considered illustrative rather than restrictive.

[0220] In the foregoing description, the methods have been described in a specific order for illustrative purposes. It should be understood that, in alternative examples, the methods may be performed in a different order than described. It should also be understood that the methods described above may be executed by hardware components or may be embodied in a sequence of machine-executable instructions that can be used to cause a machine (such as a general-purpose or special-purpose processor or logic circuit programmed with the instructions) to execute the methods. These machine-executable instructions may be stored on one or more machine-readable media, such as CD-ROMs or other types of optical discs, floppy disks, ROMs, RAMs, EPROMs, EEPROMs, magnetic or optical cards, flash memory, or other types of machine-readable media suitable for storing electronic instructions. Alternatively, the methods may be executed by a combination of hardware and software.

[0221] When a component is described as being configured to perform certain operations, such configuration can be accomplished, for example, by designing electronic circuitry or other hardware for performing the operations, by programming programmable electronic circuitry (e.g., a microprocessor or other suitable electronic circuitry) for performing the operations, or any combination thereof.

[0222] Although illustrative examples of this application have been described in detail herein, it should be understood that the inventive concepts may be embodied and employed in other ways, and the appended claims are intended to be construed as including such variations, except where limited by the prior art.

Claims

1. A method for identifying extraterritorial discourse, the method comprising: The target domain for receiving messages and chatbots; Generate sentence embeddings for the utterance; Obtain the embedding representation of each cluster of multiple clusters of intra-domain utterances associated with the target domain, wherein the embedding representation of each cluster is the average of the sentence embeddings of each intra-domain utterance in the cluster; The sentence embedding of the utterance and the embedding representation of each cluster are input into a metric learning model with learned model parameters, the metric learning model being configured to provide a first probability about whether the utterance belongs to the target domain; The metric learning model is used to determine the similarity or difference between the sentence embedding of the utterance and each embedding representation of each cluster; Using the metric learning model, a first probability is predicted regarding whether the utterance belongs to the target domain, based on the similarity or difference between the determined sentence embedding of the utterance and each embedding representation of each cluster. The sentence embedding of the utterance and the embedding representation of each cluster are input into an outlier detection model constructed using a distance or density algorithm for outlier detection. The outlier detection model is used to determine the distance or density deviation between the sentence embedding of the utterance and the embedding representation of the adjacent clusters; Using the outlier detection model, a second probability is predicted regarding whether the utterance belongs to the target domain based on the determined distance or density deviation; Evaluate the first probability and the second probability to determine a final probability regarding whether the utterance belongs to the target domain; and The utterance is classified as either in-domain or out-of-domain based on the final probability.

2. The method as described in claim 1, wherein, Obtaining the embedding representation for each cluster includes: Obtain the discourse within the target domain; Generate sentence embeddings for each domain discourse; The sentence embedding of each domain discourse is input into an unsupervised clustering model, which is configured to interpret the domain discourse and identify the multiple clusters in the feature space of the domain discourse; Using the unsupervised clustering model, the sentence embeddings of each domain discourse are classified into one of the plurality of clusters based on the similarity and difference between the features of the sentence embeddings and the features of the sentence embeddings within each cluster; Calculate the centroid of each of the plurality of clusters; and Output the embedding representation and centroid of each of the plurality of clusters.

3. The method of claim 1, further comprising: The z-score of the utterance is calculated based on the distance or density deviation between the sentence embedding of the utterance and the embedding representation of the neighboring clusters; as well as The second probability regarding whether the utterance belongs to the target domain is determined by applying a sigmoid function to the z-score.

4. The method of claim 1, wherein, The sentence embedding of the discourse is generated using an embedding model that maps natural language elements, including sentences, words, and n-grams, to an array of numbers, wherein each of the natural language elements is represented as a single point in a vector space.

5. The method of claim 1, wherein: Determining the similarity or difference between the sentence embedding of the utterance and each embedding representation of each cluster includes: (i) calculating the absolute difference between the sentence embedding of the utterance and each embedding representation of each cluster; (ii) inputting the absolute difference, the sentence embedding of the utterance, and the embedding representation of each cluster into a wide and deep learning network, wherein the wide and deep learning network includes a linear model and a deep neural network; (iii) using the linear model and the absolute difference to predict a wide-base probability regarding whether the utterance belongs to the target domain; and (iv) using the deep neural network, the sentence embedding of the utterance, and the embedding representation of each cluster to determine the similarity or difference between the sentence embedding of the utterance and each embedding representation of each cluster; and The prediction of the first probability includes: using the final layer of the wide and deep learning network to evaluate the wide base probability and the similarity or difference between the sentence embedding of the utterance and each embedding representation of each cluster.

6. The method of claim 5, wherein: The linear model includes multiple model parameters trained using a training dataset; The training dataset includes the absolute difference between sentence embeddings of utterances and each embedding representation of each cluster of in-domain utterances from multiple domains; During training the linear model using the training dataset, a hypothesis function is used to learn the linear relationship between the sentence embeddings of the utterance and each embedding representation of each cluster; as well as During the learning of the linear relationship, the plurality of model parameters are learned to minimize the loss function.

7. The method of claim 5, wherein: The deep neural network includes multiple model parameters trained using a training dataset; The training dataset includes sentence embeddings of intradomain utterances from multiple domains; and During the training of the deep neural network using the training dataset, the high-dimensional features of the sentence embeddings of the in-domain utterances are converted into lower-dimensional vectors, which are then concatenated with features from the in-domain utterances and fed into the hidden layers of the deep neural network. The values ​​of the lower-dimensional vectors are randomly initialized and learned together with the plurality of model parameters to minimize the loss function.

8. A non-transitory machine-readable storage medium, the non-transitory machine-readable storage medium comprising instructions configured to cause one or more data processors to perform actions including: The target domain for receiving messages and chatbots; Generate sentence embeddings for the utterance; Obtain the embedding representation of each cluster of multiple clusters of intra-domain discourses associated with the target domain, wherein, The embedding representation for each cluster is the average of the sentence embeddings of the utterances within each domain in the cluster; The sentence embedding of the utterance and the embedding representation of each cluster are input into a metric learning model with learned model parameters, the metric learning model being configured to provide a first probability about whether the utterance belongs to the target domain; The metric learning model is used to determine the similarity or difference between the sentence embedding of the utterance and each embedding representation of each cluster; Using the metric learning model, a first probability is predicted regarding whether the utterance belongs to the target domain, based on the similarity or difference between the determined sentence embedding of the utterance and each embedding representation of each cluster. The sentence embedding of the utterance and the embedding representation of each cluster are input into an outlier detection model constructed using a distance or density algorithm for outlier detection. The outlier detection model is used to determine the distance or density deviation between the sentence embedding of the utterance and the embedding representation of the adjacent clusters; Using the outlier detection model, a second probability is predicted regarding whether the utterance belongs to the target domain based on the determined distance or density deviation; Evaluate the first probability and the second probability to determine the final probability of whether the utterance belongs to the target domain; as well as The utterance is classified as either in-domain or out-of-domain based on the final probability.

9. The non-transitory machine-readable storage medium as described in claim 8, wherein, Obtaining the embedding representation for each cluster includes: Obtain the discourse within the target domain; Generate sentence embeddings for each domain discourse; The sentence embedding of each domain discourse is input into an unsupervised clustering model, which is configured to interpret the domain discourse and identify the multiple clusters in the feature space of the domain discourse; Using the unsupervised clustering model, the sentence embeddings of each domain discourse are classified into one of the plurality of clusters based on the similarity and difference between the features of the sentence embeddings and the features of the sentence embeddings within each cluster; Calculate the centroid of each of the plurality of clusters; and Output the embedding representation and centroid of each of the plurality of clusters.

10. The non-transitory machine-readable storage medium as described in claim 8, wherein, The action further includes: The z-score of the utterance is calculated based on the distance or density deviation between the sentence embedding of the utterance and the embedding representation of the neighboring clusters; and The second probability regarding whether the utterance belongs to the target domain is determined by applying a sigmoid function to the z-score.

11. The non-transitory machine-readable storage medium as claimed in claim 8, wherein, The sentence embedding of the discourse is generated using an embedding model that maps natural language elements, including sentences, words, and n-grams, to an array of numbers, wherein each of the natural language elements is represented as a single point in a vector space.

12. The non-transitory machine-readable storage medium as claimed in claim 8, wherein: Determining the similarity or difference between the sentence embedding of the utterance and each embedding representation of each cluster includes: (i) calculating the absolute difference between the sentence embedding of the utterance and each embedding representation of each cluster; (ii) inputting the absolute difference, the sentence embedding of the utterance, and the embedding representation of each cluster into a wide and deep learning network, wherein the wide and deep learning network includes a linear model and a deep neural network; (iii) using the linear model and the absolute difference to predict a wide-base probability regarding whether the utterance belongs to the target domain; and (iv) using the deep neural network, the sentence embedding of the utterance, and the embedding representation of each cluster to determine the similarity or difference between the sentence embedding of the utterance and each embedding representation of each cluster; and The prediction of the first probability includes: using the final layer of the wide and deep learning network to evaluate the wide base probability and the similarity or difference between the sentence embedding of the utterance and each embedding representation of each cluster.

13. The non-transitory machine-readable storage medium of claim 12, wherein: The linear model includes multiple model parameters trained using a training dataset; The training dataset includes the absolute difference between sentence embeddings of utterances and each embedding representation of each cluster of in-domain utterances from multiple domains; During training the linear model using the training dataset, a hypothesis function is used to learn the linear relationship between the sentence embeddings of the utterance and each embedding representation of each cluster; and During the learning of the linear relationship, the plurality of model parameters are learned to minimize the loss function.

14. The non-transitory machine-readable storage medium of claim 12, wherein: The deep neural network includes multiple model parameters trained using a training dataset; The training dataset includes sentence embeddings of intradomain utterances from multiple domains; and During the training of the deep neural network using the training dataset, the high-dimensional features of the sentence embeddings of the in-domain utterances are converted into lower-dimensional vectors, which are then concatenated with features from the in-domain utterances and fed into the hidden layers of the deep neural network. The values ​​of the lower-dimensional vectors are randomly initialized and learned together with the plurality of model parameters to minimize the loss function.

15. A system for identifying extraterritorial discourse, comprising: One or more data processors; as well as A non-transitory computer-readable storage medium containing instructions that, when executed on the one or more data processors, cause the one or more data processors to perform actions including: The target domain for receiving messages and chatbots; Generate sentence embeddings for the utterance; Obtain the embedding representation of each cluster of multiple clusters of intra-domain utterances associated with the target domain, wherein the embedding representation of each cluster is the average of the sentence embeddings of each intra-domain utterance in the cluster; The sentence embedding of the utterance and the embedding representation of each cluster are input into a metric learning model with learned model parameters, the metric learning model being configured to provide a first probability about whether the utterance belongs to the target domain; The metric learning model is used to determine the similarity or difference between the sentence embedding of the utterance and each embedding representation of each cluster; Using the metric learning model, a first probability is predicted regarding whether the utterance belongs to the target domain, based on the similarity or difference between the determined sentence embedding of the utterance and each embedding representation of each cluster. The sentence embedding of the utterance and the embedding representation of each cluster are input into an outlier detection model constructed using a distance or density algorithm for outlier detection. The outlier detection model is used to determine the distance or density deviation between the sentence embedding of the utterance and the embedding representation of the adjacent clusters; Using the outlier detection model, a second probability is predicted regarding whether the utterance belongs to the target domain based on the determined distance or density deviation; Evaluate the first probability and the second probability to determine a final probability regarding whether the utterance belongs to the target domain; and The utterance is classified as either in-domain or out-of-domain based on the final probability.

16. The system of claim 15, wherein, Obtaining the embedding representation for each cluster includes: Obtain the discourse within the target domain; Generate sentence embeddings for each domain discourse; The sentence embedding of each domain discourse is input into an unsupervised clustering model, which is configured to interpret the domain discourse and identify the multiple clusters in the feature space of the domain discourse; Using the unsupervised clustering model, the sentence embeddings of each domain discourse are classified into one of the plurality of clusters based on the similarity and difference between the features of the sentence embeddings and the features of the sentence embeddings within each cluster; Calculate the centroid of each of the plurality of clusters; and Output the embedding representation and centroid of each of the plurality of clusters.

17. The system of claim 15, wherein, The action further includes: The z-score of the utterance is calculated based on the distance or density deviation between the sentence embedding of the utterance and the embedding representation of the neighboring clusters; and The second probability regarding whether the utterance belongs to the target domain is determined by applying a sigmoid function to the z-score.

18. The system of claim 15, wherein, The sentence embedding of the discourse is generated using an embedding model that maps natural language elements, including sentences, words, and n-grams, to an array of numbers, wherein each of the natural language elements is represented as a single point in a vector space.

19. The system of claim 15, wherein: Determining the similarity or difference between the sentence embedding of the utterance and each embedding representation of each cluster includes: (i) calculating the absolute difference between the sentence embedding of the utterance and each embedding representation of each cluster; (ii) inputting the absolute difference, the sentence embedding of the utterance, and the embedding representation of each cluster into a wide and deep learning network, wherein the wide and deep learning network includes a linear model and a deep neural network; (iii) using the linear model and the absolute difference to predict a wide-base probability regarding whether the utterance belongs to the target domain; and (iv) using the deep neural network, the sentence embedding of the utterance, and the embedding representation of each cluster to determine the similarity or difference between the sentence embedding of the utterance and each embedding representation of each cluster; and The prediction of the first probability includes: using the final layer of the wide and deep learning network to evaluate the wide base probability and the similarity or difference between the sentence embedding of the utterance and each embedding representation of each cluster.

20. The system of claim 19, wherein: The linear model includes multiple model parameters trained using a training dataset; The training dataset includes the absolute difference between sentence embeddings of utterances and each embedding representation of each cluster of in-domain utterances from multiple domains; During training the linear model using the training dataset, a hypothesis function is used to learn the linear relationship between the sentence embeddings of the utterance and each embedding representation of each cluster; During the learning of the linear relationship, the plurality of model parameters are learned to minimize the first loss function; The deep neural network includes multiple model parameters trained using the training dataset; and During the training of the deep neural network using the training dataset, the high-dimensional features of the sentence embeddings of the in-domain utterances are converted into lower-dimensional vectors, which are then concatenated with features from the in-domain utterances and fed into the hidden layers of the deep neural network. The values ​​of the lower-dimensional vectors are randomly initialized and learned together with the plurality of model parameters to minimize a second loss function.

Images