Distance-based logit values for natural language processing

Abstract

Techniques for classifying utterances and messages input to a chatbot system using logit values in natural language processing. A method can include a chatbot system receiving an utterance generated by a user interacting with the chatbot system. The chatbot system can input the utterance into a machine learning model comprising a set of binary classifiers. Each binary classifier in the set of binary classifiers can be associated with a modified logit function. The method can further include the machine learning model generating a set of distance-based logit values for the utterance using the modified logit function. The method can further include the machine learning model applying an enhanced activation function to the set of distance-based logit values to generate a predicted output. The method can further include the chatbot system classifying the utterance as being associated with a particular classification based on the predicted output.

Description

[0001] Cross-reference to related applications

[0002] This application claims the benefit of U.S. Provisional Application No. 63 / 119,459, filed November 30, 2020, the contents of which are incorporated herein by reference in their entirety for all purposes. Technical Field

[0003] This disclosure generally relates to chatbot systems, and more specifically to techniques for determining logit values ​​for classifying utterances and messages input to a chatbot system in natural language processing. Background Technology

[0004] To achieve immediate responses, many users around the world utilize instant messaging or chat platforms. Organizations frequently use these platforms to engage in real-time conversations with customers (or end users). However, hiring service personnel to communicate with customers or end users in real time can be very expensive for organizations. Chatbots, or bots, have begun to be developed to simulate conversations with end users, especially via the internet. End users can interact with bots through messaging applications they have already installed and are using. Intelligent bots (typically powered by artificial intelligence (AI)) can communicate more intelligently and context-sensitively in real-time conversations, thus allowing for more natural conversations between the bot and the end user to improve the conversational experience. Instead of end users learning a fixed set of keywords or commands that the bot knows how to respond, intelligent bots can understand the end user's intent based on natural language user utterances and respond accordingly. Summary of the Invention

[0005] Techniques are provided for classifying utterances and messages input to a chatbot system in natural language processing. One approach may include a chatbot system receiving utterances generated by a user interacting with the chatbot system. The utterances may include text data converted from speech input by the user. The chatbot system may input the utterances into a machine learning model comprising a set of binary classifiers. Each binary classifier in the set of binary classifiers: (i) may be configured to estimate the probability of the utterance corresponding to a category within that set of categories; and (ii) may be associated with a modified logit function that transforms the probability of that category into a real number. The modified logit function may be the logarithm of the probability corresponding to that category, the logarithm of which is determined based on a distance measured between the probability of that category and the centroid of the distribution associated with that category.

[0006] The method can further include the machine learning model generating a set of distance-based logit values for the utterance. In some instances, each distance-based logit value of the set of distance-based logit values is generated by: (i) determining, by a respective binary classifier of the set of binary classifiers, a respective probability that the utterance corresponds to a classification associated with the respective binary classifier; and (ii) mapping, by the respective binary classifier and based on the modified logit function, the respective probability to a distance-based logit value. The mapping can include using a respective distance measured between the respective probability and a centroid of a distribution associated with the classification associated with the respective binary classifier.

[0007] The method can further include the machine learning model applying an enhanced activation function to the set of distance-based logit values to generate a predicted output. The predicted output can identify a normalized probability that the predicted utterance corresponds to a particular classification of the set of classifications within a probability distribution. The enhanced activation function can include a learning parameter for normalizing an initial output of the enhanced activation function to determine the normalized probability.

[0008] The method can further include the chatbot system classifying the utterance as associated with the particular classification based on the predicted output.

[0009] Techniques are also provided for training a machine learning model that uses distance-based logit values to classify utterances and messages. A method can include a training subsystem receiving a training dataset. The training dataset can include a plurality of utterances generated by users interacting with a chatbot system. At least one utterance of the plurality of utterances can include textual data converted from a speech input of a user. The training subsystem can access a machine learning model that includes a set of binary classifiers. Each binary classifier of the set of binary classifiers: (i) can be configured to estimate a probability that an utterance corresponds to a classification of the set of classifications; and (ii) can be associated with a modified logit function that transforms the probability for the classification to a real number. In some instances, the modified logit function is a logit of an odds corresponding to the probability of the classification, the logit of the odds being determined based on a distance measured between the probability of the classification and a centroid of a distribution associated with the classification.

[0010] The method can further include the training subsystem training the set of binary classifiers of the machine learning model using the training dataset. The training can include, for each binary classifier in the set of binary classifiers, determining a respective probability that an utterance in the training dataset corresponds to a classification associated with the binary classifier. The training can further include mapping the respective probability of the classification associated with the binary classifier to a distance-based logit value based on the modified logit function. The mapping can include using a respective distance measured between the respective probability and a centroid of a distribution associated with the classification associated with the binary classifier.

[0011] The training can further include applying an enhanced activation function to the distance-based logit value to generate a predicted output of the classification. The predicted output can identify a normalized probability that a predicted utterance corresponds to the classification within the probability distribution. The enhanced activation function can include a learned parameter for normalizing an initial output of the enhanced activation function to determine the normalized probability. The training can further include applying an enhanced loss function to the predicted output and an expected output of the utterance to determine a total loss. The enhanced loss function can include a set of loss terms for determining the total loss, where the set of loss terms includes: (i) a binary cross-entropy loss term; (ii) a mean squared error term; (iii) a margin loss term; and (iv) a threshold loss term. The training can further include adjusting one or more parameters of the binary classifier, where the one or more parameters include the learned parameter of the enhanced activation function.

[0012] Once the training of the machine learning model has been performed, the method can further include the training subsystem deploying the trained machine learning model.

[0013] In some embodiments, a system is provided that includes one or more data processors and a non-transitory computer-readable storage medium containing instructions that, when executed by the one or more data processors, cause the one or more data processors to perform part or all of one or more methods disclosed herein.

[0014] In some embodiments, a computer program product is provided that is tangibly embodied in a non-transitory machine-readable storage medium and includes instructions configured to cause one or more data processors to perform part or all of one or more methods disclosed herein.

[0015] Some embodiments of the present disclosure include a system comprising one or more data processors. In some embodiments, the system includes a non-transitory computer-readable storage medium containing instructions that, when executed on the one or more data processors, cause the one or more data processors to perform some or all of one or more methods disclosed herein and / or some or all of one or more processes. Some embodiments of the present disclosure include a computer program product tangibly embodied in a non-transitory machine-readable storage medium including instructions configured to cause one or more data processors to perform some or all of one or more methods disclosed herein and / or some or all of one or more processes.

[0016] The techniques described above and below can be implemented in a number of ways and in various contexts. Various example implementations and contexts are provided below in greater detail, with reference to the following drawings. However, the following implementations and contexts are merely some of the many. BRIEF DESCRIPTION OF DRAWINGS

[0017] Figure 1 is a simplified block diagram of a distributed environment incorporating example embodiments.

[0018] Figure 2 is a simplified block diagram of a computing system implementing a host robot in accordance with some embodiments.

[0019] Figure 3 is a simplified block diagram of a computing system implementing a skill robot in accordance with some embodiments.

[0020] Figure 4 is a simplified block diagram of a chatbot training and deployment system in accordance with various embodiments.

[0021] Figure 5 illustrates a schematic diagram of an example neural network in accordance with some embodiments.

[0022] Figure 6 shows a flowchart illustrating an example process for modifying a logit function to determine a distance-based logit value in accordance with some embodiments.

[0023] Figure 7 shows a flowchart illustrating an example process for modifying an activation function to estimate a probability that an utterance corresponds to a particular classification in accordance with some embodiments.

[0024] Figure 8 shows a flowchart illustrating an example process for modifying a loss function to estimate a probability that an utterance corresponds to a particular classification in accordance with some embodiments.

[0025] Figure 9is a flowchart illustrating a process for training a machine learning model that uses distance-based logit values to classify utterances, according to some embodiments.

[0026] Figure 10 is a flowchart illustrating a process for classifying an intent of an utterance or message using distance-based logit values, according to some embodiments.

[0027] Figure 11 A simplified diagram of a distributed system for implementing various embodiments is depicted.

[0028] Figure 12 is a simplified block diagram of one or more components of a system environment, in accordance with various embodiments, by which services provided by one or more components of an embodiment system can be offered as cloud services.

[0029] Figure 13 An example computer system that can be used for implementing various embodiments is illustrated. DETAILED DESCRIPTION

[0030] In the following description, for the purposes of explanation, specific details are set forth in order to provide a thorough understanding of certain embodiments. It will be apparent, however, that various embodiments can be practiced without

[0031] A. SUMMARY

[0032] 1. Intent

[0033] A digital assistant is an artificial intelligence-driven interface that helps users accomplish various tasks in natural language conversations. For each digital assistant, a customer can assemble one or more skills. A skill (also described herein as a chatbot, a bot, or a skillbot) is a separate robot that focuses on a specific type of task, such as tracking inventory, submitting time cards, and creating expense reports. When an end user engages with a digital assistant, the digital assistant evaluates the end user input and routes the conversation to and from the appropriate chatbot. The digital assistant can be made available to the end user through various channels, such as Messenger (instant messaging), SKYPE messenger, or short message service (SMS).

[0034] Intents are the intents that users convey to chatbots via user requests and statements, also referred to as utterances (e.g., get account balance, make a purchase, etc.). As used herein, an utterance or message can refer to a set of words (e.g., one or more sentences) exchanged during a session with a chatbot. An intent can be created by providing a name that describes some user action (e.g., order a pizza) and compiling a set of real-life user statements or utterances that are typically associated with triggering the action. Because the cognition of a chatbot is derived from these intents, each intent can be created from a robust data set (one to two dozen utterances) and varied so that the chatbot can interpret ambiguous user input. A rich set of utterances enables the chatbot to understand what the user wants when it receives messages such as “Forget this order!” or “Cancel delivery!” that mean the same thing but are expressed differently. In sum, intents and utterances that belong to intents make up the training corpus of a chatbot. By training a model with the corpus, a customer can essentially turn the model into a reference tool for parsing end-user input into a single intent. A customer can improve the acuity of the cognition of a chatbot through multiple rounds of intent testing and intent training.

[0035] While the examples described herein focus on intent prediction, it will be appreciated that other classification tasks can be similarly handled. For example, a chatbot system can include a classifier to address utterances for other levels, such as skill bot invocation.

[0036] However, building a chatbot that can determine the intent of an end user based on user utterances is a challenging task, in part due to the subtlety and ambiguity of natural language and the dimensionality of the input space (e.g., possible user utterances) and the size of the output space (number of intents). Illustrative examples of this difficulty arise from features of natural language, such as the use of euphemisms, synonyms, or ungrammatical speech to express an intent. For example, an utterance can express an intent to order a pizza without explicitly mentioning pizza, ordering, or delivery. For example, in some regional dialects, “pizza” is referred to as “pie.” These trends in natural language, such as imprecision or variability, create uncertainty and introduce a confidence as an intent prediction parameter, rather than, for example, an explicit indication of intent by including a keyword. As such, to improve the performance of a chatbot and the user experience with a chatbot, a chatbot can need to be trained, monitored, debugged, and retrained. In traditional systems, a training system is provided for training and retraining machine learning models of a digital assistant or chatbot in spoken language understanding (SLU) and natural language processing (NLP).

[0037] 2. Determining intent using a machine learning model

[0038] At one or more points, the chatbot system can provide an utterance as input to a neural network model that maps an output to a probability distribution using a logistic regression function. For example, for classification, ranking a set of outputs in a probability distribution allows for predicting an intent invoked in an utterance. An accurate prediction, in turn, lets the chatbot accurately interact with an end user. In this sense, accuracy depends at least in part on mapping an output of a neural network classifier to a probability distribution.

[0039] To map an output of a neural network machine learning model to a probability distribution, a logit value is determined based on the input. A logit value (also referred to as a “logit”) is a value output by a logit function of a network layer of the machine learning model. The logit value can represent a likelihood that an utterance corresponds to a particular classification. The logit function is a logarithm of the likelihood of a particular classification (e.g., order_pizza intent classification, unresolvable classification) and transforms an output of the machine learning model to a corresponding logit value that fits within a probability distribution. The probability distribution ranges between (-∞, +∞).

[0040] The logit value can then be provided as input to an activation function (e.g., a softmax function) to generate a predicted likelihood that the input (e.g., an utterance) corresponds to a particular classification in a set of classifications. In the case of intent classification, the input can be a weighted output of the neural network model. For example, a numerical output corresponding to a particular classification (e.g., order_pizza) is generated by processing an input utterance (e.g., “I want to grab a pie”) through one or more intermediate layers of a polynomial machine learning model. The output can be processed by a logit function of the particular classification to generate a logit value of 9.4. An activation function can then be applied to the logit value to determine a probability value (e.g., 0.974) ranging between 0 and 1 that indicates the input utterance corresponds to the order_pizza classification. Utterances invoking any other intent, in addition to intents that the classifier is trained to recognize, would correspond to out-of-range or out-of-domain utterances. For example, the utterance “how is the weather today?” can be considered out-of-range for a classifier trained to predict whether the utterance specifies an order for a particular food item.

[0041] Classification accuracy can be further improved by weighting one or more parameters of the logit function. For example, each intent classification can be associated with a logit function that can be weighted by a centroid of the intent classification. As used herein, the term "centroid" refers to a positioning of a central location of a cluster used to classify an utterance, where the cluster identifies data corresponding to a particular end-user intent classification. In some instances, the centroid is determined using data from a corresponding set of utterance data (e.g., a training data set). Weighting the logit function by the centroid of the distribution allows the logit function to more accurately predict a classification for a given utterance, especially when the utterance is in-domain or in-range (e.g., an utterance that the system has been trained to recognize).

[0042] 3. Overconfidence issues associated with machine learning models

[0043] Deep neural networks suffer from an overconfidence issue (of confidence scores), where the confidence scores generated by the neural network for a classification can become uncorrelated with the true confidence scores. Deep neural network models can even produce highly confident, incorrect classification predictions when the actual input is not well represented by the training data used to train the neural network model (i.e., the actual sample is drawn from outside the distribution observed during training). Thus, while deep neural networks are desirable to use due to their increased accuracy, the overconfidence issue associated with deep neural networks must be addressed to avoid performance issues with the neural network.

[0044] Conventional techniques using softmax functions do not effectively address the above overconfidence issue. In some instances, the softmax function does not take into account the individual strengths of the output signals of the neural network model, leaving the overconfidence issue to persist. Rather, the softmax function takes into account the mutual comparison of the output signals to each other, which can result in overestimating the confidence of the predictions. This is especially true for logit values where only one element is positive. As an illustrative example, consider logit values for a 3-way classification, with a first utterance of [10, 3, 1] and a second utterance of [3, -2, -4]. In the second utterance, only one of the three classifications is positive (e.g., 3), which indicates that the model is not confident about the prediction (e.g., 10) with respect to the first utterance. However, after softmax normalization, the confidence score of the positive output is amplified, resulting in confidence scores close to 100% for both utterances. In this way, the softmax normalization overestimates the model confidence in the prediction and impacts the prediction accuracy.

[0045] Other traditional techniques have been attempted to address overconfidence issues. For example, n-ary classifiers have been implemented as an alternative to n-way classifiers for n classes. The n-ary classifier implements “n” classifiers that classify an input utterance as “in” or “out” for each intent class. In this way, the n-ary approach corresponds to a “one-versus-all” classification rather than a “one-among-many” approach. Additionally, rather than computing a classification cross-entropy loss as in the softmax approach, the n-ary approach computes a binary cross-entropy loss for each class. For example, the n-ary classifier can use a sigmoid function rather than a softmax function as the output activation function.

[0046] While using n-ary classifiers can address the softmax normalization’s limitation regarding the strength of individual model output signals, n-ary classifiers can present their own drawbacks. For example, the accuracy of n-ary classifiers can be lower than the softmax approach. Additionally, n-ary classifiers can fail to meet a threshold confidence margin for predicting a single class. For example, a minimum confidence margin of 10% can be imposed between the most likely prediction and the second likely prediction to classify an utterance. Since the softmax function distributes all classes in a probability distribution that sums to 100%, if the highest prediction confidence score >= 55%, the second best prediction will <= 45% and thus will meet the minimum 10% margin. However, in the case of the n-ary classifier approach, the confidence score for each class is determined independently and defined to sum to 100%. For example, in an utterance that includes multiple different intents, the importance of the confidence margin is emphasized. Thus, it can be more likely that the confidence margin is not met and the utterance is incorrectly classified as unresolvable, out-of-domain, or out-of-scope.

[0047] Accordingly, it is challenging for a machine learning model to accurately determine its confidence when it predicts a particular class for a given utterance. This can lead to misinterpretation of the classification output in cases of overconfidence.

[0048] 4. Distance-based Logit value for classifying an utterance as having an unresolvable intent

[0049] To overcome the above deficiencies, the present technology includes a system and method for using distance-based logits in a machine learning model to accurately estimate the probability that an utterance corresponds to a particular classification. This can result in more accurate classification results from the machine learning model, i.e., improved classification of utterances as the machine learning model can more often correctly classify utterances. The method includes receiving, by an intent classifier of a chatbot system, one or more utterances or messages. The method includes determining, by the chatbot system, an intent of the one or more utterances or messages using a set of binary classifiers (e.g., n binary classifiers). Each binary classifier in the set of binary classifiers includes a modified logit function for outputting a corresponding distance-based logit value. For example, the distance-based logit value can be computed by using a modified logit function for a given classification “C i ”. The modified logit function includes determining a distance between a centroid “W i ” of the classification and an intermediate output “f(x)” of the corresponding binary classifier. In some instances, the distance between the centroid and the output vector is measured by different distance metrics, such as Euclidean distance, cosine distance, etc.

[0050] In some instances, the accuracy of the probability estimates is further improved by implementing an enhanced activation function. The enhanced activation function includes scaling the distance-based logits using a scaling value. The scaling value can be a fixed value or a learned parameter of the intent classifier, where the fixed value can be determined by hyperparameter tuning and the value of the parameter is learned during model training. Additionally or alternatively, the machine learning model can include an enhanced loss function that incorporates multiple additional loss terms to improve the accuracy of the probability estimates for a given utterance. The loss terms of the enhanced loss function can include one or more of: (i) a binary cross-entropy loss term; (ii) a mean squared error term; (iii) a margin loss term; and (iv) a threshold loss term. In some instances, each term of the loss function is weighted by a corresponding weighting factor. The weighting factors can be tuned as part of the hyperparameter tuning of the intent classifier.

[0051] Accordingly, this incorporation of distance-based logit values can facilitate the intent classifier to improve its accuracy in predicting the probability that an utterance corresponds to a particular intent classification. Additionally, the distance-based logit values can normalize the output of the n-ary classifier, thereby reducing the overconfidence problem that traditional softmax function-based techniques can encounter. The distance-based logit values can also be an improvement over traditional n-ary “one-versus-all” classifiers, as the distance-based logit values can allow the machine learning model to provide a prediction of an intent in one or more utterances with a higher than threshold confidence and with a minimum margin between the most confident intent and the second most confident intent.

[0052] B. Robot and Analytics System

[0053] Robots (also referred to as skills, chatbots, conversational bots, or conversational robots) are computer programs that can perform conversations with end users. The robots can typically respond to natural language messages (e.g., questions or comments) through a messaging application using natural language messages. An enterprise can use one or more robot systems to communicate with end users through a messaging application. The messaging application (which can be referred to as a channel) can be an end user preferred messaging application that the end user already has installed and is familiar with. Accordingly, the end user does not need to download and install a new application in order to chat with the robot system. The messaging application can include, for example, over-the-top (OTT) messaging channels (such as Facebook Messenger, Facebook WhatsApp, WeChat, Line, Kik, Telegram, Talk, Skype, Slack, or SMS), virtual personal assistants (such as Amazon Dot, Echo, or Show, Google Home, Apple HomePod, etc.), native or hybrid extended mobile and web application extensions / responsive mobile or web applications with chat functionality, or voice-based inputs (such as devices or applications with interfaces that use Siri, Microsoft Cortana, Google Voice, or other voice inputs for interaction).

[0054] In some examples, a bot system can be associated with a uniform resource identifier (URI). The URI can use a string of characters to identify the bot system. The URI can be used as a webhook for one or more messaging application systems. The URI can include, for example, a uniform resource locator (URL) or a uniform resource name (URN). The bot system can be designed to receive messages (e.g., hypertext transfer protocol (HTTP) post call messages) from the messaging application systems. The HTTP post call messages can be directed to the URI from the messaging application systems. In some embodiments, the messages can be different from the HTTP post call messages. For example, the bot system can receive messages from a short message service (SMS). While the discussion herein can refer to the communications received by the bot system as messages, it should be understood that the messages can be HTTP post call messages, SMS messages, or any other type of communication between the two systems.

[0055] End users can interact with bot systems through conversational interactions (sometimes referred to as conversational user interfaces (UIs)), much like interactions between people. In some instances, the interactions can include an end user saying "Hello" to a bot and the bot responding with "Hi" and asking the end user how the bot can assist. In some instances, the interactions can also be transactional interactions with, for example, a banking bot, such as transferring money from one account to another; informational interactions with, for example, an HR bot, such as querying a vacation balance; or interactions with, for example, a retail bot, such as discussing returning a purchased item or seeking technical support.

[0056] In some embodiments, a bot system can intelligently handle end user interactions without interaction with an administrator or developer of the bot system. For example, an end user can send one or more messages to a bot system in order to achieve a desired goal. The messages can include some content, such as text, emojis, audio, images, video, or other methods of conveying a message. In some embodiments, the bot system can convert the content into a standardized form (e.g., a representational state transfer (REST) call to an enterprise service with appropriate parameters) and generate a natural language response. The bot system can also prompt the end user for additional input parameters or request other additional information. In some embodiments, the bot system can also initiate a conversation with the end user, rather than passively responding to end user utterances. Described herein are various techniques for identifying explicit invocations of a bot system and determining the inputs of the invoked bot system. In certain embodiments, explicit invocation analysis is performed by a host bot based on detecting an invocation name in an utterance. In response to detecting the invocation name, the utterance can be refined for input to a skill bot associated with the invocation name.

[0057] A conversation with a robot can follow a particular conversation flow that includes a plurality of states. The flow can define what is going to happen next based on input. In some embodiments, a robot system can be implemented using a state machine that includes user-defined states (e.g., end user intents) and actions to take in or between states. The conversation can take different paths based on end user input, which can influence the decisions made by the robot for the flow. For example, at each state, based on end user input or utterance, the robot can determine the intent of the end user in order to determine the next appropriate action to take. As used herein and in the context of an utterance, the term "intent" refers to the intent of the user providing the utterance. For example, a user can intend to have the robot engage in a conversation for ordering a pizza, such that the intent of the user can be expressed by the utterance "Order pizza." The user intent can refer to a particular task that the user wants the chatbot to perform on behalf of the user. Thus, the utterance can be expressed as a question, command, request, etc. that reflects the intent of the user. The intent can include a goal that the end user wants to accomplish.

[0058] In the context of a chatbot's configuration, the term "intent" as used herein refers to configuration information for mapping a user's utterance to a particular task / action or a particular kind of task / action that the chatbot can perform. To distinguish the intent of an utterance (i.e., the user intent) from the intent of a chatbot, the latter is sometimes referred to herein as a "robot intent." A robot intent can include a set of one or more utterances associated with the intent. For example, an intent to order a pizza can have various permutations of utterances that express the desire to place an order to purchase a pizza. These associated utterances can be used to train an intent classifier of the chatbot to enable the intent classifier to subsequently determine whether an input utterance from a user matches the intent to order a pizza. A robot intent can be associated with one or more dialog flows for initiating a conversation with a user and at a certain state. For example, a first message for the intent to order a pizza can be the question "What kind of pizza would you like?" In addition to the associated utterances, a robot intent can further include named entities related to the intent. For example, the intent to order a pizza can include variables or parameters for performing the task of ordering a pizza, such as topping 1, topping 2, pizza type, pizza size, pizza quantity, etc. The values of the entities are typically obtained through a conversation with the user.

[0059] In some instances, the utterance is analyzed to determine if it contains a call name for the skill robot. If no call name is found, the utterance is considered an indirect call, and the process continues using an intent classifier, such as a trained model. If a call name is determined to exist, the utterance is considered an explicit call, and the process continues to determine which parts of the utterance are associated with that call name. In instances where a trained model is invoked, the entire received utterance is fed as input to the intent classifier.

[0060] The intent classifier for receiving utterances can be the main robot's intent classifier (e.g., Figure 2 The intent classifier (242) can be a machine learning-based or rule-based classifier that is trained using keyword-enhanced data to determine whether the intent of the utterance matches a system intent (e.g., exit, help) or a specific skill bot. As explained herein, intent analysis performed by the master bot can be limited to matching to a specific skill bot without determining which intent within that specific skill bot is the best match for the utterance. Thus, the intent classifier of the received utterance can identify the specific skill bot to invoke. Alternatively, if the utterance represents a specific system intent (e.g., the utterance contains the words “exit” or “help”), the intent classifier of the received utterance can identify that specific system intent to trigger a conversation between the master bot and the user based on the dialogue flow configured for that specific system intent.

[0061] An intent classifier can use a binary cross-entropy loss function and distance-based logit values ​​as part of its prediction of utterance intent. In this approach, distance refers to the distance calculated between the output vector of the intent classifier and the centroid of a specific intent. The logit values ​​for different intents are then used as inputs to a loss function included in the model's total loss function. For example, the total loss function can include a binary cross-entropy loss term, a margin loss term, and a threshold loss term. In this approach, the intent classifier can predict the intent when the most likely intent satisfies the minimum difference between the most likely and second most likely intents, and when the most likely intent satisfies a minimum threshold confidence (related to the distance measurement from the centroid of the intent).

[0062] In instances where a call name is present, one or more explicit call rules are applied to determine which portions of the utterance are associated with the call name. The determination can be based on an analysis of the sentence structure of the utterance using POS tags, dependency information, and / or other extracted information received with the utterance. For example, the portions associated with the call name can be the noun phrase that includes the call name or the prepositional object that corresponds to the call name. Any portions of the utterance that are determined to be associated with the call name are removed as based on the processing. Other portions of the utterance that are not needed to convey the meaning of the utterance (e.g., prepositions) can also be removed. The removal of certain portions of the utterance produces an input to the skill bot that is associated with the call name. If any portions of the received utterance remain after the removal, the remaining portions form a new utterance to input to the skill bot, e.g., as a text string. Otherwise, if the received utterance is completely removed, the input can be an empty string. Thereafter, the skill bot associated with the call name is invoked and provided with the produced input.

[0063] Upon receiving the produced input, the invoked skill bot processes the input, e.g., by performing intent analysis using the skill bot’s intent classifier that has been trained to identify a bot intent that matches the user intent represented in the input. The identification of a matching bot intent can result in the skill bot performing a particular action or starting a conversation with the user in accordance with a dialog flow associated with the matching bot intent. For example, if the input is an empty string, the conversation can be started in a default state defined for the dialog flow, e.g., a welcome message. Alternatively, if the input is not an empty string, the conversation can be started in some intermediate state, e.g., because the input contains values for entities or some other information that the skill bot no longer needs to ask the user for as it has received that information as part of the input. As another example, the skill bot can decide that it cannot process the input (e.g., because the confidence score for each bot intent configured for the skill bot is below some threshold). In this case, the skill bot can return the input to the host bot for processing (e.g., using the host bot’s intent classifier for intent analysis), or the skill bot can prompt the user for clarification.

[0064] 1. Overall Environment

[0065] Figure 1 is a simplified block diagram of an environment 100 incorporating a chatbot system in accordance with certain embodiments. The environment 100 includes a digital assistant builder platform (DABP) 102 that enables users of the DABP 102 to create and deploy digital assistant or chatbot systems. The DABP 102 can be used to create one or more digital assistants (or DAs) or chatbot systems. For example, as shown in Figure 1As shown, a user 104 representing a particular enterprise can use the DABP 102 to create and deploy a digital assistant 106 for users of the particular enterprise. For example, a bank can use the DABP 102 to create one or more digital assistants for use by customers of the bank. Multiple enterprises can use the same DABP 102 platform to create digital assistants. As another example, an owner of a restaurant (e.g., a pizza shop) can use the DABP 102 to create and deploy a digital assistant that enables customers of the restaurant to order food (e.g., order a pizza).

[0066] For purposes of the present disclosure, a "digital assistant" is an entity that helps users of the digital assistant accomplish various tasks through natural language conversations. A digital assistant can be implemented using only software (e.g., the digital assistant is a digital entity implemented using programs, code, or instructions executable by one or more processors), using hardware, or using a combination of hardware and software. A digital assistant can be embodied or implemented in various physical systems or devices such as computers, mobile phones, watches, appliances, vehicles, etc. Digital assistants are sometimes also referred to as chatbot systems. Thus, for purposes of the present disclosure, the terms digital assistant and chatbot system are interchangeable.

[0067] A digital assistant (such as the digital assistant 106 built using the DABP 102) can be used to perform various tasks via natural language based conversations between the digital assistant and its users 108. As part of a conversation, a user can provide one or more user inputs 110 to the digital assistant 106 and obtain a returned response 112 from the digital assistant 106. A conversation can include one or more of the inputs 110 and responses 112. Via these conversations, a user can request one or more tasks to be performed by the digital assistant, and in response, the digital assistant is configured to perform the tasks requested by the user and respond to the user with appropriate responses.

[0068] A user input 110 is typically in the form of natural language and is referred to as an utterance. A user utterance 110 can be in textual form, such as when a user types a sentence, question, piece of text, or even a single word and provides it as input to the digital assistant 106. In some embodiments, a user utterance 110 can be in the form of audio input or speech, such as when a user speaks or says something that is provided as input to the digital assistant 106. An utterance is typically in the form of a language spoken by the user 108. For example, the utterance can be in English or some other language. When the utterance is in the form of speech, the speech input is converted into a textual form of the utterance in that particular language and then the textual utterance is processed by the digital assistant 106. Various speech-to-text processing techniques can be used to convert the speech or audio input into a textual utterance that is then processed by the digital assistant 106. In some embodiments, the speech-to-text conversion can be done by the digital assistant 106 itself.

[0069] utterances. The digital assistant 106 is configured to apply natural language understanding (NLU) techniques to the utterances to understand the meaning of the user input. As part of the NLU processing of the utterances, the digital assistant 106 is configured to perform processing for understanding the meaning of the utterances that involves identifying one or more intents and one or more entities corresponding to the utterances. Upon understanding the meaning of the utterances, the digital assistant 106 can perform one or more actions or operations in response to the understood meaning or intent. For the purposes of this disclosure, it is assumed that these utterances are textual utterances that have been directly provided by the user 108 of the digital assistant 106 or are the result of converting inputted speech utterances into textual form. However, this is not intended to be limiting or constraining in any way.

[0070] For example, the input of the user 108 can request ordering a pizza by providing an utterance such as "I want to order a pizza." Upon receiving such an utterance, the digital assistant 106 is configured to understand the meaning of the utterance and take appropriate action. The appropriate action can involve, for example, responding to the user with a request for user input regarding the type of pizza the user desires to order, the size of the pizza, any toppings for the pizza. The responses provided by the digital assistant 106 can also be in natural language form and typically in the same language as the input utterance. As part of generating these responses, the digital assistant 106 can perform natural language generation (NLG). To have the user order the pizza via a conversation between the user and the digital assistant 106, the digital assistant can guide the user to provide all the necessary information for the pizza order and then cause the pizza to be ordered at the end of the conversation. The digital assistant 106 can end the conversation by outputting information to the user indicating that the pizza has been ordered.

[0071] At a conceptual level, the digital assistant 106 performs various processing in response to utterances received from a user. In some embodiments, the processing involves a series of processing steps or processing step pipelines that include, for example, understanding the meaning of an input utterance (sometimes referred to as natural language understanding (NLU), determining actions to perform in response to the utterance, causing the actions to be performed where appropriate, generating responses to output to the user in response to the user utterance, outputting the responses to the user, etc. The NLU processing can include parsing the received input utterance to understand the structure and meaning of the utterance, refining and reformulating the utterance to develop a better understandable form (e.g., logical form) or structure for the utterance. Generating the responses can include using NLG techniques.

[0072] The NLU processing performed by a digital assistant, such as the digital assistant 106, can include various NLP-related processing such as sentence syntax analysis (e.g., tokenization, categorization by inflection, identifying part-of-speech tags of a sentence, identifying named entities in a sentence, generating dependency trees to represent sentence structure, dividing a sentence into clauses, analyzing individual clauses, resolving references, performing chunking, etc.). In certain embodiments, the NLU processing, or portions thereof, is performed by the digital assistant 106 itself. In some other embodiments, the digital assistant 106 can use other resources to perform portions of the NLU processing. For example, the syntax and structure of a sentence of an input utterance can be identified by processing the sentence using a syntax analyzer, a part-of-speech tagger, and / or a named entity recognizer. In one implementation, for the English language, a sentence structure and syntax is analyzed using a syntax analyzer, a part-of-speech tagger, and a named entity recognizer provided by the Stanford Natural Language Processing (NLP) group. These are provided as part of the Stanford CoreNLP toolkit.

[0073] While various examples provided in this disclosure illustrate utterances in the English language, this is meant only as an example. In certain embodiments, the digital assistant 106 is also capable of processing utterances in languages other than English. The digital assistant 106 can provide subsystems (e.g., components implementing NLU functionality) that are configured to perform processing for different languages. These subsystems can be implemented as pluggable units that can be invoked from an NLU core server using service calls. This makes the NLU processing flexible and extensible for each language, including allowing different processing orders. Language packs can be provided for individual languages, where a language pack can register a list of subsystems that can provide services from the NLU core server.

[0074] A digital assistant, such as the digital assistant 106 depicted in FIG. 1, can be made available or accessible to its users 108 through a variety of different channels, such as but not limited to via certain applications, via social media platforms, via various messaging services and applications, and other applications or channels. Figure 1 A digital assistant, such as the digital assistant 106 depicted in FIG. 1, can be made available or accessible to its users 108 through a variety of different channels, such as but not limited to via certain applications, via social media platforms, via various messaging services and applications, and other applications or channels.

[0075] A digital assistant or chatbot system typically contains or is associated with one or more skills. In certain embodiments, these skills are individual chatbots (referred to as skill bots) that are configured to interact with users and accomplish a particular type of task, such as tracking inventory, submitting time cards, creating expense reports, ordering food, querying a bank account, making appointments, purchasing gadgets, etc. For example, for a digital assistant that is configured to interact with users in the English language, there can be a skill bot for ordering food, a skill bot for querying a bank account, a skill bot for making appointments, etc. Figure 1The depicted embodiment, the digital assistant or chatbot system 106 includes skills 116-1, 116-2, etc. For purposes of the present disclosure, the terms "a skill" and "skills" are used synonymously with the terms "a skillbot" and "skillbots," respectively.

[0076] Each skill associated with a digital assistant helps the user of the digital assistant complete a task through a conversation with the user, where the conversation can include a combination of textual or audio input provided by the user and responses provided by the skillbot. These responses can take the form of textual or audio messages to the user and / or using simple user interface elements (e.g., a list of choices) presented to the user for the user to select.

[0077] There are various ways in which a skill or skillbot can be associated with or added to a digital assistant. In some instances, a skillbot can be developed by an enterprise and then added to a digital assistant that uses the DABP 102. In other instances, a skillbot can be developed and created using the DABP 102 and then added to a digital assistant created using the DABP 102. In yet other instances, the DABP 102 provides an online digital store (referred to as a "skill store") that provides a plurality of skills that point to a wide variety of tasks. The skills provided through the skill store can also expose various cloud services. To add a skill to a digital assistant that is being generated using the DABP 102, a user of the DABP 102 can access the skill store via the DABP 102, select a desired skill, and instruct that the selected skill be added to the digital assistant created using the DABP 102. Skills from the skill store can be added to the digital assistant as-is or in modified form (e.g., a user of the DABP 102 can select and copy a particular skillbot provided by the skill store, customize or modify the selected skillbot, and then add the modified skillbot to the digital assistant created using the DABP 102).

[0078] Various different architectures can be used to implement a digital assistant or chatbot system. For example, in certain embodiments, a digital assistant created and deployed using the DABP 102 can be implemented using a master bot / sub (or child) bot paradigm or architecture. According to this paradigm, a digital assistant is implemented as a master bot that interacts with one or more sub-bots that are skillbots. For example, in the depicted embodiment, the digital assistant 106 includes a master bot 114 and skillbots 116-1, 116-2, etc. that are sub-bots of the master bot 114. In certain embodiments, the digital assistant 106 itself is considered to function as a master bot. Figure 1 The depicted embodiment, the digital assistant or chatbot system 106 includes skills 116-1, 116-2, etc. For purposes of the present disclosure, the terms "a skill" and "skills" are used synonymously with the terms "a skillbot" and "skillbots," respectively.

[0079] A digital assistant implemented in accordance with a primary-secondary robot architecture enables users of the digital assistant to interact with multiple skills through a unified user interface (i.e., via the primary robot). When a user engages with the digital assistant, the primary robot receives user input. The primary robot then performs processing to determine the meaning of the user input utterance. The primary robot then determines whether the task requested by the user in the utterance can be handled by the primary robot itself, or else the primary robot selects an appropriate skill robot to handle the user request and routes the conversation to the selected skill robot. This enables users to have conversations with the digital assistant through a common single interface, and still provide the ability to use multiple skill robots that are configured to perform specific tasks. For example, for a digital assistant developed for an enterprise, the primary robot of the digital assistant can interface with skill robots having specific functionality, e.g., a CRM robot for performing functions related to customer relationship management (CRM), an ERP robot for performing functions related to enterprise resource planning (ERP), an HCM robot for performing functions related to human capital management (HCM), etc. In this way, the end users or consumers of the digital assistant only need to know how to access the digital assistant through the common primary robot interface, and multiple skill robots are provided in the background to handle user requests.

[0080] In certain embodiments, in a primary robot / secondary robot infrastructure, the primary robot is configured to understand a list of available skill robots. The primary robot can access metadata that identifies various available skill robots, and for each skill robot, access the capabilities of the skill robot including the tasks that can be performed by the skill robot. Upon receiving a user request in the form of an utterance, the primary robot is configured to identify or predict a particular skill robot from the multiple available skill robots that can best serve or handle the user request. The primary robot then routes the utterance (or a portion of the utterance) to that particular skill robot for further processing. Thus, control flows from the primary robot to the skill robot. The primary robot can support multiple input channels and output channels.

[0081] While Figure 1 Embodiments of the digital assistant 106 are shown to include a primary robot 114 and skill robots 116-1, 116-2, and 116-3, but this is not intended to be limiting. The digital assistant can include various other components (e.g., other systems and subsystems) that provide the functionality of the digital assistant. These systems and subsystems can be implemented in software only (e.g., code, instructions stored on a computer-readable medium and executable by one or more processors), in hardware only, or in embodiments that use a combination of software and hardware.

[0082] DABP 102 provides infrastructure and various services and features that enable users of DABP 102 to create digital assistants (including one or more skill bots associated with a digital assistant). In some instances, a skill bot can be created by cloning an existing skill bot, e.g., cloning a skill bot provided by a skill store. As previously mentioned, DABP 102 provides a skill store or skill catalog that provides a plurality of skill bots for performing various tasks. A user of DABP 102 can clone a skill bot from the skill store. The cloned skill bot can be modified or customized as needed. In some other instances, a user of DABP 102 creates a skill bot from scratch using tools and services provided by DABP 102. As previously mentioned, the skill store or skill catalog provided by DABP 102 can provide a plurality of skill bots for performing various tasks.

[0083] In certain embodiments, at a high level, creating or customizing a skill bot involves the following steps:

[0084] (1) configuring settings for the new skill bot

[0085] (2) configuring one or more intents for the skill bot

[0086] (3) configuring one or more entities for the one or more intents

[0087] (4) training the skill bot

[0088] (5) creating a dialog flow for the skill bot

[0089] (6) adding custom components to the skill bot as needed

[0090] (7) testing and deploying the skill bot

[0091] Each of the above steps is briefly described below.

[0092] (1) configuring settings for the new skill bot - Various settings can be configured for a skill bot. For example, a skill bot designer can specify one or more invocation names for the skill bot being created. These invocation names can then be used by users of the digital assistant to explicitly invoke the skill bot. For example, a user can enter an invocation name in a user utterance to explicitly invoke the corresponding skill bot.

[0093] (2) configuring one or more intents and associated example utterances for the skill robot - a skill robot designer specifies one or more intents (also referred to as robot intents) for the skill robot being created. The skill robot is then trained based on these specified intents. These intents represent the categories or classifications that the skill robot is trained to infer for an input utterance. Upon receiving an utterance, the trained skill robot infers the intent of the utterance, where the inferred intent is selected from a predefined set of intents used to train the skill robot. The skill robot then takes appropriate action in response to the utterance based on the inferred intent for the utterance. In some instances, the intents of the skill robot represent tasks that the skill robot can perform for a user of the digital assistant. Each intent is given an intent identifier or intent name. For example, for a skill robot trained for a bank, the intents specified for the skill robot can include "CheckBalance," "TransferMoney," "DepositCheck," etc.

[0094] For each intent defined for the skill robot, the skill robot designer can also provide one or more example utterances that represent and illustrate the intent. These example utterances are intended to represent utterances that a user can input to the skill robot for the intent. For example, for the CheckBalance intent, example utterances can include "What's my savings account balance?" "How much is in my checking account?" "How much money do I have in my account," etc. Thus, various permutations of typical user utterances can be specified as example utterances for an intent.

[0095] These intents and their associated example utterances serve as training data for training the skill robot. Various different training techniques can be used. As a result of this training, a prediction model is generated that is configured to take an utterance as input and output the intent that the prediction model infers for the utterance. In some instances, an input utterance is provided to an intent analysis engine that is configured to use the trained model to predict or infer the intent of the input utterance. The skill robot can then take one or more actions based on the inferred intent.

[0096] (3) One or more entities are configured for one or more intents of the skill robot - in some instances, additional context can be needed to enable the skill robot to respond appropriately to user utterances. For example, there can be instances in which user input utterances resolve to the same intent in the skill robot. For example, in the above example, the utterances "What's my savings account balance?" and "How much is in my checking account?" both resolve to the same CheckBalance intent, but these utterances are different requests that request different things. To clarify such requests, one or more entities are added to the intent. Using the example of a banking skill robot, an entity called AccountType (which defines values called "checking" and "savings") can enable the skill robot to parse the user request and respond appropriately. In the above example, while the utterances resolve to the same intent, the values associated with the AccountType entity for the two utterances are different. This enables the skill robot to perform potentially different actions for the two utterances, despite the two utterances resolving to the same intent. One or more entities can be specified for certain intents configured for a skill robot. Thus, entities are used to add context to the intent itself. Entities help to more fully describe the intent and enable the skill robot to fulfill the user request.

[0097] In certain embodiments, there are two types of entities: (a) built-in entities provided by the DABP 102; and (2) custom entities that can be specified by the skill robot designer. The built-in entities are general-purpose entities that can be used with a variety of robots. Examples of built-in entities include, but are not limited to, entities related to time, date, address, number, email address, duration, recurring time period, currency, phone number, URL, etc. The custom entities are used for more customized applications. For example, for a banking skill, an AccountType entity can be defined by the skill robot designer to enable various banking transactions by checking the user's input for keywords such as checking, savings, credit card, etc.

[0098] (4) Training the skill bot - The skill bot is configured to receive user input in the form of utterances, parse or otherwise process the received input and identify or select an intent that is relevant to the received user input. As indicated above, the skill bot must be trained to do this. In certain embodiments, the skill bot is trained based on intents that are configured for the skill bot and example utterances associated with the intents (collectively, training data) so that the skill bot can parse a user input utterance into one of the intents that it is configured for. In certain embodiments, the skill bot uses a predictive model that is a machine learning model that is trained using the training data and that allows the skill bot to discern what a user is saying (or in some cases, is trying to say). The DABP 102 provides various different training techniques that can be used by a skill bot designer to train a skill bot, including various machine learning based training techniques, rule based training techniques, and / or combinations thereof. In certain embodiments, a portion of the training data (e.g., 80%) is used to train the skill bot model and another portion (e.g., the remaining 20%) is used to test or validate the model. Once trained, the trained model (sometimes also referred to as a trained skill bot) can be used to process and respond to user utterances. In certain cases, a user utterance can be a question that only requires a single answer and no further conversation. To handle such cases, a Q&A (question and answer) intent can be defined for the skill bot. This enables the skill bot to output a reply to the user request without having to update the dialog definition. A Q&A intent is created in a similar manner as a regular intent. The dialog flow for a Q&A intent can be different from the dialog flow for a regular intent.

[0099] (5) Creating a dialog flow for the skill bot - The dialog flow specified for the skill bot describes how the skill bot reacts when resolving different intents of the skill bot in response to received user input. The dialog flow defines the operations or actions that the skill bot will take, e.g., how the skill bot responds to a user utterance, how the skill bot prompts the user for input, how the skill bot returns data. The dialog flow is like a flowchart that the skill bot follows. The skill bot designer specifies the dialog flow using a language, such as a markdown language. In certain embodiments, the dialog flow for the skill bot can be specified using a YAML version called OBotML. The dialog flow definition for the skill bot acts as a model for the conversation itself, which is a model that enables the skill bot designer to choreograph the interaction between the skill bot and the user that the skill bot is serving.

[0100] In certain embodiments, the dialog flow definition for the skill bot contains the following three parts:

[0101] (a) Context section

[0102] (b) Default Transition section

[0103] 5 State section

[0104] Context section - The skill bot designer can define variables used in the conversation flow in the context section. Other variables that can be named in the context section include, but are not limited to: variables for error handling, variables for built-in entities or custom entities, user variables that enable the skill bot to recognize and save user preferences, etc.

[0105] Default Transition section - Transitions of the skill bot can be defined in the dialog flow state section or in the default transition section. Transitions defined in the default transition section act as a fallback and are triggered when no applicable transition is defined within the state or the conditions required to trigger a state transition are not met. The default transition section can be used to define a route that allows the skill bot to gracefully handle unexpected user actions.

[0106] State section - The dialog flow and its related operations are defined as a sequence of temporary states that manage the logic within the dialog flow. Each state node within the dialog flow definition names the component that provides the functionality required at that point in the conversation. Thus, states are built around components. The state contains properties specific to the component and defines transitions to other states that are triggered after the component executes.

[0107] The state section can be used to handle special case scenarios. For example, you might sometimes want to provide the user with the option to do something in a second skill within the digital assistant that temporarily lets the user engage with a first skill. For example, if the user is engaged in a conversation with a shopping skill (e.g., the user has made some selections for a purchase), the user might want to jump to a banking skill (e.g., the user might want to make sure he / she has enough money for the purchase) and then return to the shopping skill to complete the user's order. To address this, an action in the first skill can be configured to initiate an interaction with a different second skill in the same digital assistant and then return to the original flow.

[0108] (6) Adding custom components to the skill bot - As described above, the states specified in the dialog flow of the skill bot name the components that provide the functionality required for the state. The components enable the skill bot to perform the functionality. In certain embodiments, the DABP 102 provides a set of pre-configured components for performing a wide variety of functionality. The skill bot designer can select one or more of these pre-configured components and associate them with a state in the dialog flow of the skill bot. The skill bot designer can also create custom or new components using the tools provided by the DABP 102 and associate the custom components with one or more states in the dialog flow of the skill bot.

[0109] (7) Testing and deploying skill robots - DABP 102 provides several features that enable skill robot designers to test skill robots that are under development. Skill robots can then be deployed and included in digital assistants.

[0110] While the above description describes how skill robots are created, similar techniques can also be used to create digital assistants (or host robots). At the host robot or digital assistant level, the digital assistant can be configured with built-in system intents. These built-in system intents are used to identify general tasks that the digital assistant itself (i.e., the host robot) can handle without invoking a skill robot associated with the digital assistant. Examples of system intents defined for the host robot include: (1) Exit: applicable when a user signals a desire to exit a current session or context in the digital assistant; (2) Help: applicable when a user requests help or direction; and (3) Unresolved Intent: applicable to user input that does not closely match the exit intent and the help intent. The digital assistant also stores information about one or more skill robots associated with the digital assistant. This information enables the host robot to select a particular skill robot for processing an utterance.

[0111] At the host robot or digital assistant level, when a user inputs a phrase or utterance to the digital assistant, the digital assistant is configured to perform processing to determine how to route the utterance and the related session. The digital assistant uses a routing model to determine this, which can be rule-based, AI-based, or a combination thereof. The digital assistant uses the routing model to determine whether a session corresponding to a user input utterance is to be routed to a particular skill for processing, to be processed by the digital assistant or host robot itself according to built-in system intents, or to be handled as a different state in a current session flow.

[0112] In certain embodiments, as part of this process, the digital assistant determines whether the user input utterance explicitly identifies a skill bot using its invocation name. If the invocation name is present in the user input, the invocation name is considered to be an explicit invocation of the skill bot corresponding to the invocation name. In this scenario, the digital assistant can route the user input to the explicitly invoked skill bot for further processing. In certain embodiments, if there is no specific invocation or explicit invocation, the digital assistant evaluates the received user input utterance and computes confidence scores for system intents and skill bots associated with the digital assistant. The scores computed for skill bots or system intents represent how likely the user input represents a task that the skill bot is configured to perform or represents a system intent. Any system intent or skill bot whose associated computed confidence score exceeds a threshold (e.g., a confidence threshold routing parameter) is selected as a candidate for further evaluation. The digital assistant then selects a particular system intent or skill bot from the identified candidates for further processing of the user input utterance. In certain embodiments, after one or more skill bots are identified as candidates, the intents associated with those candidate skills are evaluated (according to the intent model for each skill) and a confidence score is determined for each intent. Generally, any intent whose confidence score exceeds a threshold (e.g., 70%) is considered a candidate intent. If a particular skill bot is selected, the user utterance is routed to that skill bot for further processing. If a system intent is selected, one or more actions are performed by the host bot itself according to the selected system intent.

[0113] 2. Components of a host bot system

[0114] Figure 2 is a simplified block diagram of a host bot (MB) system 200 according to certain embodiments. The MB system 200 can be implemented in software only, hardware only, or a combination of hardware and software. The MB system 200 includes a pre-processing subsystem 210, a plurality of intent subsystems (MIS) 220, an explicit invocation subsystem (EIS) 230, a skill bot invoker 240, and a data store 250. Figure 2 The MB system 200 depicted in FIG. 2 is merely an example of an arrangement of components in a host bot. Those skilled in the art will recognize many possible variations, alternatives, and modifications. For example, in some implementations, the MB system 200 can have more or fewer systems or components than those shown, can combine two or more subsystems, or can have a different configuration or arrangement of subsystems. Figure 2 For example, in some implementations, the MB system 200 can have more or fewer systems or components than those shown, can combine two or more subsystems, or can have a different configuration or arrangement of subsystems.

[0115] The pre-processing subsystem 210 receives the utterance "A" 202 from a user and processes the utterance through a language detector 212 and a language syntax analyzer 214. As indicated above, the utterance can be provided in various ways, including audio or text. The utterance 202 can be a sentence fragment, a complete sentence, multiple sentences, etc. The utterance 202 can include punctuation. For example, if the utterance 202 is provided as audio, the pre-processing subsystem 210 can use a speech-to-text converter (not shown) that inserts punctuation (e.g., commas, semicolons, periods, etc.) into the resulting text to convert the audio to text.

[0116] The language detector 212 detects the language of the utterance 202 based on the text of the utterance 202. The way the utterance 202 is processed depends on the language, as each language has its own syntax and semantics. Differences between languages are taken into account when analyzing the syntax and structure of the utterance.

[0117] The language syntax analyzer 214 performs syntax analysis on the utterance 202 to extract part-of-speech (POS) tags for individual linguistic units (e.g., words) in the utterance 202. POS tags include, for example, noun (NN), pronoun (PN), verb (VB), etc. The language syntax analyzer 214 can also tokenize (e.g., convert each word to an individual token) and lemmatize the linguistic units of the utterance 202. A lemma is the principal form of a set of words as represented in a dictionary (e.g., "run" is the lemma for run, runs, ran, running, etc.). Other types of pre-processing that the language syntax analyzer 214 can perform include chunking of compound expressions, e.g., combining "credit" and "card" into a single expression "credit card." The language syntax analyzer 214 can also identify relationships between words in the utterance 202. For example, in some embodiments, the language syntax analyzer 214 generates a dependency tree that indicates which part of the utterance (e.g., a particular noun) is a direct object, which part of the utterance is a preposition, etc. The result of the processing performed by the language syntax analyzer 214 forms extracted information 205 and is provided as input to the MIS 220 along with the utterance 202 itself.

[0118] As indicated above, the utterance 202 can include more than one sentence. For the purposes of detecting multiple intents and explicit invocation, the utterance 202 can be treated as a single unit, even if it includes multiple sentences. However, in certain embodiments, pre-processing can be performed, e.g., by the pre-processing subsystem 210, to identify individual sentences in the multiple sentences for multiple intent analysis and explicit invocation analysis. Generally, the results produced by the MIS 220 and the EIS 230 are substantially the same whether the utterance 202 is processed at the level of individual sentences or as a single unit that includes multiple sentences.

[0119] MIS 220 determines whether the utterance 202 represents multiple intents. Although MIS 220 can detect multiple intents present in the utterance 202, the processing performed by MIS 220 does not involve determining whether the intent of the utterance 202 matches any of the intents that have been configured for the robot. Instead, the processing of determining whether the intent of the utterance 202 matches a robot intent can be performed by the intent classifier 242 of the MB system 200 or by the intent classifier of the skill robot (e.g., as shown by the embodiments of Figure 3 The processing performed by MIS 220 assumes that there is a robot (e.g., a particular skill robot or the host robot itself) that can process the utterance 202. Thus, the processing performed by MIS 220 does not need to know which robots are in the chatbot system (e.g., the identity of the skill robots registered with the host robot) nor what intents have been configured for a particular robot.

[0120] To determine that the utterance 202 includes multiple intents, MIS 220 applies one or more rules of a set of rules 252 in the data store 250. The rules applied to the utterance 202 depend on the language of the utterance 202 and can include sentence patterns that indicate the presence of multiple intents. For example, a sentence pattern can include a coordinating conjunction that connects two parts of a sentence (e.g., a conjunction), where the two parts correspond to different intents. If the utterance 202 matches the sentence pattern, then it can be inferred that the utterance 202 represents multiple intents. It should be noted that an utterance with multiple intents does not necessarily have different intents (e.g., intents that point to different robots or to different intents within the same robot). Instead, the utterance can have different instances of the same intent, such as “Place a pizza order using payment account X, then place a pizza order using payment account Y.”

[0121] As part of determining that the utterance 202 represents multiple intents, MIS 220 also determines which part of the utterance 202 is associated with each intent. MIS 220 constructs a new utterance for individual processing for each intent represented in the utterance containing multiple intents in place of the original utterance (e.g., utterance “B” 206 and utterance “C” 208, as shown by the embodiments of Figure 2The depicted). Thus, the original utterance 202 can be split into two or more separate utterances that are processed one at a time. The MIS 220 uses the extracted information 205 and / or analysis of the utterance 202 itself to determine which of the two or more utterances should be processed first. For example, the MIS 220 can determine that the utterance 202 contains a marker word that indicates that a particular intent should be processed first. The newly formed utterance corresponding to that particular intent (e.g., one of the utterances 206 or 208) will be sent first for further processing by the EIS 230. After the conversation triggered by the first utterance has ended (or has been temporarily paused), the next highest priority utterance (e.g., the other one of the utterances 206 or 208) can then be sent to the EIS 230 for processing.

[0122] The EIS 230 determines whether the utterance it receives (e.g., the utterance 206 or the utterance 208) contains the invocation name of a skill robot. In certain embodiments, each skill robot in the chatbot system is assigned a unique invocation name that distinguishes that skill robot from other skill robots in the chatbot system. The list of invocation names can be saved in the data store 250 as part of the skill robot information 254. When an utterance contains a word that matches an invocation name, the utterance is considered to be an explicit invocation. If the robot is not explicitly invoked, the utterance received by the EIS 230 is considered to be a non-explicit invocation utterance 234 and is input to the intent classifier of the master robot (e.g., the intent classifier 242) to determine which robot to use to process the utterance. In some instances, the intent classifier 342 will determine that the master robot should process the non-explicit invocation utterance. In other instances, the intent classifier 242 will determine a skill robot to which to route the utterance for processing.

[0123] The explicit invocation functionality provided by the EIS 230 has multiple advantages. It can reduce the amount of processing that the master robot has to perform. For example, when there is an explicit invocation, the master robot can not have to perform any intent classification analysis (e.g., using the intent classifier 242) or can have to perform a simplified intent classification analysis to select a skill robot. Thus, the explicit invocation analysis can enable a particular skill robot to be selected without resorting to intent classification analysis.

[0124] Moreover, there can be instances of functional overlap between multiple skill robots. This can occur, for example, if the intents handled by two skill robots overlap or are very close to each other. In this case, it can be difficult for the master robot to identify which of the multiple skill robots to select based on intent classification analysis alone. In this scenario, explicit invocation makes it unambiguous as to which particular skill robot to use.

[0125] In addition to determining that the utterance is an explicit invocation, the EIS 230 is also responsible for determining whether any portion of the utterance should be used as input to the skill robot that was explicitly invoked. Specifically, the EIS 230 can determine whether a portion of the utterance is unrelated to the invocation. The EIS 230 can perform this determination by analyzing the utterance and / or analyzing the extracted information 205. Rather than sending the entire utterance as received by the EIS 230, the EIS 230 can send the portion of the utterance that is unrelated to the invocation to the skill robot that was invoked. In some instances, the input to the skill robot that was invoked is simply formed by deleting any portion of the utterance that is associated with the invocation. For example, "I want to order pizza using Pizza Bot" can be shortened to "I want to order pizza" because "using Pizza Bot" is related to invoking the pizza robot, but is unrelated to any processing to be performed by the pizza robot. In some instances, the EIS 230 can reformat the portion to be sent to the invoked robot, e.g., to form a complete sentence. Thus, the EIS 230 not only determines that there is an explicit invocation, but also determines what to send to the skill robot when there is an explicit invocation. In some instances, there can be no text to input to the invoked robot. For example, if the utterance is "Pizza Bot," the EIS 230 can determine that the pizza robot is being invoked, but there is no text to be processed by the pizza robot. In such a scenario, the EIS 230 can indicate to the skill robot invoker 240 that there is nothing to send.

[0126] The skill robot invoker 240 invokes skill robots in various ways. For example, the skill robot invoker 240 can invoke a robot in response to receiving an indication 235 that a particular skill robot has been selected as a result of an explicit invocation. The indication 235 can be sent by the EIS 230 along with the input to the skill robot that was explicitly invoked. In this scenario, the skill robot invoker 240 hands over control of the conversation to the skill robot that was explicitly invoked. The skill robot that was explicitly invoked will determine an appropriate response to the input from the EIS 230 by treating the input as a standalone utterance. For example, the response can be to perform a particular action or to begin a new conversation in a particular state, where the initial state of the new conversation depends on the input sent from the EIS 230.

[0127] Another way in which the skill robot invoker 240 can invoke skill robots is through implicit invocation using an intent classifier 242. The intent classifier 242 can be trained using machine learning and / or rule-based training techniques to determine the likelihood that an utterance represents a task for which a particular skill robot is configured to perform. The intent classifier 242 is trained on different classes, one for each skill robot. For example, whenever a new skill robot is registered with the host robot, a list of example utterances associated with the new skill robot can be used to train the intent classifier 242 to determine the likelihood that a particular utterance represents a task that the new skill robot can perform. The parameters that result from this training (e.g., a set of parameter values for a machine learning model) can be stored as part of the skill robot information 254.

[0128] In certain embodiments, the intent classifier 242 is implemented using a machine learning model, as described in further detail herein. Training of the machine learning model can involve inputting at least a subset of utterances from example utterances associated with various skill robots to generate, as output of the machine learning model, an inference as to which robot is the correct robot for handling any particular training utterance. For each training utterance, an indication of the correct robot for the training utterance can be provided as ground truth information. The behavior of the machine learning model can then be adapted (e.g., through backpropagation) to minimize the difference between the generated inference and the ground truth information.

[0129] In certain embodiments, the intent classifier 242 determines, for each skill robot registered with the host robot, a confidence score indicating the likelihood that the skill robot can handle an utterance (e.g., a non-explicitly invoked utterance 234 received from the EIS 230). The intent classifier 242 can also determine a confidence score for each system-level intent that has been configured (e.g., help, exit). If a particular confidence score satisfies one or more conditions, then the skill robot invoker 240 will invoke the robot associated with that particular confidence score. For example, it can be required that a threshold confidence score value be satisfied. Thus, the output 245 of the intent classifier 242 is either an identification of a system intent or an identification of a particular skill robot. In some embodiments, in addition to satisfying a threshold confidence score value, the confidence score must exceed the next highest confidence score by a winning margin. Such a condition will enable routing to a particular skill robot when multiple skill robots each have a confidence score that exceeds the threshold confidence score value.

[0130] After identifying a robot based on an assessment of the confidence scores, the skill robot invoker 240 hands off processing to the identified robot. In the case of a system intent, the identified robot is the host robot. Otherwise, the identified robot is a skill robot. Further, the skill robot invoker 240 will determine what to provide as input 247 to the identified robot. As indicated above, in the case of an explicit invocation, the input 247 can be based on a portion of the utterance that is not associated with the invocation, or the input 247 can be nothing (e.g., an empty string). In the case of an implicit invocation, the input 247 can be the entire utterance.

[0131] The data store 250 includes one or more computing devices that store data used by the various subsystems of the host robot system 200. As explained above, the data store 250 includes rules 252 and skill robot information 254. The rules 252 include, for example, rules for determining by the MIS 220 when an utterance represents multiple intents and how to split an utterance that represents multiple intents. The rules 252 further include rules for determining by the EIS 230 which portions of an utterance that explicitly invokes a skill robot are to be sent to the skill robot. The skill robot information 254 includes invocation names of skill robots in the chatbot system, e.g., a list of invocation names of all skill robots registered with a particular host robot. The skill robot information 254 can also include information used by the intent classifier 242 to determine a confidence score for each skill robot in the chatbot system, e.g., parameters of a machine learning model.

[0132] 3. Components of a skill robot system

[0133] Figure 3 is a simplified block diagram of a skill robot system 300 according to certain embodiments. The skill robot system 300 is a computing system that can be implemented in software only, in hardware only, or in a combination of hardware and software. In the embodiment depicted in FIG. 3, the skill robot system 300 is implemented in software. Figure 1 In certain embodiments of the embodiment depicted in FIG. 3, the skill robot system 300 can be used to implement one or more skill robots within a digital assistant.

[0134] The skill robot system 300 includes a MIS 310, an intent classifier 320, and a session manager 330. The MIS 310 is similar to the MIS 220 described above with respect to FIG. 2. Figure 2MIS 220 provides similar functionality, including operablely using rule 352 in data storage 350 to determine: (1) whether the utterance represents multiple intents, and if so, (2) how to split the utterance into separate utterances for each of the multiple intents. In some embodiments, the rules applied by MIS 310 for detecting multiple intents and for splitting utterances are the same as those applied by MIS 220. MIS 310 receives utterance 302 and extracted information 304. The extracted information 304 is similar to... Figure 2 The information extracted from 205 can be generated using a language parser 214 or a language parser native to the skill robot system 300.

[0135] The intent classifier 320 can be combined with the above. Figure 2 The intent classifier 242 discussed in the embodiments is trained in a similar manner and will be described in further detail herein. For example, in some embodiments, the intent classifier 320 is implemented using a machine learning model. For a specific skill robot, the machine learning model of the intent classifier 320 is trained using at least a subset of example utterances associated with that specific skill robot as training utterances. The underlying fact of each training utterance will be the specific robot intent associated with the training utterance.

[0136] Message 302 can be received directly from the user or supplied through the main robot. When Message 302 is supplied through the main robot (e.g., as via...), Figure 2 The results processed by MIS 220 and EIS 230 in the embodiments depicted can bypass MIS 310 to avoid repeating the processing already performed by MIS 220. However, if utterance 302 is received directly from the user (e.g., during a session after routing to a skill bot), MIS 310 can process utterance 302 to determine whether utterance 302 represents multiple intents. If so, MIS 310 applies one or more rules to split utterance 302 into separate utterances for each intent, such as utterance “D” 306 and utterance “E” 308. If utterance 302 does not represent multiple intents, MIS 310 forwards utterance 302 to intent classifier 320 for intent classification without splitting utterance 302.

[0137] The intent classifier 320 is configured to match received utterances (e.g., utterances 306 or 308) with intents associated with the skill robot system 300. As explained above, the skill robot can be configured with one or more intents, each intent including at least one example utterance associated with that intent and used to train the classifier. Figure 2In embodiments, the intent classifier 242 of the host robotic system 200 is trained to determine a confidence score for each skill robot and a confidence score for the system intent. Similarly, the intent classifier 320 can be trained to determine a confidence score for each intent associated with the skill robotic system 300. The classification performed by the intent classifier 242 is at the robot level, while the classification performed by the intent classifier 320 is at the intent level and thus more granular. The intent classifier 320 can have access to intent information 354. For each intent associated with the skill robotic system 300, the intent information 354 includes a list of utterances that represent the intent and explain the meaning of the intent and are generally associated with tasks that can be performed by the intent. The intent information 354 can further include parameters that result from training on the list of utterances.

[0138] The session manager 330 receives an indication 322 of a particular intent as output of the intent classifier 320 that is identified by the intent classifier 320 as the best match to the utterance input to the intent classifier 320. In some instances, the intent classifier 320 is unable to determine any match. For example, if the utterance points to a system intent or an intent of a different skill robot, the confidence score computed by the intent classifier 320 can fall below a threshold confidence score value. When this occurs, the skill robotic system 300 can hand off the utterance to the host robot for processing, e.g., routing to a different skill robot. However, if the intent classifier 320 is successful in identifying an intent within the skill robotic system 300, the session manager 330 will initiate a session with the user.

[0139] The session initiated by the session manager 330 is a session specific to the intent identified by the intent classifier 320. For example, the session manager 330 can be implemented using a state machine that is configured to execute a dialog flow for the identified intent. The state machine can include a default start state (e.g., when the intent is invoked without any additional input) and one or more additional states, where each state is associated with an action to be performed by the skill robot (e.g., executing a purchase transaction) and / or a dialog (e.g., questions, responses) to be presented to the user. Thus, the session manager 330 can determine the action / dialog 335 upon receiving the indication 322 that the intent was identified, and can determine additional actions or dialogs in response to subsequent utterances received during the session.

[0140] The data store 350 includes one or more computing devices that store data used by the various subsystems of the skill robotic system 300. As Figure 3 Depicted, the data store 350 can include rules 352 and intent information 354. In certain embodiments, the data store 350 can be integrated into a data store of the host robot or digital assistant, e.g.,Figure 2 The data storage in the middle is 250.

[0141] 4. A scheme to classify utterances using a trained intent classifier.

[0142] Figure 4 The illustration shows a configuration for training and utilizing a classifier based on text data 405 (e.g., about...). Figure 2 and Figure 3 A block diagram of various aspects of a chatbot system 400 (describing an intent classifier 242 or 320). Figure 4 As shown, the text classification performed by the chatbot system 400 in this example includes the following stages: a prediction model training stage 410; a skill bot invocation stage 415 for determining the probability that a utterance represents a task configured to be performed by a particular skill bot; and an intent prediction stage 420 for classifying the utterance into one or more intents. The prediction model training stage 410 establishes and trains one or more prediction models 425a-425n ('n' represents any natural number) for use by the other stages (this prediction model may be referred to herein individually as prediction model 425 or collectively as prediction model 425). For example, prediction model 425 may include a model for determining the probability that a utterance represents a task configured to be performed by a particular skill bot, another model for predicting intents based on utterances from a first type of skill bot, and yet another model for predicting intents based on utterances from a second type of skill bot. Other types of prediction models may also be implemented in other examples according to this disclosure.

[0143] The prediction model 425 can be a machine learning ("ML") model, such as a convolutional neural network ("CNN") (e.g., an initial neural network, a residual neural network ("Resnet")), or a recurrent neural network, such as a long short-term memory ("LSTM") model or a gated recurrent unit ("GRU") model, other variants of a deep neural network ("DNN") (e.g., a multi-label n-ary DNN classifier or a multi-class DNN classifier for single-intent classification. The prediction model 425 can also be any other suitable ML model trained for natural language processing, such as a Naive Bayes classifier, a linear classifier, a support vector machine, a bagging model such as a random forest model, a boosting model, a shallow neural network, or a combination of one or more of such techniques - e.g., a CNN-HMM or MCNN (multi-scale convolutional neural network). The chatbot system 400 can employ the same type of prediction model or different types of prediction models to determine the likelihood of a task that a particular skillbot is configured to perform, to predict an intent from a utterance of a first type of skillbot, and to predict an intent from a utterance of a second type of skillbot. Other types of prediction models can still be implemented in other examples in accordance with the present disclosure.

[0144] To train the various prediction models 425, the training phase 410 is composed of three main parts: dataset preparation 430, feature engineering 435, and model training 440. The dataset preparation 430 includes loading data assets 445, splitting the data assets 445 into training and validation sets 445a-n to enable the system to train and test the prediction models 425, and performing processes of basic pre-processing. The data assets 445 can include at least a subset of utterances from example utterances associated with various skill robots. As indicated above, the utterances can be provided in various ways including audio or text. The utterances can be sentence fragments, complete sentences, multiple sentences, etc. For example, if the utterances are provided as audio, the data preparation 430 can use a speech-to-text converter (not shown) that inserts punctuation (e.g., commas, semicolons, periods, etc.) into the resulting text to convert the audio to text. In some instances, the example utterances are provided by a client or customer. In other instances, the example utterances are automatically generated from a previous corpus of utterances (e.g., identifying utterances from the corpus that are specific to a skill that the chat robot is to learn). The data assets 445 for the prediction models 425 can include input text or audio (or input features of text or audio frames) and labels 450 as matrices or tables of values corresponding to the input text or audio (or input features). For example, for each training utterance, an indication of the correct robot for the training utterance can be provided as basic fact information for the labels 450. The behavior of the prediction models 425 can then be adapted (e.g., by backpropagation) to minimize the difference between the generated inferences and the basic fact information. Alternatively, for a particular skill robot, the prediction models 425 can be trained using at least a subset of example utterances associated with that particular skill robot as training utterances. The basic fact information for the labels 450 for each training utterance would be the particular robot intent associated with the training utterance.

[0145] In some embodiments, feature engineering 435 includes transforming data assets 445 into feature vectors and / or creating new features using data assets 445. Feature vectors can include count vectors as features, term frequency-inverse document frequency (TF-IDF) vectors as features (e.g., at word level, n-gram level, or character level), word embeddings as features, text / NLP as features, topic models as features, or combinations thereof. A count vector is a matrix notation of data assets 445 where each row represents a utterance, each column represents a word from a utterance, and each cell represents a frequency count of a particular word in a utterance. A TF-IDF score represents the relative importance of a word in a utterance. Word embeddings are a form of representing words and utterances using dense vector representations. The position of a word within a vector space is learned from the text and is based on the words surrounding the word when it is used. Text / NLP based features can include the number of words in a utterance, the number of characters in a utterance, average word density, number of punctuation marks, number of capital letters, number of title words, frequency distribution of part-of-speech tags (e.g., nouns and verbs), or any combination thereof. Topic modeling is a technique that identifies groups of words (called topics) from a collection of utterances that contain the best information in that collection of utterances.

[0146] In some embodiments, model training 440 includes training a classifier using the feature vectors and / or new features created in feature engineering 435. In some instances, the training process includes an iterative operation to find a set of parameters for prediction model 425 that minimizes a loss or error function for prediction model 425. Each iteration can involve finding a set of parameters for prediction model 425 such that the value of the loss or error function using that set of parameters is less than the value of the loss or error function using another set of parameters in a previous iteration. The loss or error function can be constructed to measure the difference between the output predicted using prediction model 425 and labels 450 contained in data assets 445. Once the set of parameters is identified, prediction model 425 has been trained and can be used according to its design for prediction.

[0147] In addition to the data assets 445, tags 450, feature vectors, and / or new features, other techniques and information can be employed to refine the training process of the prediction model 425. For example, the feature vectors and / or new features can be combined together to help improve the accuracy of the classifier or model. Additionally or alternatively, hyperparameters can be adjusted or optimized, e.g., multiple parameters such as tree length, leaves, network parameters, etc. can be fine-tuned to obtain the best fitting model. Although the training mechanisms described herein primarily focus on training the prediction model 425. These training mechanisms can also be used to fine-tune existing prediction models 425 trained from other data assets. For example, in some cases, the prediction model 425 can have been pre-trained using utterances specific to another skill robot. In these cases, the data assets 445 can be used to retrain the prediction model 425 as discussed herein.

[0148] The prediction model training stage 410 outputs trained prediction models 425, including a task prediction model 460 and an intent prediction model 465. The task prediction model 460 can be used in a skill robot invocation stage 415 to determine the likelihood that an utterance represents a task 470 that a particular skill robot is configured to perform, and the intent prediction model 465 can be used in an intent prediction stage 420 to classify an utterance as one or more intents 475. In some instances, the skill robot invocation stage 415 and the intent prediction stage 420 can be conducted independently in some examples utilizing separate models. For example, the trained intent prediction model 465 can be used in the intent prediction stage 420 to predict the intent of a skill robot without first identifying the skill robot in the skill robot invocation stage 415. Similarly, the task prediction model 460 can be used in the skill robot invocation stage 415 to predict the task or skill robot to be used for an utterance without identifying the intent of the utterance in the intent prediction stage 420.

[0149] Alternatively, the skill robot invocation stage 415 and the intent prediction stage 420 can be performed sequentially, with one stage using the output of the other stage as input, or for a particular skill robot, one stage being invoked based on the output of the other stage in a particular manner. For example, for a given textual data 405, the skill robot invoker can use the skill robot invocation stage 415 and a task prediction model 460 to invoke a skill robot through implicit invocation. The task prediction model 460 can be trained using machine learning and / or rule-based training techniques to determine the likelihood that an utterance represents a task that a particular skill robot 470 is configured to perform. The intent prediction stage 420 and an intent prediction model 465 can then be used to match the received utterance (e.g., an utterance within a given data asset 445) to an intent 475 associated with the skill robot for the skill robot identified or invoked and the given textual data 405. As explained herein, a skill robot can be configured with one or more intents, each intent including at least one example utterance associated with the intent and used to train a classifier. In some embodiments, the skill robot invocation stage 415 and the task prediction model 460 used in the host robot system are trained to determine a confidence score for each skill robot and a confidence score for a system intent. Similarly, the intent prediction stage 420 and the intent prediction model 465 can be trained to determine a confidence score for each intent associated with the skill robot system. The classification performed by the skill robot invocation stage 415 and the task prediction model 460 is at the robot level, while the classification performed by the intent prediction stage 420 and the intent prediction model 465 is at the intent level and is thus more granular.

[0150] C. Logit function

[0151] Figure 5 A diagram illustrating an example neural network 500 is shown in accordance with some embodiments. The neural network 500 can be a machine learning model trained by a training system and implemented by a chatbot system, where the machine learning model is trained to predict whether an utterance corresponds to a particular intent classification. In some instances, the neural network 500 includes an input layer 502, a hidden layer 504, and an output layer 506. In some instances, the neural network 500 includes multiple hidden layers, where the hidden layer 504 corresponds to the final hidden layer of the neural network.

[0152] The input layer 502 can receive input data or a representation of input data (e.g., an n-dimensional array of values representing an utterance) and apply one or more learned parameters to the input data to generate a set of outputs (e.g., a set of numerical values). The set of outputs can be processed by the hidden layer 504.

[0153] Hidden layer 504 may include one or more learned parameters that transform the set of outputs to a different feature space in which input data points from different classifications are better separated.

[0154] Output layer 506 may include a classification layer that maps the output from the hidden layers to logit values, where each value corresponds to a specific category (e.g., parsable category, unparsable category). In some instances, output layer 506 includes an activation function that constrains the logit values ​​to a set of probability values ​​that sum to 1. Thus, the logit values ​​generated by the classification layer (logit function) for each category can then be processed by the activation function of the output layer (also referred to herein as the "activation layer") to predict the category for the discourse.

[0155] As part of predicting a single intent from a set of output values, the machine learning model can employ a logit function (short for "logistic regression function") in the network layers of neural network 500. Output layer 506 can include a logit function to transform intermediate outputs (e.g., probability values ​​predicting whether a utterance corresponds to a particular category) into logit values. In some instances, the logit function is the logarithm of the odds, which takes inputs corresponding to probability values ​​between zero and one for a particular category and outputs logit values ​​over an unbounded range from negative infinity to positive infinity. The logit function can be used to normalize each intermediate output in the set such that the resulting set of logit values ​​can be expressed as a symmetric unimodal probability distribution over the predicted output category. In mathematical terms, the logit function is defined as... Here, p is the probability that the input corresponds to a specific class. The probability p can be set in the range (0,1). The output logit can correspond to a logit value in the range (-∞, +∞). In this way, the machine learning model can employ the logit function so that the subsequent activation function can predict the most likely output (e.g., intent) from the other unlikely outputs of the classifier.

[0156] As a training tool for machine learning models (e.g., Figure 3 As part of the intent classifier 320, the machine learning model can learn the centroid of each category, where the centroid is part of the logit function. For the intent classifier, which has been trained to recognize the identified set of intents, the centroid plays a role in classifying a utterance as belonging to a specific intent and not to others, for example, by using the centroid as a location to measure the distance between adjacent clusters and between a cluster and the model output.

[0157] The centroid can be used to weight the logit function, for example, as part of filtering the classifier output to separate possible intents from an utterance. In mathematical terms, the equation relating to the logit function can be: logit i = f(x) * W i where x is the input to the model, f(x) is a set of transformations (e.g., highway network functions), and W i is the centroid for intent “i” used as a weighting factor for the model output for that intent. Thus, for a set of recognized intents that the intent classifier has been trained to recognize, the centroid plays a role in classifying an utterance as belonging to one particular intent and not to others, for example, by using the centroid as a location for measuring distance between adjacent clusters.

[0158] The logit values generated for each of the prediction output classifications can then be processed through an activation function to map the odds represented by the logit values into a probability distribution over all of the prediction output classifications. As one example of an activation function, a softmax function can be used to normalize the output of the network (e.g., the logit value corresponding to the order_pizza classification) into a probability distribution over all of the prediction output classifications (e.g., order_pizza, unresolved intent).

[0159] Prior to applying the softmax, the logit values can be negative, or greater than one; and can not add up to one. With the softmax applied, each output will be in the interval (0, 1) and the outputs will add up to one. Furthermore, larger input logit values will correspond to larger probabilities. In function terms, the softmax function is represented as: i = 1 : K and z is a set of K real numbers (z1: z K ). In some instances, the softmax function can be weighted by a base factor b, creating a probability distribution that is more concentrated around the location of the largest input value. In this case, the softmax function is where β is a real number. In the case of intent classification, z i may be set to f(x) * w i .

[0160] D. Techniques for Distance-Based Logit Determination for Utterance Classification

[0161] However, the denominator of the softmax function indicates that the softmax function’s value for a real number “i” is a function of the exponentials of all other classes that the machine learning model is trained to distinguish. Thus, softmax normalization can be less sensitive to the individual’s strength that the classifier model outputs. Additionally, softmax can overestimate the confidence values, especially when some model outputs are negative values.

[0162] As an illustrative example, the logit values for a given 3-way classification of 2 utterances are [10, 3, 1] and [3, -2, -4]. In the second utterance, the model is less confident in its prediction than the first utterance, which is reflected in the lower overall logit values and the inclusion of negative logit values, but after normalization using the softmax function, the confidence scores for both utterances are close to 100%. Since the nominal values of the logits should correspond to the model’s confidence in the corresponding classification (e.g., a larger logit value should correspond to a higher probability), the softmax function introduces overconfidence and inaccuracy in the classifier in this example.

[0163] As part of addressing the shortcomings of the n-ary classifier approach, distance-based logit values can be generated to predict whether an utterance corresponds to a particular classification. The distance-based logit value corresponding to the probability of an intent classification can be determined by measuring the distance (e.g., Euclidean distance) between the intermediate output “f(x)” generated by the final layer of the machine learning model and the centroid W i for a given classification “i”.

[0164] For example, a machine learning model can be configured to generate a set of outputs, where each output in the set of outputs includes a probability that predicts whether an utterance corresponds to a respective classification in the set of classifications. The machine learning model can include a set of binary classifiers. Each binary classifier in the set of binary classifiers can generate an intermediate output that represents whether an utterance corresponds to a particular classification as opposed to other classifications in the set of classifications. The intermediate output can be applied to a modified logit function to generate a distance-based logit value for the particular classification. The distance-based logit value can represent odds that the utterance corresponds to the particular classification, where the odds for the particular classification are normalized based on a distance between the intermediate output of the binary classifier and a centroid for the particular classification. The logit function can be a logarithm of the odds for the particular classification (e.g., order_pizza intent classification, cancel_pizza intent classification) and transforms the intermediate output to a corresponding logit value that fits within a probability distribution. The range of the logit value distribution can be between (-∞, +∞). The logit value can be processed by an activation function to generate a probability value for the particular classification that ranges between [0, 1].

[0165] In some examples, the accuracy of the probabilities is further improved by implementing an enhanced activation function. The enhanced activation function includes the use of a scaling value to scale the distance-based logits. The scaling value can be a fixed value determined during hyperparameter tuning. Alternatively, the scaling value can be a learned parameter of the intent classifier, where the value of the parameter is learned during model training. Additionally or alternatively, the machine learning model can incorporate the use of an enhanced loss function to improve the accuracy of the probability estimates for a given utterance. The loss terms of the enhanced loss function can include one or more of: (i) a binary cross-entropy loss term; (ii) a mean squared error term; (iii) a margin loss term; and (iv) a threshold loss term. In some examples, each term of the loss function is weighted by a respective weighting factor. The weighting factors can be tuned as part of the hyperparameter tuning of the intent classifier.

[0166] Thus, wherever it is possible to classify an utterance (e.g., intent, scope, skill, etc.), implementing distance-based logits involves an automated process that can be integrated into a chatbot system, as described with respect to Figure 1 、 Figure 2 and Figure 3 . Advantageously, using distance-based logits, the model and chatbot perform better on utterances, at least in part because the model reduces the impact of the overconfidence problem caused by the softmax function and does not suffer from the uncertainty disadvantage of n-ary “one-versus-all” classifiers. Moreover, because these processes are applied automatically, the customer or client experiences fewer misleading inquiries from the chatbot system.

[0167] 1. Distance-based logits

[0168] In some embodiments, an n-ary “one-versus-all” approach is implemented in the skill robot invocation 415 and / or intent prediction 420. Distance-based logits describe an approach in which the classification includes determining a logit value by measuring the distance between a representation f(x) of the utterance and a centroid W i of a given classification “i”. For example, a function “distance” can be defined such that logits = -1.0 * distance(Wi, f(x)), where “distance” can be measured by a Euclidean distance function. In this way, a confidence score measured by the probability of the classification can be derived from the logit value by computing a scaled sigmoid activation of the logit value.

[0169] Figure 6 A flowchart illustrating an example process 600 for modifying a logit function to determine distance-based logits, in accordance with some embodiments, is shown. Figure 6The depicted processes can be implemented in software (e.g., code, instructions, programs) executed by one or more processing units (e.g., processors, cores) of the respective system, hardware, or combinations thereof. The software can be stored on a non-transitory storage medium (e.g., on a memory device). Figure 6 The methods presented and described below are intended to be illustrative and not limiting. Although Figure 6 Various processing steps are depicted as occurring in a particular sequence or order, this is not intended to be limiting. In some embodiments, these steps are performed by a training subsystem to map probability values corresponding to unresolvable classifications to distance-based logit values of one or more machine learning models (e.g., intent classifiers 242 or 320). The training subsystem can be part of a data processing system (e.g., with respect to the described chatbot system 400), or a component of another system configured to train and deploy machine learning models. Figure 4 The described chatbot system 400.

[0170] At 605, the training subsystem receives a training dataset. The training dataset can include a set of utterances or messages. Each utterance in the set is associated with a training label, where the training label identifies a predicted intent classification of the utterance. In some instances, the training dataset is grouped into one or more training sub-datasets, where each training sub-dataset includes training data (e.g., utterances) that can be used to train a binary classifier in a set of binary classifiers.

[0171] At 610, the training subsystem initializes a machine learning model. The machine learning model can be a convolutional neural network (“CNN”) (e.g., an initial neural network, a residual neural network (“Resnet”)); or a recurrent neural network (e.g., a long short-term memory (“LSTM”) model or a gated recurrent unit (“GRU”) model); other variants of deep neural networks (“DNN”) (e.g., a multi-label n-binary DNN classifier or a multi-class DNN classifier for single intent classification. The machine learning model can also be any other suitable ML model trained for natural language processing, such as a Naive Bayes classifier, a linear classifier, a support vector machine, a bagging model such as a random forest model, a boosting model, a shallow neural network, or a combination of one or more of such techniques— e.g., a CNN-HMM or a MCNN (multi-scale convolutional neural network).

[0172] In some instances, the machine learning model is configured to generate a set of outputs, where each output in the set of outputs includes a probability that predicts whether the utterance corresponds to a respective classification in the set of classifications. The machine learning model can include a set of binary classifiers. Each binary classifier in the set of binary classifiers can generate an intermediate output that includes a value that represents whether the utterance corresponds to a particular classification as opposed to other classifications in the set of classifications. The intermediate output can be input into a logit function to generate a logit value for the particular classification. The logit value can represent an odds ratio that the utterance corresponds to the particular classification. The logit function is a logarithm of the odds ratio for the particular classification (e.g., order_pizza intent classification, cancel_pizza intent classification) and transforms the output of the machine learning model to a corresponding logit value that fits within a probability distribution. The logit value distribution ranges between (-∞, +∞).

[0173] The logit value can be processed by an activation function to generate a probability value for the particular classification. The probability value ranges between [0, 1]. The initialization of the machine learning model can include defining the number of layers, the type of each layer (e.g., fully connected neural network, convolutional neural network), and the type of activation function for each layer (e.g., sigmoid).

[0174] At 615, the training subsystem retrieves a set of logit functions from a final layer of the machine learning model. As mentioned above, the intermediate output of a binary classifier can be input into a respective logit function in the set of logit functions to generate a logit value for a particular classification. Thus, a logit function in the set of logit functions can be a logarithm of an odds ratio that corresponds to a probability of a classification in the set of classifications, weighted by a centroid of a distribution associated with the classification. Further, each logit function in the set of logit functions can be associated with a binary classifier in the set of binary classifiers of the machine learning model. To retrieve the set of logit functions, the training system can select the final layer from a set of layers in the fully connected neural network. The set of logit functions for the final layer can then be accessed for modification.

[0175] At 620, the training subsystem modifies the set of logit functions. For example, each logit function in the set of logit functions can be modified to determine a distance between the intermediate output “f(x)” of a binary classifier and the centroid “Wi” for a given classification “i” in the set of classifications. Thus, the modified function can be represented as logits = -1.0 * distance(Wi, f(x)). The distance can include an Euclidean distance or a cosine distance between the intermediate output and the centroid of the classification. i

[0176] ​At 625, the training subsystem processes the training dataset to train the machine learning model with the modified logit functions. The training of the machine learning model can include training each binary classifier. The training helps the machine learning model to generate distance-based logit values corresponding to probabilities that an utterance corresponds to a particular classification. Thus, the machine learning model can generate distance-based logit values output by each of the modified logit functions in the set of modified logit functions. The training subsystem can train the machine learning model by training each of the binary classifiers in the set of binary classifiers, where values corresponding to the centroids and intermediate outputs of the binary classifiers are learned based on the training dataset. In some instances, a first training dataset used to train a first binary classifier of the machine learning model is different from a second training dataset used to train a second binary classifier. An activation function (e.g., a sigmoid function) can be applied to the distance-based logit values output by the modified logit functions to generate expected outputs, where the expected outputs identify probabilities that a predicted utterance corresponds to a respective classification in the set of classifications.

[0177] Then, one or more parameters of each binary classifier can be learned to minimize an overall loss between the predicted outputs generated by the binary classifiers and the expected outputs indicated by the training labels of the corresponding utterances. The overall loss can be determined by applying a loss function to the predicted outputs and the expected outputs. The training of the machine learning model can be performed until the loss reaches a minimum error threshold.

[0178] At 630, the training subsystem deploys the trained machine learning model to the chatbot system (e.g., as part of a skill bot), where the trained machine learning model includes the modified logit functions. Thereafter, the process 600 terminates.

[0179] 2. Enhanced activation function

[0180] In some instances, the accuracy of the probability estimates is further improved by implementing an enhanced activation function. The enhanced activation function includes scaling the distance-based logit using a scaling value. The scaling value can be a fixed value determined through hyperparameter tuning; or a learned parameter of the intent classifier, where the value of the parameter is learned during model training. The scaling value of the enhanced activation function can be learned to impose a minimum margin value between the highest output and the second highest output of the machine learning model. By incorporating the minimum margin value, the uncertainty problem encountered by traditional n-ary “one-versus-all” classifiers can be reduced.

[0181] Figure 7 A flowchart illustrating an example process 700 for modifying an activation function to estimate probabilities that an utterance corresponds to a particular classification, in accordance with some embodiments, is shown.Figure 7 The depicted processes can be implemented in software (e.g., code, instructions, programs) executed by one or more processing units (e.g., processors, cores) of the respective system, hardware, or combinations thereof. The software can be stored on a non-transitory storage medium (e.g., on a memory device). Figure 7 The methods presented and described below are intended to be illustrative and not limiting. Although Figure 7 Various processing steps are depicted as occurring in a particular sequence or order, this is not intended to be limiting. In some embodiments, these steps are performed by a training subsystem to map probability values corresponding to unresolvable classifications to distance-based logit values of one or more machine learning models (e.g., intent classifiers 242 or 320). The training subsystem can be part of a data processing system (e.g., with respect to the described chatbot system 400), or a component of another system configured to train and deploy machine learning models. Figure 4 The described chatbot system 400) or a component of another system configured to train and deploy machine learning models.

[0182] At 705, the training subsystem receives a training dataset. The training dataset can include a set of utterances or messages. Each utterance in the set is associated with a training label, where the training label identifies a predicted intent classification of the utterance. In some instances, the training dataset is grouped into one or more training sub-datasets, where each training sub-dataset includes training data (e.g., utterances) that can be used to train a binary classifier in a set of binary classifiers.

[0183] At 710, the training subsystem initializes a machine learning model. The machine learning model can be a convolutional neural network (“CNN”) (e.g., an initial neural network, a residual neural network (“Resnet”)); or a recurrent neural network (e.g., a long short-term memory (“LSTM”) model or a gated recurrent unit (“GRU”) model); other variants of deep neural networks (“DNN”) (e.g., a multi-label n-binary DNN classifier or a multi-class DNN classifier for single intent classification. The machine learning model can also be any other suitable ML model trained for natural language processing, such as a Naive Bayes classifier, a linear classifier, a support vector machine, a bagging model such as a random forest model, a boosting model, a shallow neural network, or a combination of one or more of such techniques— e.g., a CNN-HMM or a MCNN (multi-scale convolutional neural network).

[0184] In some instances, the machine learning model is configured to generate a set of outputs, where each output in the set of outputs includes a probability that predicts whether the utterance corresponds to a respective classification in the set of classifications. The machine learning model can include a set of binary classifiers. Each binary classifier in the set of binary classifiers can generate an intermediate output that includes a value that represents whether the utterance corresponds to a particular classification as opposed to other classifications in the set of classifications. The intermediate output can be input into a logit function to generate a logit value for the particular classification. The logit value can represent an odds ratio that the utterance corresponds to the particular classification. The logit function is a logarithm of the odds ratio for the particular classification (e.g., order_pizza intent classification, cancel_pizza intent classification) and transforms the output of the machine learning model to a corresponding logit value that fits within a probability distribution. The logit value distribution ranges between (-∞, +∞).

[0185] The logit value can be processed by an activation function to generate a probability value for the particular classification. The probability value ranges between [0, 1]. The initialization of the machine learning model can include defining the number of layers, the type of each layer (e.g., fully connected neural network, convolutional neural network), and the type of activation function for each layer (e.g., sigmoid).

[0186] At 715, the training subsystem retrieves a set of logit functions from a final layer of the machine learning model. As mentioned above, the intermediate output of a binary classifier can be input into a respective logit function in the set of logit functions to generate a logit value for a particular classification. Thus, a logit function in the set of logit functions can be a logarithm of an odds ratio that corresponds to a probability of a classification in the set of classifications, weighted by a centroid of a distribution associated with the classification. Further, each logit function in the set of logit functions can be associated with a binary classifier in the set of binary classifiers of the machine learning model. To retrieve the set of logit functions, the training system can select the final layer from a set of layers in the fully connected neural network. The set of logit functions for the final layer can then be accessed for modification.

[0187] At 720, the training subsystem modifies the set of logit functions. For example, each logit function in the set of logit functions can be modified to determine a distance between the intermediate output “f(x)” of a binary classifier and the centroid “Wi” for a given classification “i” in the set of classifications. Thus, the modified function can be represented as logits = -1.0 * distance(Wi, f(x)). The distance can include an Euclidean distance or a cosine distance between the intermediate output and the centroid of the classification. i

[0188] ​At 725, the training subsystem modifies the activation function of the machine learning model by including a scaling value. As mentioned above, the enhanced activation function includes scaling the distance-based logit using a scaling value. The scaling value can be a fixed value determined through hyperparameter tuning; or a learned parameter of the intent classifier, where the value of the parameter is learned during model training. The enhanced activation function can be represented by confidence_score = distance_scale * sigmoid(-1.0 * dist(W i ,f(x))), where confidence_score represents the output of the machine learning model and corresponds to a probability that a predicted utterance corresponds to a respective classification in the set of classifications. distance_scale can be a fixed value determined through hyperparameter tuning; or a trainable model parameter determined automatically as part of the training phase 410, as described below.

[0189] Accordingly, a probability can be determined from the enhanced activation function (e.g., the scaled sigmoid function) by normalizing the output between zero and one and by imposing a minimum margin value between the highest output and the second highest output generated by the machine learning model. For example, the enhanced activation function can be used to impose a minimum difference of 0.1 between the two highest classification scores. If the difference between the two highest scores is less than 0.1, then the output of the remaining classifications can be explicitly reduced by 0.1. The difference between the two highest scores can then be recalculated.

[0190] At 730, the training subsystem processes the training data set to train the machine learning model with the set of logit functions and the enhanced activation function. The training helps the machine learning model generate a predicted output that corresponds to a normalized probability that an utterance corresponds to a particular classification. The normalized probability can be determined based at least in part on a learned parameter of the enhanced activation function. The training subsystem can train the machine learning model by training each binary classifier in the set of binary classifiers, where values corresponding to the centroids and the intermediate outputs of the binary classifiers are learned based on the training data set. In some instances, a first training data set used to train a first binary classifier of the machine learning model is different from a second training data set used to train a second binary classifier.

[0191] In some instances, the machine learning model is trained to apply the enhanced activation function to a distance-based logit value output by the modified logit function to generate a predicted output. The predicted output corresponds to a probability that a predicted utterance corresponds to a respective classification in the set of classifications. The training subsystem can train the machine learning model such that a scaling value of the enhanced activation function is learned based on the training data set.

[0192] The one or more parameters of each binary classifier can then be learned to minimize an overall loss between the predicted output generated by the binary classifier and an expected output indicated by the training label for the corresponding utterance. The overall loss can be determined by applying a loss function to the predicted output and the expected output. Training of the machine learning model can be performed until the loss reaches a minimum error threshold.

[0193] At 735, the training subsystem deploys the trained machine learning model to the chatbot system (e.g., as part of a skill bot), where the trained machine learning model includes the modified logit function. Thereafter, process 700 terminates.

[0194] 3. Enhanced loss function

[0195] To further improve the accuracy of the classification of utterances by the machine learning model, the loss function is modified to incorporate additional loss terms, resulting in an enhanced loss function. The additional loss terms of the enhanced loss function can include one or more of: (i) a binary cross-entropy loss term; (ii) a mean squared error term; (iii) a margin loss term; and (iv) a threshold loss term. In some embodiments, the loss function combines the four loss terms with different weights, where each weight can be learned using hyperparameter tuning. The use of the enhanced loss function can further facilitate training of the machine learning model to provide an accurate estimate of its confidence in predicting whether a given utterance corresponds to a particular classification.

[0196] Figure 8 A flowchart illustrating an example process 800 for modifying a loss function to estimate a probability that an utterance corresponds to a particular classification, in accordance with some embodiments, is shown. Figure 8 The depicted processes can be implemented in software (e.g., code, instructions, programs) executed by one or more processing units (e.g., processors, cores) of the respective systems, hardware, or combinations thereof. The software can be stored on a non-transitory storage medium (e.g., on a memory device). Figure 8 The methods presented and described below are intended to be illustrative and not limiting. Although Figure 8 Various processing steps are depicted as occurring in a particular sequence or order, which is not intended to be limiting. In some embodiments, these steps are performed by a training subsystem to map probability values corresponding to unresolvable classifications to distance-based logit values of one or more machine learning models (e.g., intent classifiers 242 or 320). The training subsystem can be part of a data processing system (e.g., with respect to chatbot system 400 described below) or a component of another system configured to train and deploy machine learning models. Figure 4 The chatbot system 400 described below is configured to train and deploy machine learning models to classify utterances into one or more categories. The chatbot system 400 includes a training subsystem 410 and a chatbot subsystem 420.

[0197] At 805, the training subsystem receives a training dataset. The training dataset can include a set of utterances or messages. Each utterance in the set is associated with a training label, where the training label identifies a predicted intent classification of the utterance. In some instances, the training dataset is grouped into one or more training sub-datasets, where each training sub-dataset includes training data (e.g., utterances) that can be used to train a binary classifier in a set of binary classifiers.

[0198] At 810, the training subsystem initializes a machine learning model. The machine learning model can be a convolutional neural network (“CNN”) (e.g., an initial neural network, a residual neural network (“Resnet”)); or a recurrent neural network (e.g., a long short-term memory (“LSTM”) model or a gated recurrent unit (“GRU”) model); other variants of deep neural networks (“DNN”) (e.g., a multi-label n-binary DNN classifier or a multi-class DNN classifier for single intent classification. The machine learning model can also be any other suitable ML model trained for natural language processing, such as a Naive Bayes classifier, a linear classifier, a support vector machine, a bagging model such as a random forest model, a boosting model, a shallow neural network, or a combination of one or more of such techniques - e.g., a CNN-HMM or a MCNN (multi-scale convolutional neural network).

[0199] In some instances, the machine learning model is configured to generate a set of outputs, where each output in the set of outputs includes a probability that a predicted utterance corresponds to a respective classification in the set of classifications. The machine learning model can include a set of binary classifiers. Each binary classifier in the set of binary classifiers can generate an intermediate output that includes a value representing whether an utterance corresponds to a particular classification as opposed to other classifications in the set of classifications. The intermediate output can be input into a logit function to generate a logit value for the particular classification. The logit value can represent an odds ratio that the utterance corresponds to the particular classification. The logit function is the logarithm of the odds ratio for the particular classification (e.g., an order_pizza intent classification, a cancel_pizza intent classification) and transforms the output of the machine learning model to a corresponding logit value that fits within a probability distribution. The range of logit values is between (-∞, +∞).

[0200] The logit value can be processed by an activation function to generate a probability value for the particular classification. The probability ranges between [0, 1]. The initialization of the machine learning model can include defining the number of layers, the type of each layer (e.g., a fully connected neural network, a convolutional neural network), and the type of activation function for each layer (e.g., sigmoid).

[0201] At 815, the training subsystem retrieves a loss function for the machine learning model. For a given utterance in the training dataset, the loss function can be used to determine an error (i.e., a loss) between an output of an activation function of the machine learning model and a target value indicated in a training label for the utterance. Based on the error, the loss function can express how far the computed output is from the target, and the error can be used to adjust weights of parameters of the machine learning model.

[0202] In some instances, the loss function is a binary cross-entropy loss function. For a first classification “1”, the binary cross-entropy loss function (also referred to as a sigmoid cross-entropy loss) is defined as:

[0203] For each of C (e.g., for a skill robot with three defined classifications, C = 3) classifications where C independent binary classification problems are defined (C = 2), where “f(s)” is a sigmoid function, and where “s” is a score and “t” is a ground truth label for the classification. In some instances, the cross-entropy loss can be implemented by a one-hot vector approach that defines C terms. For example, the cross-entropy loss term can be implemented as “xent_loss” = CE(logits, one hot labels), where “CE” is the cross-entropy loss function defined above, and “one hot labels” is a vector that defines C` for each classification (e.g., for C = 3, one hot labels can equal [1 0 0], [0 1 0], and / or [0 0 1]).

[0204] Other types of loss functions can be considered for the machine learning model to classify utterances. For example, the loss function can include, but is not limited to, a mean squared error loss function, a mean squared log error loss function, a mean absolute error loss function, a likelihood loss function, a hinge loss function, a squared hinge loss function, a multi-class cross-entropy loss function, a surrogate multi-class cross-entropy loss, and a Kullback Leibler divergence loss function.

[0205] At 820, the training subsystem modifies the loss function to generate an enhanced loss function. The loss function (e.g., a binary cross-entropy loss function) can be modified to include additional loss terms. The additional loss terms can include one or more of the following:

[0206] • Mean Squared Error (MSE). The MSE term can smooth the output of the activation function (e.g., the confidence values). The MSE term can be expressed as "mse_loss" = MSE(distance_scale * sigmoid(-1.0 * distance(Wi, f(x)), one hot labels), where MSE is the Mean Squared Error function;

[0207] • Margin Loss Term. The margin loss term can be used to ensure a margin between the highest scoring and the second highest scoring in the classification. For example, the margin loss term can be expressed as "margin_loss" = max(0, M - (top_1_confidence_score - top_2_confidence_score)); and

[0208] • Threshold Loss Term. The threshold loss term can be used to constrain the true label confidence score for in-domain utterances and user-defined out-of-domain utterances to be above a first threshold Tl (e.g., Tl = 0.7). Additionally, the threshold loss term can also be used to constrain the confidence scores of all classifications to be below a second threshold T2 (e.g., T2 = 0.5). Thus, the threshold loss term can be expressed as "threshold_loss" = max(0, Tl - true_label_confidence_score) + max(0, top_1_confidence_score - T2).

[0209] In some embodiments, the enhanced loss function combines the four losses with their respective weights (λ i ). Each weight can be tuned during training. For example, the enhanced loss function can be expressed as "overall loss" = λ1 * xent_loss + λ2 * mse_loss + λ3 * margin_loss + λ4 * threshold_loss, where λ i is a tunable parameter for hyperparameter tuning.

[0210] At 825, the training subsystem processes the training dataset to train the machine learning model with the enhanced loss function. In some instances, the training of the machine learning model includes training each binary classifier in the set of classifiers. Training the machine learning model with the enhanced loss function facilitates the machine learning model accurately estimating probabilities that utterances correspond to particular classifications. The training can include performing hyperparameter tuning of weights associated with one of the loss terms of the enhanced loss function. With reference to the above representation in step 820, as part of the hyperparameter tuning of the classifiers for skill robot invocation and / or intent classification, one or more weights of the enhanced loss function (e.g., λ1, λ2, λ3, λ4) can be tuned during the training phase 410.

[0211] In some instances, the training subsystem processes the training dataset to train the machine learning model with the modified logit function, the enhanced activation function, and the enhanced loss function. In addition to the weights of the enhanced loss function, the training subsystem can thus train the machine learning model by training each binary classifier in the set of binary classifiers, where values corresponding to the centroid and the intermediate output of the binary classifier are learned based on the training dataset. In some instances, the machine learning model is also trained to apply the enhanced activation function to the distance-based logit value output by the modified logit function to generate a probability that predicts whether an utterance corresponds to a respective classification in the set of classifications. The training subsystem can train the machine learning model such that a scaling value of the enhanced activation function is learned based on the training dataset.

[0212] The one or more parameters of each binary classifier can then be learned to minimize a total loss between the predicted output generated by the binary classifier and an expected output indicated by a training label of the corresponding utterance. The total loss can be determined by applying a loss function to the predicted output and the expected output. The training of the machine learning model can be performed until the loss reaches a minimum error threshold.

[0213] At 830, the training subsystem deploys the trained machine learning model to a chatbot system (e.g., as part of a skill robot), where the trained machine learning model includes the modified logit function. Thereafter, the process 800 terminates.

[0214] 4. Experimental Results

[0215] As described above, implementing distance-based logit values ​​in trained and tuned classifiers can provide improved accuracy and confidence in intent classification. In the illustrative example, for a set of user-defined skills, implementing distance-based logit values ​​using an n-binary classifier improves the overall accuracy of classifier predictions by up to approximately 6% or more, and can improve the accuracy of individual skill robots by up to approximately 100% or more. As shown in Table 1 below, the performance of skill robots in correctly classifying intents in the test data is significantly improved for most cases and overall. For example, in the example in Table 1, in-domain accuracy is improved by 6%, in-domain recall by 28%, out-of-domain precision is generally unaffected, and out-of-domain recall is improved by 8%.

[0216] Table 1: Classifier performance for user-defined skill test sets

[0217]

[0218]

[0219]

[0220]

[0221] E. The process of training a machine learning model that performs discourse classification based on distance-based Logit values.

[0222] Figure 9 This is a flowchart illustrating a process 900 for training a machine learning model that uses distance-based logit values ​​to classify discourse, according to some embodiments. Figure 9 The described process can be implemented as software (e.g., code, instructions, programs), hardware, or a combination thereof, executed by one or more processing units (e.g., processors, cores) of a corresponding system. The software can be stored on a non-transitory storage medium (e.g., a memory device). Figure 9 The methods presented and described below are intended to be illustrative rather than restrictive. Although Figure 9 The individual processing steps, occurring in a specific sequence or order, are described, but this is not intended to be limiting. In some alternative embodiments, the steps may be performed in a different order, or some steps may be performed in parallel. In some embodiments, such as in Figures 1 to 4 In the described embodiments, Figure 9 The described processing can be handled by a subsystem of the chatbot system (e.g., Figure 4 The skill robot invokes 415 or intent prediction 420 to perform the task of classifying utterances into user-defined categories.

[0223] At 905, the training subsystem receives a training dataset. The training dataset can include a set of utterances or messages. In some instances, the utterances include text data converted from speech input (e.g., speech utterances), where the speech input can be converted into a text form utterance in that particular language and then the text utterance can be processed. Each utterance in the set is associated with a training label, where the training label identifies a predicted intent classification of the utterance. In some instances, the training dataset is grouped into one or more training sub-datasets, where each training sub-dataset includes training data (e.g., utterances) that can be used to train a binary classifier in a set of binary classifiers.

[0224] At 910, the training subsystem initializes a machine learning model. The machine learning model can include a set of binary classifiers. For example, the machine learning model can implement an n-binary “one-versus-all” classifier, where “n” is the number of classifications “C”. Each binary classifier in the set of binary classifiers: (i) is configured to estimate a probability that an utterance corresponds to a classification in the set of classifications; (ii) is associated with a modified logit function that transforms the probability of the classification into a real number. The modified logit function is a log of an odds corresponding to the probability of the classification, which is determined based on a distance measured between the probability of the classification and a centroid of a distribution associated with the classification. In some instances, the distance measured between the probability of the classification and the centroid of the distribution associated with the classification is one of a Euclidean distance or a cosine distance.

[0225] The machine learning model can be a convolutional neural network (“CNN”) (e.g., an initial neural network, a residual neural network (“Resnet”)); or a recurrent neural network (e.g., a long short-term memory (“LSTM”) model or a gated recurrent unit (“GRU”) model); other variants of deep neural networks (“DNN”) (e.g., a multi-label n-binary DNN classifier or a multi-class DNN classifier for single intent classification. The machine learning model 425 can also be any other suitable ML model trained for natural language processing, such as a Naive Bayes classifier, a linear classifier, a support vector machine, a bagging model such as a random forest model, a boosting model, a shallow neural network, or a combination of one or more of such techniques - e.g., a CNN-HMM or MCNN (multi-scale convolutional neural network).

[0226] The initialization of the machine learning model can include defining a number of layers, a type of each layer (e.g., a fully connected neural network, a convolutional neural network), and a type of activation function for each layer (e.g., sigmoid, ReLU, softmax).

[0227] At 915, the training subsystem trains the machine learning model using the training dataset to predict whether an utterance or message represents a task that the skill bot is configured to perform or matches the utterance or message to an intent associated with the skill bot. The training of the machine learning model includes training the set of binary classifiers of the machine learning model with the training dataset. For example, each binary classifier of the set of binary classifiers can be trained by determining respective probabilities of utterances of the training dataset corresponding to a classification associated with the binary classifier. The training of each binary classifier can further include mapping the respective probabilities of the classification associated with the binary classifier to distance-based logit values based on a modified logit function. In some instances, the mapping includes using respective distances measured between the respective probabilities and a centroid of a distribution associated with the classification associated with the binary classifier.

[0228] The training of each binary classifier can further include applying an enhanced activation function to the distance-based logit values to generate a predicted output of the classification. The predicted output can identify a normalized probability of a prediction that the utterance corresponds to the classification within the probability distribution. The enhanced activation function can include a learning parameter for normalizing an initial output of the enhanced activation function to determine the normalized probability. The training of each binary classifier can further include applying an enhanced loss function to the predicted output and an expected output of the utterance to determine a total loss. The enhanced loss function can include a set of loss terms for determining the total loss, and the set of loss terms can include: (i) a binary cross-entropy loss term; (ii) a mean squared error term; (iii) a margin loss term; and (iv) a threshold loss term. The training of each binary classifier can further include adjusting one or more parameters of the binary classifier. In some instances, the one or more parameters include the learning parameter of the enhanced activation function.

[0229] The one or more parameters of each binary classifier can thus be learned to minimize a total loss between the predicted output generated by the binary classifier and an expected output indicated by a training label of a corresponding utterance. The training of the machine learning model can be performed until the loss reaches a minimum error threshold.

[0230] At 920, the trained machine learning model can be deployed within the chatbot system (e.g., as part of the skill bot) to predict whether an utterance or message represents a task that the skill bot is configured to perform or matches the utterance or message to an intent associated with the skill bot, or to predict whether the utterance or message corresponds to an unresolvable classification. Thereafter, the process 900 terminates.

[0231] F. Process for classifying utterances using distance-based logit values

[0232] Figure 10This is a flowchart illustrating a process 1000 for classifying the intent of a discourse or message using distance-based logit values, according to some embodiments. Figure 10 The described process can be implemented as software (e.g., code, instructions, programs), hardware, or a combination thereof, executed by one or more processing units (e.g., processors, cores) of a corresponding system. The software can be stored on a non-transitory storage medium (e.g., a memory device). Figure 10 The methods presented and described below are intended to be illustrative rather than restrictive. Although Figure 10 The individual processing steps, occurring in a specific sequence or order, are described, but this is not intended to be limiting. In some alternative embodiments, the steps may be performed in a different order, or some steps may be performed in parallel. In some embodiments, such as in Figures 1 to 4 In the described embodiments, Figure 10 The described processing can be handled by a subsystem of the chatbot system (e.g., Figure 4 The skill robot invokes 415 or intent prediction 420 to perform the task of classifying utterances into user-defined categories.

[0233] At point 1005, the chatbot system receives utterances generated by a user interacting with the chatbot system. A utterance can refer to a set of words (e.g., one or more sentences) exchanged during a conversation with the chatbot. In some instances, utterances include text data converted from speech input (e.g., speech utterances), where the speech input can be converted into a textual form of utterances in that particular language and then the textual utterances can be processed.

[0234] At point 1010, the chatbot system inputs utterances into a machine learning model. The machine learning model may include a set of binary classifiers. Each binary classifier in this set is: (i) configured to estimate the probability that a utterance corresponds to a category within that set of categories; and (ii) associated with a modified logit function that transforms the probability of a category into a real number. The modified logit function is the logarithm of the probability corresponding to that category, which is determined based on the distance measured between the probability of that category and the centroid of the distribution associated with that category. In some instances, the distance measured between the probability of that category and the centroid of the distribution associated with that category is either Euclidean distance or cosine distance.

[0235] The machine learning model can perform operations 1015 and 1020 in order to generate a prediction output that predicts whether the utterance corresponds to a particular classification. At 1015, the machine learning model generates a set of distance-based logit values for the utterance. Each distance-based logit value in the set of distance-based logit values is generated by (i) determining, by a respective binary classifier in the set of binary classifiers, a respective probability that the utterance corresponds to a classification associated with the respective binary classifier and (ii) mapping, by the respective binary classifier and based on the modified logit function, the respective probability to the distance-based logit value. The mapping includes using a respective distance measured between the respective probability and a centroid of a distribution associated with the classification that is associated with the respective binary classifier.

[0236] At 1020, the machine learning model applies the enhanced activation function to the set of distance-based logit values to generate the prediction output. The prediction output can identify a normalized probability that predicts whether the utterance corresponds to a particular classification in the set of classifications within the probability distribution. In some instances, the enhanced activation function includes a learned parameter for normalizing an initial output of the enhanced activation function to determine the normalized probability.

[0237] For example, the enhanced activation function can be represented by confidence_score = distance_scale * sigmoid(-1.0 * dist(W i , f(x))), where confidence_score represents an output of the machine learning model and corresponds to the normalized probability that predicts whether the utterance corresponds to the particular classification. distance_scale is a learned parameter for normalizing the output of the enhanced activation function.

[0238] At 1025, the chatbot system classifies the utterance as being associated with the particular classification based on the prediction output. Thereafter, process 1000 terminates.

[0239] G. Illustrative Systems

[0240] Figure 11 A simplified diagram depicting a distributed system 1100 is shown. In the illustrated example, the distributed system 1100 includes one or more client computing devices 1102, 1104, 1106, and 1108 coupled to one or more servers 1112 via one or more communication networks 1110. The client computing devices 1102, 1104, 1106, and 1108 can be configured to execute one or more applications.

[0241] In various examples, the server 1112 can be adapted to run one or more services or software applications that implement one or more embodiments described in the present disclosure. In certain examples, the server 1112 can also provide other services or software applications that can include non-virtual and virtual environments. In some examples, these services can be provided as web-based services or cloud services (as under a software-as-a-service (SaaS) model) to users of the client computing devices 1102, 1104, 1106, and / or 1108. Users operating the client computing devices 1102, 1104, 1106, and / or 1108 can in turn utilize one or more client applications to interact with the server 1112 to utilize the services provided by these components.

[0242] In Figure 11 In the depicted configuration, the server 1112 can include one or more components 1118, 1120, and 1122 that implement functionality performed by the server 1112. These components can include software components that can be executed by one or more processors, hardware components, or a combination thereof. It will be appreciated that various different system configurations are possible, and therefore the above-described Figure 11 The illustrated example is one example of a distributed system for implementing an example system and is not intended to be limiting.

[0243] A user can use a client computing device 1102, 1104, 1106, and / or 1108 to execute one or more applications, models, or chatbots that can generate one or more events or models that can then be implemented or served in accordance with the teachings of the present disclosure. The client device can provide an interface that enables a user of the client device to interact with the client device. The client device can also output information to the user via the interface. Although Figure 11 Only four client computing devices are depicted, but any number of client computing devices can be supported.

[0244] The client devices can include various types of computing systems, such as portable handheld devices, general purpose computing devices such as personal computers and laptop computers, workstation computers, wearable devices, game systems, thin clients, various messaging devices, sensors or other sensing devices, and the like. These computing devices can run various types and versions of software applications and operating systems (e.g., Microsoft Windows®, Apple Macintosh® or UNIX®-based operating systems, Linux or Linux-based operating systems (e.g., Google Chrome OS®), and the like), including various mobile operating systems (e.g., Microsoft Windows CE®, Apple iOS®, Google Android®, and the like). Apple or UNIX-based operating systems, Linux or Linux-based operating systems (e.g., Google Chrome OS®), and the like), including various mobile operating systems (e.g., Microsoft Windows CE®, Apple iOS®, Google Android®, and the like). TM OS)) including various mobile operating systems (e.g., Microsoft Windows Windows Android TM , Palm Portable handheld devices can include cellular phones, smartphones (e.g., ), tablet computers (e.g., Wearable devices include Google's personal digital assistants (PDAs), personal digital assistants (PDAs), and more. Head-mounted displays and other devices. Gaming systems can include various handheld gaming devices, internet-enabled gaming devices (e.g., with or without internet access). Microsoft gesture input devices Game console, Sony System, by The client device can run a variety of different applications, such as various Internet-related applications, communication applications (e.g., email applications, short message service (SMS) applications), and can use various communication protocols.

[0245] One or more networks 1110 can be any type of network familiar to those skilled in the art that supports data communication using any of the various available protocols, including but not limited to TCP / IP (Transmission Control Protocol / Internet Protocol), SNA (System Network Architecture), and IPX (Internet Packet Switching). Etc. By way of example only, one or more networks 1110 can be a local area network (LAN), an Ethernet-based network, a token ring network, a wide area network (WAN), the Internet, a virtual network, a virtual private network (VPN), an intranet, an extranet, a public switched telephone network (PSTN), an infrared network, a wireless network (e.g., according to the IEEE 1002.11 protocol suite), etc. (and / or any other wireless protocol operating on a network) and / or any combination of these networks and / or other networks.

[0246] Server 1112 may consist of: one or more general-purpose computers, dedicated server computers (including PC (personal computer) servers by way of example), servers, server farms, server clusters, or any other appropriate arrangement and / or combination. The server 1112 can include one or more virtual machines running a virtual operating system or other computing architecture involving virtualization, such as one or more flexible pools of logical storage devices that can be virtualized to maintain virtual storage devices for servers. In various examples, the server 1112 can be adapted to run one or more services or software applications that provide the functionality described in the foregoing disclosure.

[0247] The computing systems in the server 1112 can run one or more operating systems including any of those discussed above, as well as any commercially available server operating systems. Server 1112 can also be running one or more additional server applications and / or mid-tier applications, including HTTP (HyperText Transfer Protocol) servers, FTP (File Transfer Protocol) servers, CGI (Common Gateway Interface) servers, JAVA® servers, database servers, and the like. The server, database server, and the like. Exemplary database servers include, but are not limited to, those available from International Business Machines (IBM) DB2® Database Server, Microsoft Access®, Microsoft SQL Server®, Oracle Database, etc.

[0248] In some embodiments, the server 1112 can include one or more applications to analyze and correlate data feeds and / or event updates received from users of the client computing devices 1102, 1104, 1106, and 1108. As an example, the data feeds and / or event updates can include, but are not limited to feeds, updates, or real-time updates received from one or more third-party information sources and continuous data streams, which can include real-time events related to sensor data applications, financial tickers, network performance measuring tools (e.g., network monitoring and traffic management applications), clickstream analysis tools, automobile traffic monitoring, and the like. The server 1112 can also include one or more applications to display the data feeds and / or real-time events via one or more display devices of the client computing devices 1102, 1104, 1106, and 1108.

[0249] ​The distributed system 1100 can also include one or more data repositories 1114, 1116. In certain examples, these data repositories can be used to store data and other information. For example, one or more of the data repositories 1114, 1116 can be used to store information, such as information related to chatbot performance or generated models, for use by chatbots used by the servers 1112 in performing various functions in accordance with various embodiments. The data repositories 1114, 1116 can reside in various locations. For example, data repositories used by the servers 1112 can be local to the servers 1112 or can be remote from the servers 1112 and communicate with the servers 1112 via a network-based or dedicated connection. The data repositories 1114, 1116 can be of different types. In certain examples, data repositories used by the servers 1112 can be databases, for example, relational databases such as those provided by Oracle and other vendors. One or more of these databases can be adapted to implement storage, updating, and retrieval of data to and from the databases in response to commands in SQL format.

[0250] In certain examples, one or more of the data repositories 1114, 1116 can also be used by application programs to store application data. Data repositories used by application programs can be of different types, for example, key-value store repositories, object store repositories, or general purpose storage repositories supported by file systems.

[0251] In certain examples, the functionality described in this disclosure can be provided as a service via a cloud environment. Figure 12 is a simplified block diagram of a cloud-based system environment in which various services in accordance with certain examples can be provided as cloud services. In Figure 12 In the depicted example, the cloud infrastructure system 1202 can provide one or more cloud services that can be requested by users using one or more client computers 1204, 1206, and 1208. The cloud infrastructure system 1202 can include one or more computers and / or servers that can include those described above for the servers 1112. The computers in the cloud infrastructure system 1202 can be organized as general-purpose computers, specialized server computers, server farms, server clusters, or any other appropriate arrangement and / or combination.

[0252] One or more networks 1210 can facilitate communications and exchange of data between clients 1204, 1206, and 1208 and cloud infrastructure system 1202. One or more networks 1210 can include one or more networks. The networks can be the same or different types. One or more networks 1210 can support one or more communication protocols, including wired and / or wireless protocols, to facilitate communications.

[0253] Figure 12 The depicted example is merely an example and is not intended to be limiting. In some other examples, cloud infrastructure system 1202 can have more or fewer components, can combine two or more components, or can have a different configuration or arrangement of components than that depicted in FIG. 1. Figure 12 For example, although three client computing devices are depicted, in alternative examples any number of client computing devices can be supported. Figure 12 In some examples, cloud infrastructure system 1202 can include a suite of applications, middleware, and database, and other resources that are

[0254] The term cloud service is generally used to refer to a service offered by a service provider's system (e.g., cloud infrastructure system 1202) that becomes available on demand to users via a communication network, such as the Internet. Generally, in a public cloud environment, the servers and systems that make up the cloud service provider's system are different from the customer's own pre-existing servers and systems. The cloud service provider's system is managed by the cloud service provider. Customers can thus avail themselves of the cloud service offered by the cloud service provider without having to purchase a separate license, support, or hardware and software resources for the service. For example, the cloud service provider's system can host an application, and a user can order and use the application on demand via the Internet without the user having to purchase the infrastructure resources for executing the application. Cloud services are designed to provide easy, scalable access to applications, resources and services. Multiple providers offer cloud services. For example, Oracle Oracle Corporation of Redwood Shores, California offers a variety of cloud services such as middleware services, database services, Java Cloud Service, and other services.

[0255] In certain examples, cloud infrastructure system 1202 can provide one or more cloud services using different models, such as under a software- as-a-service (SaaS) model, a platform-as-a-service (PaaS) model, an infrastructure-as-a-service (IaaS) model, and other models (including hybrid models).

[0256] The SaaS model enables applications or software to be delivered to customers as a service over a communication network, such as the Internet, without the customers having to purchase the hardware or software for the underlying application. For example, customers can be provided with access to on-demand applications hosted by cloud infrastructure system 1202 using the SaaS model. Oracle Examples of SaaS services provided include, without limitation, various services for human resources / capital management, customer relationship management (CRM), enterprise resource planning (ERP), supply chain management (SCM), enterprise performance management (EPM), analytics services, social applications, and the like.

[0257] The IaaS model is typically used to provide customers with infrastructure resources, such as servers, storage, hardware, and networking resources, as cloud services to provide elastic computing and storage capacity. Oracle Various IaaS services are provided by Oracle

[0258] The PaaS model is typically used to provide customers with platform and environment resources that enable customers to develop, run, and manage applications and services without the customers having to purchase, construct, and / or maintain the underlying hardware and software. Oracle Examples of PaaS services provided include, without limitation, Oracle Java Cloud Service (JCS), Oracle Database Cloud Service (DBCS), data management cloud services, various application development solution services, and the like.

[0259] Cloud services are typically provided in a manner that is on-demand self-service, subscription-based, elastically scalable, reliable, highly available, and secure. For example, a customer can order one or more services provided by cloud infrastructure system 1202 via a subscription order. Cloud infrastructure system 1202 then performs processing to provide the services requested by the customer in their subscription order. For example, a user can use a spoken utterance to request that the cloud infrastructure system take some action (e.g., an intent) and / or provide a service for a chatbot system as described herein. Cloud infrastructure system 1202 can be configured to provide one or even multiple cloud services.

[0260] The cloud infrastructure system 1202 can provide cloud services via different deployment models. In a public cloud model, the cloud infrastructure system 1202 can be owned by a third-party cloud service provider and the cloud services are provided to any general public customer, where the customer can be an individual or an enterprise. In certain other examples, under a private cloud model, the cloud infrastructure system 1202 can be operated within an organization (e.g., within an enterprise organization) and the services are provided to customers within the organization. For example, the customers can be various departments of the enterprise, or even individuals within the enterprise, as in the human resources department, the payroll department, etc. In certain other examples, under a community cloud model, the cloud infrastructure system 1202 and the services provided can be shared by multiple organizations within a related community. Various other models, such as mixtures of the above-noted models, can also be used.

[0261] The client computing devices 1204, 1206, and 1208 can be different types of client computing devices (e.g., such as the client computing devices 1102, 1104, 1106, and 1108 depicted in FIG. 11) and can be capable of operating one or more client applications. Users can use the client devices to interact with the cloud infrastructure system 1202, such as to request services provided by the cloud infrastructure system 1202. For example, a user can use a client device to request information or actions from a chatbot as described in the present disclosure. Figure 11

[0262] In some examples, the processing performed by the cloud infrastructure system 1202 to provide services can involve model training and deployment. This analysis can involve using, analyzing, and manipulating datasets to train and deploy one or more models. The analysis can be performed by one or more processors, processing data possibly in parallel, performing simulations using the data, etc. For example, a big data analysis can be performed by the cloud infrastructure system 1202 for generating and training one or more models for a chatbot system. The data used for this analysis can include structured data (e.g., data stored in a database or structured according to a structured model) and / or unstructured data (e.g., data blocks (binary large objects)).

[0263] As depicted in the examples in Figure 12 The cloud infrastructure system 1202 can include infrastructure resources 1230 that are used to facilitate the provisioning of various cloud services provided by the cloud infrastructure system 1202. The infrastructure resources 1230 can include, for example, processing resources, storage or memory resources, networking resources, etc. In certain examples, storage virtual machines that can be used to service storage requests from applications can be part of the cloud infrastructure system 1202. In other examples, the storage virtual machines can be part of a different system.

[0264] ​In certain examples, to facilitate efficient provisioning of these resources to support the various cloud services offered by cloud infrastructure system 1202 for different customers, resources can be bound into resource groups or resource modules (also referred to as "pods"). Each resource module or pod can include a pre-integrated and optimized combination of one or more types of resources. In certain examples, different pods can be provisioned in advance for different types of cloud services. For example, a first set of pods can be provisioned for database services, a second set of pods (which can include a different combination of resources than the pods in the first set of pods) can be provisioned for Java services, and so on. For some services, the resources allocated for provisioning the services can be shared between services.

[0265] Cloud infrastructure system 1202 itself can use, internally, services 1232 that are shared and that facilitate provisioning of services by cloud infrastructure system 1202 by different components of cloud infrastructure system 1202. These internally shared services can include, but are not limited to, security and identity services, integration services, enterprise repository services, enterprise manager services, virus scanning and whitelist services, high availability, backup and recovery services, services for enabling cloud support, email services, notification services, file transfer services, and others.

[0266] Cloud infrastructure system 1202 can include a plurality of subsystems. These subsystems can be implemented in software or hardware, or combination of software and hardware. As Figure 12 As depicted, the subsystems can include a user interface subsystem 1212 that enables users or customers of cloud infrastructure system 1202 to interact with cloud infrastructure system 1202. User interface subsystem 1212 can include a variety of different interfaces, such as a web interface 1214, an online store interface 1216 (where cloud services offered by cloud infrastructure system 1202 are advertised and consumers can purchase), and other interfaces 1218. For example, a customer can use a client device to request (service request 1234) one or more services offered by cloud infrastructure system 1202 using one or more of interfaces 1214, 1216, and 1218. For example, a customer can access an online store, browse cloud services offered by cloud infrastructure system 1202, and place an order to subscribe to one or more services offered by cloud infrastructure system 1202 that the customer wishes to subscribe to. The service request can include information identifying the customer and the one or more services that the customer desires to subscribe to. For example, a customer can place an order to subscribe to a service offered by cloud infrastructure system 1202. As part of the order, the customer can provide information identifying a chatbot system for which the service is to be provided and optionally one or more credentials for the chatbot system.

[0267] In certain examples (such as Figure 12In the depicted example, cloud infrastructure system 1202 can include an order management subsystem (OMS) 1220 configured to process new orders. As part of this processing, OMS 1220 can be configured to: create an account for a customer (if not already created); receive billing and / or accounting information from the customer to be used to bill the customer for providing the requested service to the customer; verify customer information; after verification, place the order for the customer; and orchestrate various workflows to prepare the order for provisioning.

[0268] Once properly verified, OMS 1220 can invoke an order provisioning subsystem (OPS) 1224 configured to provision resources (including processing resources, memory resources, and networking resources) for the order. Provisioning can include allocating resources for the order and configuring the resources to facilitate the service requested by the customer order. The manner in which resources are provisioned and the types of resources provisioned can depend on the type of cloud service the customer has ordered. For example, according to one workflow, OPS 1224 can be configured to determine the particular cloud service being requested and identify the number of pods that can have been pre-configured for that particular cloud service. The number of pods allocated for the order can depend on the size / amount / level / scope of the requested service. For example, the number of pods to allocate can be determined based on the number of users the service supports, the duration of the service being requested, and so forth. The pods allocated can then be customized for the particular requesting customer for providing the requested service.

[0269] In certain examples, the setup phase processing as described above can be performed by cloud infrastructure system 1202 as part of the provisioning process. Cloud infrastructure system 1202 can generate an application ID and select a storage virtual machine for the application from storage virtual machines provided by cloud infrastructure system 1202 itself or from storage virtual machines provided by other systems than cloud infrastructure system 1202.

[0270] Cloud infrastructure system 1202 can send a response or notification 1244 to the requesting customer to indicate when the requested service is now ready for use. In some instances, the customer can be sent information (e.g., a link) that enables the customer to begin using and taking advantage of the benefits of the requested service. In certain examples, the response can include, for the customer requesting the service, a chatbot system ID generated by cloud infrastructure system 1202 and information identifying a chatbot system selected by cloud infrastructure system 1202 for the chatbot system corresponding to the chatbot system ID.

[0271] Cloud infrastructure system 1202 can provide services to multiple customers. For each customer, cloud infrastructure system 1202 is responsible for managing information related to one or more subscription orders received from the customer, maintaining customer data related to the orders, and providing the requested services to the customer. Cloud infrastructure system 1202 can also collect usage statistics about customers' use of subscribed services. For example, it can collect statistics on storage usage, data transfer, number of users, system uptime, and system downtime. This usage information can be used to issue bills to customers. Billing can be done, for example, on a monthly basis.

[0272] Cloud infrastructure system 1202 can provide services to multiple customers in parallel. Cloud infrastructure system 1202 can store information (potentially including proprietary information) for these customers. In some examples, cloud infrastructure system 1202 includes an Identity Management Subsystem (IMS) 1228 configured to manage customer information and provide separation of the managed information so that information related to one customer cannot be accessed by another customer. IMS 1228 can be configured to provide various security-related services, such as information access management, authentication and authorization services, services for managing customer identities and roles, and related functions.

[0273] Figure 13 An example of computer system 1300 is illustrated. In some examples, computer system 1300 can be used to implement any digital assistant or chatbot system within a distributed environment, as well as the various servers and computer systems described above. Figure 13 As shown, the computer system 1300 includes various subsystems, including a processing subsystem 1304 that communicates with multiple other subsystems via a bus subsystem 1302. These other subsystems may include a processing acceleration unit 1306, an I / O subsystem 1308, a storage subsystem 1318, and a communication subsystem 1324. The storage subsystem 1318 may include a non-transitory computer-readable storage medium, including storage medium 1322 and system memory 1310.

[0274] Bus subsystem 1302 provides an mechanism for letting the various components and subsystems of computer system 1300 communicate with each other as intended. Although bus subsystem 1302 is illustrated as a single bus, alternative implementations of bus subsystem 1302 can utilize multiple buses. Bus subsystem 1302 can be any of various types of bus structures including a memory bus or memory controller, a peripheral bus, a local bus using any of a variety of bus architectures, etc. Such architectures can include Industry Standard Architecture (ISA) bus, Micro Channel Architecture (MCA) bus, Enhanced ISA (EISA) bus, Video Electronics Standards Association (VESA) local bus, and Peripheral Component Interconnect (PCI) bus (which can be implemented as a Mezzanine bus manufactured to the IEEE P 1186.1 standard) etc.

[0275] Processing subsystem 1304 controls the operation of computer system 1300 and can include one or more processors, application specific integrated circuits (ASICs), or field programmable gate arrays (FPGAs). A processor can include a single core or multiple cores. Processing resources of computer system 1300 can be organized into one or more processing units 1332, 1334, etc. A processing unit can include one or more processors, one or more cores from the same or different processors, combinations of cores and processors, or other combinations of cores and processors. In some examples, processing subsystem 1304 can include one or more special purpose co-processors such as a graphics processor, a digital signal processor (DSP), etc. In some examples, some or all of the processing units of processing subsystem 1304 can be implemented using custom circuitry such as an application specific integrated circuit (ASIC) or a field programmable gate array (FPGA).

[0276] In some examples, processing units in processing subsystem 1304 can execute instructions stored in system memory 1310 or on computer readable storage media 1322. In various examples, processing units can execute various programs or code instructions and can maintain multiple simultaneously executing programs or processes. Some or all of the program code to be executed can reside on system memory 1310 and / or computer readable storage media 1322 (potentially including on one or more storage devices) at any given time. Processing subsystem 1304, through an appropriate programming, can provide various functionalities described above. In examples where computer system 1300 executes instances of one or more virtual machines, one or more processing units can be assigned to each virtual machine.

[0277] In some examples, a processing acceleration unit 1306 can optionally be provided to perform a specialized processing or to offload some of the processing performed by processing subsystem 1304, thereby accelerating the overall processing performed by computer system 1300.

[0278] I / O subsystem 1308 can include devices and mechanisms for inputting information to computer system 1300 and / or for outputting information from or via computer system 1300. In general, use of the term input device is intended to be broadly inclusive of all possible types of devices and mechanisms for inputting information to computer system 1300. User interface input devices can include, for example, a keyboard, a pointing device such as a mouse or trackball, a touch panel or touch screen incorporated into a display, a scroll wheel, a click wheel, a dial, a button, a switch, a keypad, audio input devices with voice command recognition systems, microphones, and other types of input devices. User interface input devices can also include motion sensing and / or gesture recognition devices such as the Microsoft Kinect® motion sensor that enables users to control and interact with an input device through a motion such as the Microsoft Xbox® 360 game controller, devices that provide an interface for receiving input using gestures and spoken commands. User interface input devices can also include eye gesture recognition devices such as Google Glass®, that detect eye activity such as “blinking” to enter input to an input device. Additionally, user interface input devices can include voice recognition sensing devices that enable a user to interact with a voice recognition system (e.g., Apple Siri®,

[0279] Other examples of user interface input devices include, without limitation, three dimensional (3D) mice, joysticks or pointing sticks, gamepads and graphic tablets, and audio / visual input devices such as speakers, digital cameras, digital camcorders, portable media players, webcams, image scanners, fingerprint scanners, barcode reader 3D scanners, 3D printers, laser rangefinders, and eye gaze tracking devices. Additionally, user interface input devices can include medical imaging input devices such as computerized tomography scanners, magnetic resonance imaging systems, positron emission tomography scanners, and medical ultrasonic imaging equipment. User interface input devices can also include, for example, audio input devices such as MIDI keyboards, digital musical instruments, and the like.

[0280] ​​Generally, use of the term output device is intended to include all possible types of devices and mechanisms for outputting information from computer system 1300 to a user or other computer. User interface output devices can include a display subsystem, indicator lights, or non-visual displays such as audio output devices. Display subsystems can be cathode ray tube (CRT), a flat-panel device such as that using a liquid crystal display (LCD) or plasma display, a projection device, a touch screen, and the like. For example, user interface output devices can include, but are not limited to, various display devices that visually convey text, graphics, and audio / video information such as monitors, printers, speakers, headphones, automobile navigation systems, plotters, voice output devices, and modems.

[0281] Storage subsystem 1318 provides a repository or data storage for storing information and data used by computer system 1300. Storage subsystem 1318 provides a tangible, non-transitory computer-readable storage medium for storing the basic programming and data constructs that provide the functionality of some embodiments. Storage subsystem 1318 can store software (e.g., programs, code modules, instructions) that when executed by processing subsystem 1304 provide the functionality described above of the present disclosure. The software can be executed by one or more processing units of processing subsystem 1304. Storage subsystem 1318 can also provide authentication in accordance with the teachings of this disclosure.

[0282] Storage subsystem 1318 can include one or more non-transitory memory devices, including volatile memory devices and non-volatile memory devices. As shown, storage subsystem 1318 includes system memory 1310 and computer-readable storage media 1322. System memory 1310 can include a number of memories including a volatile main random access memory (RAM) for storing Figure 13 instructions and data during program execution and a non-volatile read-only memory (ROM) or flash memory, in which the basic inputs / outputs system (BIOS), containing the basic routines that help to transfer information between elements within computer system 1300, such as during start-up, is typically stored. RAM typically contains data and / or program modules that are presently being operated and executed by processing subsystem 1304. In some embodiments, system memory 1310 can include a number of different types of memory, such as static random access memory (SRAM), dynamic random access memory (DRAM), or a combination of both.

[0283] By way of example, and not limitation, as Figure 13As depicted, system memory 1310 can load executing applications 1312 (which can include various applications such as a web browser, a middle tier application, a relational database management system (RDBMS), and the like), program data 1314, and operating system 1316. By way of example, operating system 1316 can include various versions of Microsoft Windows® operating systems (e.g., Windows® 10, Windows® Server 2016, Windows® Server 2012, Windows® 8, Windows® 7, Windows® Server 2008, Windows® Vista, Windows® Server 2003, Windows® XP, Windows® 2000, etc.), various versions of Apple Macintosh® operating systems (e.g., macOS, iOS, watchOS, etc.), various versions of UNIX® operating systems (e.g., FreeBSD, OpenBSD, etc.), various versions of LINUX® operating systems (e.g., Google Chrome OS®), various versions of IBM's AIX® operating system, operating systems for mobile computing devices (e.g., iOS, Android, Windows® 10 Mobile, etc.), other mobile operating systems, other proprietary operating systems, or others. Apple and / or Linux operating systems, various commercially available UNIX®-like operating systems (including without limitation the various GNU / Linux operating systems, the Google Chrome OS®, and the like) and / or mobile operating systems (e.g., iOS, Android, Windows® 10 Mobile, and the like).

[0284] Computer-readable storage media 1322 can store programming and data constructs that provide the functionality of some examples. Computer-readable media 1322 can also be used to store software, programs, code modules, instructions, and other data relating to the processes described herein. Software (programs, code modules, instructions) that when executed provide the functionality described above can be stored in storage subsystem 1318. By way of example, computer-readable storage media 1322 can include non-volatile memory such as a hard disk drive, a magnetic disk drive, an optical disk drive (e.g., a CD ROM, a DVD, a Blu-Ray® disk, or other optical media), or the like. Computer-readable storage media 1322 can also include volatile memory such as a random access memory (RAM), a flash memory, a universal serial bus (USB) flash drive, a secure digital (SD) card, a DVD disk, a digital video tape, or the like. Computer-readable storage media 1322 can also include solid-state drives (SSDs) based on non-volatile memory such as flash memory based SSDs, enterprise flash drives, solid-state ROM, and the like, SSDs based on volatile memory such as solid-state RAM, dynamic RAM, static RAM, and the like, DRAM-based SSDs, magnetoresistive RAM (MRAM) SSDs, and hybrid SSDs that use a combination of DRAM and flash-based SSDs.

[0285] In some examples, storage subsystem 1318 can also include a computer- readable storage media reader 1320 that can further connect to computer-readable storage media 1322. Reader 1320 can receive data from and / or write data to computer- readable storage media 1322. In some examples, computer-readable storage media 1322 can include volatile memory, non-volatile memory, removable storage, and / or non-removable storage. ​​​​​​​

[0286] In some examples, computer system 1300 can support virtualization techniques including, without limitation, virtualization of processing and memory resources. For example, computer system 1300 can provide support for executing one or more virtual machines. In some examples, computer system 1300 can execute a program, e.g., a hypervisor, that facilitates the virtualization of the computer system 1300. Each virtual machine can be allocated memory, compute (e.g., processor, core), I / O, and networking resources. Each virtual machine typically runs an operating system independently of the other virtual machines. The virtual machines typically run their own versions of an operating system, which can be same as or different from operating systems executed by other virtual machines executed by computer system 1300. Thus, multiple operating systems can potentially be run simultaneously by computer system 1300.

[0287] Communication subsystem 1324 provides an interface to other computer systems and networks. Communication subsystem 1324 serves as an interface for receiving data from and transmitting data to other systems from computer system 1300. For example, communication subsystem 1324 can enable computer system 1300 to establish a communication channel to one or more client devices via the Internet for receiving and sending information to and from the client devices. For example, when computer system 1300 is used to implement a chatbot system 120 as depicted, the communication subsystem can be used to communicate with the chatbot system selected for the application. Figure 1

[0288] Communication subsystem 1324 can support both wired and / or wireless communication protocols. In some examples, communication subsystem 1324 can include radio-frequency (RF) transceiver components (e.g., using cellular telephone technology, advanced data network technology as 3G, 4G or EDGE (enhanced data rates for global evolution) technology, WiFi (IEEE 802.XX family standards), or other mobile communication technologies, or any combination thereof), global positioning system (GPS) receiver components, and / or other components commonly used in wireless communication subsystems. In some examples, in addition to or instead of a wireless interface, communication subsystem 1324 can provide wired network connectivity (e.g., Ethernet).

[0289] Communication subsystem 1324 can receive and transmit various forms of data. In some examples, communication subsystem 1324 can receive input communications in the form of structured and / or unstructured data feeds 1326, event updates 1328, event updates 1330, and the like. For example, communication subsystem 1324 can be configured to receive (or send) data feeds 1326 in real-time from users of social media networks and / or other communication services such as, for example, Twitter® feeds, Facebook® updates, web feeds such as Rich Site Summary (RSS) feeds, and / or web streams, among other examples. feeds, ​updates, web feeds (such as a rich site summary (RSS) feed), and / or real-time updates from one or more third party information sources.

[0290] In some examples, the communication subsystem 1324 can be configured to receive data in the form of continuous data streams, which can include event streams 1328 of real-time events and / or event updates 1330 that can be inherently continuous or unbounded in nature (may be without explicit end). Examples of applications that generate continuous data can include, for example, sensor data applications, financial ticker applications, network performance monitoring applications (e.g., network monitoring and traffic management applications), clickstream analysis applications, automotive traffic monitoring, and so forth.

[0291] The communication subsystem 1324 can also be configured to transmit data from the computer system 1300 to other computer systems or networks. The data can be transmitted in various different forms including, for example, structured and / or unstructured data feeds 1326, event streams 1328, event updates 1330, and so forth to one or more databases that can be in communication with one or more stream data source computers coupled to the computer system 1300.

[0292] The computer system 1300 can be one of various different generations, including handheld personal computers (PCs), tablet computers, PDAs), wearable devices (e.g., Google cellular phones, tablet computers, PDAs), wearable devices (e.g., Google head-mounted displays), personal computers, workstations, mainframes, kiosks, server racks, or any other data processing system. As the nature of computer and network systems is ever- changing, the teachings of this disclosure should in no way be limited to a specific Figure 13 The depicted computer system 1300 is meant to be only one specific example. Many other configurations having more or fewer components than those shown can be possible. Based on the disclosure and teachings provided herein, a person of ordinary skill in the art will appreciate other ways and / or methods to implement various examples. Figure 13 The depicted system is only one specific example. Numerous other configurations having more or less components than the depicted system are possible. Based on the disclosure and teachings provided herein, a person of ordinary skill in the art will appreciate other ways and / or methods to implement various examples.

[0293] Although specific examples have been described, various modifications, changes, alternative constructions, and equivalents can be used. The examples are not restricted to operation within certain specific data processing environments, but are free to operate within a plurality of data processing environments. Additionally, although a specific series of acts has been described, alternative steps could be performed instead, per the disclosure herein. Also, a person of ordinary skill in the art will appreciate that the order of the steps in which they are presented is immaterial so long as the underlying teachings are maintained. Moreover, not all features are necessarily required to practice the examples described herein. A variety of further modifications can be made that will be apparent to those of ordinary skill in the art without departing from the scope of the disclosure.

[0294] Further, while certain examples have been described using a particular combination of hardware and software, it is recognized that other combinations of hardware and software are also possible. Certain examples can be implemented only in hardware, or only in software, or using combinations thereof. The various processes described herein can be implemented on the same processor or different processors in any combination.

[0295] Where a device, system, component or module is described as being configured to perform certain operations or fulfill certain functions, such configuration can be accomplished, for example, by designing electronic circuits to perform the operation, by programming programmable electronic circuits (such as microprocessors) to perform the operation such as by executing computer instructions or code or by programming processor(s) or core(s) to implement or use an

[0296] The detailed description set forth in this disclosure is intended to provide a thorough understanding of the examples. However, the examples can be practiced without resorting to the details discussed in this disclosure. For example, well-known circuits, processes, algorithms, structures, and techniques have not been shown in detail in order to avoid obscuring the examples. This description provides exemplary examples and is not intended to limit the scope, applicability or configuration of other examples. Rather, the previous description of the examples will provide those skilled in the art with an enabling description of how the various examples are implemented. Various changes can be made to the function and arrangement of elements.

[0297] Accordingly, the specification and drawings are to be regarded in an illustrative rather than a restrictive sense. It will be apparent, however, that modifications, additions, subtractions, and other changes can be made to the specific examples described herein without departing from the broader spirit and scope of the disclosure as set forth in the following claims. Accordingly, while specific examples have been described in detail, these examples are not intended to be limiting. Various modifications and equivalents are within the scope of the following claims.

[0298] In the foregoing specification, aspects of the disclosure are described with reference to specific examples thereof, but those skilled in the art will recognize that the disclosure is not limited thereto. Various features and aspects of the disclosure described above can be used individually or jointly. Further, examples described herein can incorporate any number of features and aspects described above. Further, additional or other examples can incorporate any number of features and aspects described above. Accordingly, the description of examples should not be interpreted as a limitation on the scope or functionality of the disclosure. Rather, aspects of the disclosure are merely illustrative and should not be construed as limiting the scope or functionality of the disclosure.

[0299] In the foregoing description, for the purposes of explanation, methods were described in a particular order. It should be appreciated that in alternative examples, the methods can be performed in an order different than that described. It should also be appreciated that the methods described above can be performed by hardware components or can be embodied in sequences of machine-executable instructions, which can be used to cause a machine (e.g., a general or special purpose processor or logic circuit programmed with the instructions) to perform the methods. These machine-executable instructions can be stored on one or more machine-readable media (e.g., CD-ROMs or other type of optical disk, floppy disks, ROMs, RAMs, EPROMs, EEPROMs, magnetic or optical cards, flash memory, or other type of machine-readable medium suitable for storing electronic instructions). Alternatively, the methods can be performed by a combination of hardware and software.

[0300] Where components are described as being configured to perform certain operations, such configuration can be accomplished, for example, by designing the electronic circuitry or other hardware to perform the operation, by programming the programmable electronic circuitry (e.g., microprocessor or other suitable electronic circuitry) to perform the operation, or any combination thereof.

[0301] While the present application has been described in detail herein in accordance with certain illustrative examples, it should be appreciated that the application concept can be embodied in other ways to realize the benefits thereof, and that such embodiments should be considered within the scope of the appended claims to the extent that they do not teach materially beyond the scope to those in possession of the pertinent art.

Claims

1. A method for discourse classification, comprising: The chatbot system receives utterances generated by a user interacting with the chatbot system, wherein the utterances include text data converted from the user's voice input; The chatbot system inputs the utterance into a machine learning model comprising a set of binary classifiers, wherein each binary classifier in the set of binary classifiers: (i) is configured to estimate the probability that the utterance corresponds to a category in the set of categories; and (ii) is associated with a modified logit function that transforms the probability of the category into a real number, wherein the modified logit function is the logarithm of the odds corresponding to the probability of the category, the logarithm of the odds being determined based on a distance measured between the probability of the category and the centroid of the distribution associated with the category. The machine learning model generates a set of distance-based logit values ​​for the utterance, wherein each distance-based logit value in the set is generated by the following operation: The probability of the utterance corresponding to the category associated with the corresponding binary classifier is determined by the corresponding binary classifier in the set of binary classifiers; and The corresponding probability is mapped to the distance-based logit value by the corresponding binary classifier and based on the modified logit function, wherein the mapping includes using the corresponding distance measured between the corresponding probability and the centroid of the distribution associated with the classification, which is associated with the corresponding binary classifier; The machine learning model applies an enhanced activation function to the set of distance-based logit values ​​to generate a predicted output, wherein the predicted output identifies a normalized probability of whether the predicted utterance corresponds to a specific category within the set of categories in a probability distribution, and wherein the enhanced activation function includes learning parameters for normalizing the initial output of the enhanced activation function to determine the normalized probability; and The chatbot system categorizes the utterances as associated with the specific category based on the predicted output.

2. The method of claim 1, further comprising the chatbot system responding to the user based on classifying the utterance as associated with the specific category.

3. The method of claim 1 or claim 2, further comprising training the machine learning model by applying an enhancement loss function to the predicted output and the expected output corresponding to the utterance to determine a total loss, wherein, The total loss is used to adjust one or more parameters of the machine learning model, wherein the enhanced loss function includes a set of loss terms for determining the total loss, and wherein the set of loss terms includes: (i) a binary cross-entropy loss term; (ii) a mean squared error term; (iii) a margin loss term; and (iv) a threshold loss term.

4. The method of claim 3, wherein, Each of the set of loss terms is associated with a weight parameter, and training the machine learning model includes adjusting the weight parameters of the loss terms in the set of loss terms based on the total loss.

5. The method of claim 3, wherein, Training the machine learning model includes adjusting the learning parameters of the enhanced activation function based on the total loss.

6. The method of claim 3, wherein, The margin loss term identifies the minimum confidence margin as 0.

1.

7. The method of claim 3, wherein, The threshold loss term indicates a minimum threshold confidence level of 0.

5.

8. The method as claimed in claim 1 or claim 2, wherein, The distance measured between the probability of the classification and the centroid of the distribution associated with the classification is either Euclidean distance or cosine distance.

9. A system for discourse classification, comprising: One or more data processors; as well as A non-transitory computer-readable storage medium containing instructions that, when executed on the one or more data processors, cause the one or more data processors to perform operations including: Receive utterances generated by a user interacting with the chatbot system, wherein the utterances include text data converted from the user's voice input; The utterance is input into a machine learning model comprising a set of binary classifiers, wherein each binary classifier in the set of binary classifiers: (i) is configured to estimate the probability of the utterance corresponding to a category in the set of categories; and (ii) is associated with a modified logit function that transforms the probability of the category into a real number, wherein the modified logit function is the logarithm of the odds corresponding to the probability of the category, the logarithm of the odds being determined based on a distance measured between the probability of the category and the centroid of the distribution associated with the category. The machine learning model generates a set of distance-based logit values ​​for the utterance, wherein each distance-based logit value in the set is generated by the following operation: The probability of the utterance corresponding to the category associated with the corresponding binary classifier is determined by the corresponding binary classifier in the set of binary classifiers; and The corresponding probability is mapped to the distance-based logit value by the corresponding binary classifier and based on the modified logit function, wherein the mapping includes using the corresponding distance measured between the corresponding probability and the centroid of the distribution associated with the classification, which is associated with the corresponding binary classifier; The machine learning model applies an enhanced activation function to the set of distance-based logit values ​​to generate a predicted output, wherein the predicted output identifies a normalized probability of whether the predicted utterance corresponds to a specific category within the set of categories in a probability distribution, and wherein the enhanced activation function includes learning parameters for normalizing the initial output of the enhanced activation function to determine the normalized probability; and The discourse is classified into categories associated with the specific category based on the predicted output.

10. The system of claim 9, wherein, The instructions further cause the one or more data processors to perform the following operations: The response to the user is based on classifying the discourse as associated with the specific category.

11. The system as claimed in claim 9 or claim 10, wherein, The instructions further cause the one or more data processors to perform the following operations: The machine learning model is trained by applying an enhanced loss function to the predicted output and the expected output corresponding to the utterance to determine a total loss, wherein the total loss is used to adjust one or more parameters of the machine learning model, wherein the enhanced loss function includes a set of loss terms for determining the total loss, and wherein the set of loss terms includes: (i) a binary cross-entropy loss term; (ii) a mean squared error term; (iii) a margin loss term; and (iv) a threshold loss term.

12. The system of claim 11, wherein, Each of the set of loss terms is associated with a weight parameter, and training the machine learning model includes adjusting the weight parameters of the loss terms in the set of loss terms based on the total loss.

13. The system of claim 11, wherein, Training the machine learning model includes adjusting the learning parameters of the enhanced activation function based on the total loss.

14. The system of claim 11, wherein, The margin loss term identifies the minimum confidence margin as 0.

1.

15. The system of claim 11, wherein, The threshold loss term indicates a minimum threshold confidence level of 0.

5.

16. The system of claim 9 or claim 10, wherein, The distance measured between the probability of the classification and the centroid of the distribution associated with the classification is either Euclidean distance or cosine distance.

17. A computer program product tangibly embodied in a non-transitory machine-readable storage medium, the computer program product comprising instructions configured to cause one or more data processors to perform operations including: Receives utterances generated by users interacting with the chatbot system, where... The speech includes text data converted from the user's voice input; The utterance is input into a machine learning model comprising a set of binary classifiers, wherein each binary classifier in the set of binary classifiers: (i) is configured to estimate the probability of the utterance corresponding to a category in the set of categories; and (ii) is associated with a modified logit function that transforms the probability of the category into a real number, wherein the modified logit function is the logarithm of the odds corresponding to the probability of the category, the logarithm of the odds being determined based on a distance measured between the probability of the category and the centroid of the distribution associated with the category. The machine learning model generates a set of distance-based logit values ​​for the utterance, wherein each distance-based logit value in the set is generated by the following operation: The probability of the utterance corresponding to the category associated with the corresponding binary classifier is determined by the corresponding binary classifier in the set of binary classifiers; and The corresponding probability is mapped to the distance-based logit value by the corresponding binary classifier and based on the modified logit function, wherein the mapping includes using the corresponding distance measured between the corresponding probability and the centroid of the distribution associated with the classification, which is associated with the corresponding binary classifier; The machine learning model applies an enhanced activation function to the set of distance-based logit values ​​to generate a predicted output, wherein the predicted output identifies a normalized probability of whether the predicted utterance corresponds to a specific category within the set of categories in a probability distribution, and wherein the enhanced activation function includes learning parameters for normalizing the initial output of the enhanced activation function to determine the normalized probability; and The discourse is classified into categories associated with the specific category based on the predicted output.

18. The computer program product of claim 17, wherein, The instructions further cause the one or more data processors to perform the following operations: The response to the user is based on classifying the discourse as associated with the specific category.

19. The computer program product as claimed in claim 17 or claim 18, wherein, The instructions further cause the one or more data processors to perform the following operations: The machine learning model is trained by applying an enhanced loss function to the predicted output and the expected output corresponding to the utterance to determine a total loss, wherein the total loss is used to adjust one or more parameters of the machine learning model, wherein the enhanced loss function includes a set of loss terms for determining the total loss, and wherein the set of loss terms includes: (i) a binary cross-entropy loss term; (ii) a mean squared error term; (iii) a margin loss term; and (iv) a threshold loss term.

20. The computer program product of claim 19, wherein, Each of the set of loss terms is associated with a weight parameter, and training the machine learning model includes adjusting the weight parameters of the loss terms in the set of loss terms based on the total loss.

21. The computer program product as claimed in claim 19, wherein, Training the machine learning model includes adjusting the learning parameters of the enhanced activation function based on the total loss.

22. The computer program product of claim 19, wherein, The margin loss term identifies the minimum confidence margin as 0.

1.

23. The computer program product as claimed in claim 19, wherein, The threshold loss term indicates a minimum threshold confidence level of 0.

5.

24. The computer program product as claimed in claim 17 or claim 18, wherein, The distance measured between the probability of the classification and the centroid of the distribution associated with the classification is either Euclidean distance or cosine distance.

25. A method for training a model, comprising: The training subsystem receives a training dataset, wherein the training dataset includes multiple utterances generated by a user interacting with the chatbot system, wherein at least one of the multiple utterances includes text data converted from the user's voice input; The training subsystem accesses a machine learning model comprising a set of binary classifiers, wherein each binary classifier in the set of binary classifiers: (i) is configured to estimate the probability that the utterance corresponds to a category in the set of categories; and (ii) is associated with a modified logit function that transforms the probability of the category into a real number, wherein the modified logit function is the logarithm of the odds corresponding to the probability of the category, the logarithm of the odds being determined based on a distance measured between the probability of the category and the centroid of the distribution associated with the category. The training subsystem uses the training dataset to train the set of binary classifiers for the machine learning model, wherein the training includes: For each of the binary classifiers in the set of binary classifiers: Determine the probabilities of the utterances in the training dataset corresponding to the categories associated with the binary classifier; The modified logit function maps the corresponding probability of the class associated with the binary classifier to a distance-based logit value, wherein the mapping includes using a corresponding distance measured between the corresponding probability and the centroid of the distribution associated with the class, which is associated with the binary classifier. An enhanced activation function is applied to the distance-based logit value to generate a predicted output for the classification, wherein the predicted output identifies a normalized probability of whether the predicted utterance corresponds to the classification within a probability distribution, and wherein the enhanced activation function includes learning parameters for normalizing the initial output of the enhanced activation function to determine the normalized probability. An enhanced loss function is applied to the predicted output and the expected output of the utterance to determine the total loss, wherein the enhanced loss function includes a set of loss terms for determining the total loss, and wherein the set of loss terms includes: (i) a binary cross-entropy loss term; (ii) a mean squared error term; (iii) a margin loss term; and (iv) a threshold loss term; and Adjust one or more parameters of the binary classifier, wherein the one or more parameters include the learned parameters of the enhanced activation function; and The trained machine learning model is deployed by the training subsystem.

26. The method of claim 25, wherein, Each of the set of loss terms is associated with a weight parameter, and the training of the machine learning model includes adjusting the weight parameters of the loss terms in the set of loss terms based on the total loss.

27. The method of claim 25 or claim 26, wherein, The distance measured between the probability of the classification and the centroid of the distribution associated with the classification is either Euclidean distance or cosine distance.

28. The method of claim 25 or claim 26, wherein, Training the machine learning model includes: The first binary classifier in the set of binary classifiers is trained using the first training subset of the training dataset; and The second binary classifier in the set of binary classifiers is trained using a second training subset of the training dataset, wherein the second training subset includes one or more utterances that are not present in the first training subset of the plurality of utterances.

29. The method of claim 25 or claim 26, wherein, The margin loss term identifies the minimum confidence margin as 0.

1.

30. The method of claim 25 or claim 26, wherein, The threshold loss term indicates a minimum threshold confidence level of 0.

5.

31. A system for training a model, comprising: One or more data processors; as well as A non-transitory computer-readable storage medium containing instructions that, when executed on the one or more data processors, cause the one or more data processors to perform operations including: Receive a training dataset, wherein the training dataset includes multiple utterances generated by a user interacting with the chatbot system, wherein at least one of the multiple utterances includes text data converted from the user's voice input; Access includes a machine learning model comprising a set of binary classifiers, wherein each binary classifier in the set of binary classifiers: (i) is configured to estimate the probability that the utterance corresponds to a category in the set of categories; and (ii) is associated with a modified logit function that transforms the probability of the category into a real number, wherein the modified logit function is the logarithm of the odds corresponding to the probability of the category, the logarithm of the odds being determined based on a distance measured between the probability of the category and the centroid of the distribution associated with the category. The set of binary classifiers for the machine learning model is trained using the training dataset, wherein the training includes: For each of the binary classifiers in the set of binary classifiers: Determine the probabilities of the utterances in the training dataset corresponding to the categories associated with the binary classifier; The modified logit function maps the corresponding probability of the class associated with the binary classifier to a distance-based logit value, wherein the mapping includes using a corresponding distance measured between the corresponding probability and the centroid of the distribution associated with the class, which is associated with the binary classifier. An enhanced activation function is applied to the distance-based logit value to generate a predicted output for the classification, wherein the predicted output identifies a normalized probability of whether the predicted utterance corresponds to the classification within a probability distribution, and wherein the enhanced activation function includes learning parameters for normalizing the initial output of the enhanced activation function to determine the normalized probability. An enhanced loss function is applied to the predicted output and the expected output of the utterance to determine the total loss, wherein the enhanced loss function includes a set of loss terms for determining the total loss, and wherein the set of loss terms includes: (i) a binary cross-entropy loss term; (ii) a mean squared error term; (iii) a margin loss term; and (iv) a threshold loss term; and Adjust one or more parameters of the binary classifier, wherein the one or more parameters include the learned parameters of the enhanced activation function; and Deploy the trained machine learning model.

32. The system of claim 31, wherein, Each of the set of loss terms is associated with a weight parameter, and the training of the machine learning model includes adjusting the weight parameters of the loss terms in the set of loss terms based on the total loss.

33. The system of claim 31 or claim 32, wherein, The distance measured between the probability of the classification and the centroid of the distribution associated with the classification is either Euclidean distance or cosine distance.

34. The system of claim 31 or claim 32, wherein, Training the machine learning model includes: The first binary classifier in the set of binary classifiers is trained using the first training subset of the training dataset; and The second binary classifier in the set of binary classifiers is trained using a second training subset of the training dataset, wherein the second training subset includes one or more utterances that are not present in the first training subset of the plurality of utterances.

35. The system of claim 31 or claim 32, wherein, The margin loss term identifies the minimum confidence margin as 0.

1.

36. The system of claim 31 or claim 32, wherein, The threshold loss term indicates a minimum threshold confidence level of 0.

5.

37. A computer program product tangibly embodied in a non-transitory machine-readable storage medium, the computer program product comprising instructions configured to cause one or more data processors to perform operations including: Receive the training dataset, where, The training dataset includes multiple utterances generated by users interacting with the chatbot system, wherein at least one of the multiple utterances includes text data converted from the user's voice input; Access includes a machine learning model comprising a set of binary classifiers, wherein each binary classifier in the set of binary classifiers: (i) is configured to estimate the probability that the utterance corresponds to a category in the set of categories; and (ii) is associated with a modified logit function that transforms the probability of the category into a real number, wherein the modified logit function is the logarithm of the odds corresponding to the probability of the category, the logarithm of the odds being determined based on a distance measured between the probability of the category and the centroid of the distribution associated with the category. The set of binary classifiers for the machine learning model is trained using the training dataset, wherein the training includes: For each of the binary classifiers in the set of binary classifiers: Determine the probabilities of the utterances in the training dataset corresponding to the categories associated with the binary classifier; The modified logit function maps the corresponding probability of the class associated with the binary classifier to a distance-based logit value, wherein the mapping includes using a corresponding distance measured between the corresponding probability and the centroid of the distribution associated with the class, which is associated with the binary classifier. An enhanced activation function is applied to the distance-based logit value to generate a predicted output for the classification, wherein the predicted output identifies a normalized probability of whether the predicted utterance corresponds to the classification within a probability distribution, and wherein the enhanced activation function includes learning parameters for normalizing the initial output of the enhanced activation function to determine the normalized probability. An enhanced loss function is applied to the predicted output and the expected output of the utterance to determine the total loss, wherein the enhanced loss function includes a set of loss terms for determining the total loss, and wherein the set of loss terms includes: (i) a binary cross-entropy loss term; (ii) a mean squared error term; (iii) a margin loss term; and (iv) a threshold loss term; and Adjust one or more parameters of the binary classifier, wherein the one or more parameters include the learned parameters of the enhanced activation function; and Deploy the trained machine learning model.

38. The computer program product of claim 37, wherein, Each of the set of loss terms is associated with a weight parameter, and the training of the machine learning model includes adjusting the weight parameters of the loss terms in the set of loss terms based on the total loss.

39. The computer program product as claimed in claim 37 or claim 38, wherein, The distance measured between the probability of the classification and the centroid of the distribution associated with the classification is either Euclidean distance or cosine distance.

40. The computer program product as claimed in claim 37 or claim 38, wherein, Training the machine learning model includes: The first binary classifier in the set of binary classifiers is trained using the first training subset of the training dataset; and The second binary classifier in the set of binary classifiers is trained using a second training subset of the training dataset, wherein the second training subset includes one or more utterances that are not present in the first training subset of the plurality of utterances.

41. The computer program product as claimed in claim 37 or claim 38, wherein, The margin loss term identifies the minimum confidence margin as 0.

1.

42. The computer program product as claimed in claim 37 or claim 38, wherein, The threshold loss term indicates a minimum threshold confidence level of 0.5.

Images