Enhanced logit for natural language processing

Abstract

Techniques for classifying utterances and messages input to a chatbot system using enhanced logit values in natural language processing. A method can include a chatbot system receiving an utterance generated by a user interacting with the chatbot system and inputting the utterance into a machine learning model comprising a series of network layers. A final network layer in the series of network layers can include a logit function. The machine learning model can map a first probability of a resolvable classification to a first logit value using the logit function. The machine learning model can map a second probability of a non-resolvable classification to an enhanced logit value. The method can also include the chatbot system classifying the utterance as the resolvable classification or the non-resolvable classification based on the first logit value and the enhanced logit value.

Description

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS

[0002] This application claims the benefit of U.S. Provisional Application No. 63 / 119,449, filed on November 30, 2020, and U.S. Non-Provisional Application No. 17 / 456,687, filed on November 29, 2021. Each of the above-referenced applications is incorporated herein by reference in its entirety for all purposes. Technical Field

[0003] The present disclosure relates generally to chatbot systems, and more particularly to techniques for classifying utterances and messages input to chatbot systems using augmented logit values ​​in natural language processing. Background Art

[0004] In order to get an immediate response, many users around the world use instant messaging or chat platforms. Organizations often use these instant messaging or chat platforms to conduct real-time conversations with customers (or end users). However, hiring service personnel to communicate with customers or end users in real time can be very expensive for organizations. Chatbots or robots have begun to be developed to simulate conversations with end users, especially over the Internet. End users can communicate with robots through messaging applications that the end users have installed and use. Intelligent robots (usually powered by artificial intelligence (AI)) can communicate more intelligently and contextually in real-time conversations, and therefore can allow more natural conversations between robots and end users to improve the conversation experience. Instead of the end user learning a fixed set of keywords or commands that the robot knows how to respond to, the intelligent robot can understand the end user's intentions based on the user's utterances in natural language and respond accordingly. Summary of the Invention

[0005] A technique for classifying utterances and messages input to a chatbot system using enhanced logit values ​​in natural language processing is provided. A method may include a chatbot system receiving an utterance generated by a user interacting with the chatbot system and inputting the utterance into a machine learning model comprising a series of network layers. The utterance may include text data converted from speech input by the user. A final network layer in the series of network layers may include a logit function that transforms a first probability of a parsable classification into a first real number representing a first logit value and transforms a second probability of an unparsable classification into a second real number representing a second logit value.

[0006] The method may also include the machine learning model determining a first probability of the resolvable class and a second probability of the unresolvable class. The machine learning model may map the first probability of the resolvable class to a first logit value using a logit function. The logit function used to map the first probability may be a logarithm of the odds corresponding to the first probability of the resolvable class, the logarithm of the odds being weighted by a centroid of a distribution associated with the resolvable class.

[0007] The method may also include the machine learning model mapping the second probability of the unresolvable classification to an enhanced logit value. The enhanced logit value may be a third real number determined independently of the logit function used to map the first probability. The enhanced logit value may include: (i) a statistical value determined based on a set of logit values ​​generated from a training data set; (ii) a bounded value selected from a range of values ​​defined by a first logarithm of the probability corresponding to the second probability of the unresolvable classification, the first logarithm of the probability being constrained to the range of values ​​by a bounding function and weighted by a centroid of a distribution associated with the unresolvable classification; (iii) a weighted value generated by a second logarithm of the probability corresponding to the second probability of the unresolvable classification, the second logarithm of the probability being constrained to the range of values ​​by a bounding function, scaled by a scaling factor, and weighted by a centroid of a distribution associated with the unresolvable classification; (iv) a hyperparameter optimized value generated based on hyperparameter tuning of the machine learning model; or (v) a learned value adjusted during training of the machine learning model. The method may further include the chatbot system classifying the utterance as a parsable classification or an unparsable classification based on the first logit value and the enhanced logit value.

[0008] Also provided are techniques for training a machine learning model that uses augmented logit values ​​to classify utterances and messages. A method may include a training subsystem receiving a training dataset. The training dataset may include a plurality of utterances generated by a user interacting with a chatbot system. At least one of the plurality of utterances may include text data converted from a voice input of the user. The method may also include the training subsystem accessing a machine learning model comprising a series of network layers. A final network layer in the series of network layers comprises a logit function that transforms a first probability of a resolvable classification into a first real number representing a first logit value and transforms a second probability of an unresolvable classification into a second real number representing a second logit value.

[0009] The method can further include the training subsystem utilizing the training dataset to train the machine learning model such that the machine learning model: (i) determines a first probability of the resolvable classification and a second probability of the non-resolvable classification; and (ii) maps the first probability of the resolvable classification using a logit function. The logit function used to map the first probability can be a log of an odds corresponding to the first probability of the resolvable classification, the log of the odds being weighted by a centroid of a distribution associated with the resolvable classification.

[0010] The method can further include the training subsystem replacing the logit function with an enhanced logit value such that the second probability of the non-resolvable classification is mapped to the enhanced logit value. The enhanced logit value can be a third real number determined independently of the logit function used to map the first probability. The enhanced logit value can further include: (i) a statistical value determined based on a set of logit values generated from the training dataset; (ii) a bounded value selected from a range of values defined by a first log of an odds corresponding to the second probability of the non-resolvable classification, the first log of the odds being constrained to the range of values by a clipping function and weighted by a centroid of a distribution associated with the non-resolvable classification; (iii) a weighted value generated from a second log of the odds corresponding to the second probability of the non-resolvable classification, the second log of the odds being constrained to the range of values by the clipping function, scaled by a scaling factor, and weighted by the centroid of the distribution associated with the non-resolvable classification; (iv) a hyperparameter optimization value generated based on hyperparameter tuning of the machine learning model; or (v) a learning value adjusted during training of the machine learning model. The method can further include the training subsystem deploying the trained machine learning model utilizing the enhanced logit value.

[0011] In various embodiments, a system is provided that includes one or more data processors and a non-transitory computer-readable storage medium containing instructions that, when executed by the one or more data processors, cause the one or more data processors to perform part or all of one or more methods disclosed herein.

[0012] In various embodiments, a computer program product is provided that is tangibly embodied in a non-transitory machine-readable storage medium and that includes instructions configured to cause one or more data processors to perform part or all of one or more methods disclosed herein.

[0013] The techniques described above and below can be implemented in a number of ways and in various contexts. Various example implementations and contexts are provided below in greater detail, with reference to the following drawings. However, those implementations and contexts are merely some of the many. BRIEF DESCRIPTION OF DRAWINGS

[0014] Figure 1 is a simplified block diagram of a distributed environment incorporating an exemplary embodiment.

[0015] Figure 2 is a simplified block diagram of a computing system implementing a master robot, according to some embodiments.

[0016] Figure 3 is a simplified block diagram of a computing system that implements a skill bot, according to some embodiments.

[0017] Figure 4 is a simplified block diagram of a chatbot training and deployment system according to various embodiments.

[0018] Figure 5 Illustrated is a schematic diagram of an example neural network in accordance with some embodiments.

[0019] Figure 6 Shown is a flowchart illustrating an example process for determining a statistic representing an augmented logit value for predicting whether an utterance corresponds to an unparseable classification in accordance with some embodiments.

[0020] Figure 7 Shown is a flowchart illustrating an example process for modifying a logit function to determine augmented logit values ​​within a specified range for predicting whether an utterance corresponds to an unparseable classification, in accordance with some embodiments.

[0021] Figure 8 Shown is a flowchart illustrating an example process for adding a scaling factor to a logit function to determine an enhanced logit value for predicting whether an utterance corresponds to an unparseable classification, in accordance with some embodiments.

[0022] Figure 9 Shown is a flowchart illustrating an example process for determining augmented logit values ​​for predicting whether an utterance corresponds to an unparseable classification using hyperparameter tuning in accordance with some embodiments.

[0023] Figure 10 Shown is a flowchart illustrating an example process of using learned values ​​as augmented logit values ​​for predicting whether an utterance corresponds to an unparseable classification in accordance with some embodiments.

[0024] Figure 11 is a flow diagram illustrating a process for training a machine learning model that implements augmented logit values ​​for unparseable classifications, according to some embodiments.

[0025] Figure 12 is a flow diagram illustrating a process for classifying an utterance into unparseable categories using augmented logit values, according to some embodiments.

[0026] Figure 13 Depicted is a simplified diagram of a distributed system for implementing various embodiments.

[0027] Figure 14 is a simplified block diagram of one or more components of a system environment according to various embodiments by which services provided by one or more components of an embodiment system may be provided as a cloud service.

[0028] Figure 15 An example computer system is illustrated that can be used to implement various embodiments. DETAILED DESCRIPTION

[0029] In the following description, for purposes of explanation, specific details are set forth in order to provide a thorough understanding of some embodiments. However, it will be apparent that the various embodiments can be practiced without these specific details. The drawings and description are not intended to be limiting. The word "exemplary" is used herein to mean "serving as an example, instance, or illustration." Any embodiment or design described herein as "exemplary" is not necessarily to be construed as preferred or advantageous over other embodiments or designs.

[0030] A. Overview

[0031] 1. Intent

[0032] A digital assistant is an AI-driven interface that helps users complete various tasks in natural language conversations. For each digital assistant, customers can assemble one or more skills. Skills (also described herein as chatbots, robots, or skillbots) are individual robots that focus on specific types of tasks such as tracking inventory, submitting time cards, and creating expense reports. When an end user engages with a digital assistant, the digital assistant evaluates the end user input and routes the conversation to and from the appropriate chatbot. Messenger (instant messaging), SKYPE Various channels, such as messenger or Short Message Service (SMS), make digital assistants available to end users. Channels carry chats back and forth between end users on various messaging platforms and digital assistants and their various chatbots. Channels can also support user agent escalation, event-initiated conversations, and testing.

[0033] Intents allow a chatbot to understand what the user wants the chatbot to do. Intents include a sequence of typical user requests and statements, also known as utterances (e.g., "get an account balance," "make a purchase," etc.). As used herein, an utterance or message can refer to a set of words (e.g., one or more sentences) exchanged during a conversation with the chatbot. An utterance can be text converted from voice input entered by a user via a user interface. Intents can be created by providing a name that describes a user action (e.g., "order a pizza") and compiling a set of real-life user statements or utterances that are typically associated with the triggering action. Because the chatbot's cognition is derived from these intents, each intent can be created and varied from a robust dataset (one to two dozen utterances) so that the chatbot can interpret ambiguous user input. A rich set of utterances enables the chatbot to understand what the user wants when it receives messages such as "Forget this order!" or "Cancel delivery!" (messages that mean the same thing but are expressed in different ways). In summary, intents and the utterances that belong to an intent constitute the chatbot's training corpus. By training the model with a corpus, customers can essentially turn the model into a reference tool for parsing end-user input into individual intents. Customers can improve the cognitive acuity of the chatbot through multiple rounds of intent testing and intent training.

[0034] However, building a chatbot that can determine the end user's intent based on user utterances is a challenging task, partly due to the subtlety and ambiguity of natural language and the dimensionality of the input space (e.g., possible user utterances) and the size of the output space (the number of intents). An illustrative example of this difficulty comes from the characteristics of natural language, such as the use of euphemisms, synonyms, or ungrammatical speech to express intent. For example, an utterance may express the intention to order a pizza without explicitly mentioning pizza, ordering, or delivery. For example, in some regional dialects, "pizza" is called "pie." These trends in natural language (such as imprecision or variability) create uncertainty and introduce confidence as an intent prediction parameter, rather than explicitly indicating intent by, for example, including keywords. Thus, in order to improve the performance of the chatbot and the user experience with the chatbot, it may be necessary to train, monitor, debug, and retrain the chatbot. In traditional systems, training systems are provided for training and retraining machine learning models of digital assistants or chatbots in spoken language understanding (SLU) and natural language processing (NLP).

[0035] 2. Use machine learning models to determine intent

[0036] At one or more points, the chatbot system can provide an utterance as input to a neural network model, which uses a logistic regression function to map the outputs to a probability distribution. For example, for classification, ranking a set of outputs in a probability distribution allows prediction of the intent invoked in the utterance. Accurate predictions, in turn, enable the chatbot to interact accurately with the end user. In this sense, accuracy depends, at least in part, on mapping the outputs of the neural network classifier to a probability distribution.

[0037] To map the output of a neural network machine learning model into a probability distribution, a logit value is determined based on the input. The logit value (also known as "logit") is the value output by the logit function of the network layer of the machine learning model. The logit value can represent the probability that the utterance corresponds to a specific classification. The logit function is the logarithm of the probability of a specific classification (e.g., order_pizza intent classification, unparseable classification) and transforms the output of the machine learning model into a corresponding logit value that fits within the probability distribution. The range of logit values ​​is between (-∞, +∞). The logit value can then be provided as input to an activation function (e.g., a softmax function) to generate a predicted probability of whether the input (e.g., utterance) corresponds to a specific classification in a set of classifications, where the predicted probability is within the probability distribution of the set of classifications. The predicted probability can range between [0,1]. For example, a numerical output corresponding to a specific classification (e.g., order_pizza) is generated by processing an input utterance (e.g., "I want to grab a pie") through one or more intermediate layers of a polynomial machine learning model. The output can be processed by the logit function of the specific classification to generate a logit value of 9.4. An activation function can then be applied to the logit values ​​to determine a probability value ranging between 0 and 1 (e.g., 0.974), indicating that the input utterance corresponds to the order_pizza classification. Utterances that do not invoke any of the intents that the intent classifier is trained to recognize would correspond to out-of-scope or out-of-domain utterances. For example, the utterance "How is the weather today?" would be considered out-of-scope for a classifier trained to predict whether the utterance specifies an order for a particular food item.

[0038] Classification accuracy can be further improved by weighting one or more parameters of the logit function. For example, each intent classification can be associated with a logit function that can be weighted by the centroid of the intent classification. As used herein, the term "centroid" refers to the location of the central position of a cluster used to classify an utterance, where the cluster identifies data corresponding to a particular end-user intent classification. In some instances, the centroid is determined using data from a corresponding utterance dataset (e.g., a training dataset). Weighting the logit function by the centroid of the distribution allows the logit function to more accurately predict the classification of a given utterance, especially when the utterance is in-domain or within a range (e.g., utterances that the system has been trained to recognize).

[0039] 3. Determine the Intent of Out-of-Scope Discourse

[0040] Neural networks suffer from an overconfidence problem. Overconfidence may occur because the confidence scores being generated for classification by a trained neural network (e.g., a trained NLP algorithm) may become uncorrelated with the true confidence scores. This problem may typically occur in deeper neural networks (i.e., neural network models with a higher number of layers). Although the output predictions of deep neural network models are generally more accurate than those of shallower neural network models, deep neural network models may produce highly confident misclassification predictions when the actual input is not well represented by the training data used to train the neural network model. Therefore, while deep neural networks are desirable for use due to their increased accuracy, the overconfidence problem associated with deep neural networks must be addressed to avoid performance issues with the neural networks.

[0041] Traditional techniques involving the use of a centroid-weighted logit function do not effectively address the overconfidence problem. Typically, for out-of-domain or out-of-scope utterances, using a centroid-weighted logit function can be said to treat the classifier as determining whether the utterance fits into a cluster of unparsed utterances. However, clustering of unparsed utterances can be inaccurate because out-of-domain or out-of-scope utterances can be scattered rather than clustered due to their different natural language definitions and grammars. In other words, using a centroid-weighted logit function for out-of-scope utterances would mean applying a centroid based on highly scattered clusters. Applying such a centroid can introduce inaccuracies into utterance classification, for example, by underestimating the probability that the intent is outside the domain of the chatbot.

[0042] 4. Augmented logit for classifying utterances as having unparsed intent

[0043] To overcome the above shortcomings, the present technology includes a system and method for using enhanced logit in a machine learning model to accurately predict whether out-of-range utterances correspond to unresolvable classifications. This can improve the accuracy of the utterance classifier, for example, providing an improved utterance classifier that can more accurately identify out-of-domain or out-of-range utterances that the classifier cannot reliably classify. This improvement in the ability to identify out-of-domain / out-of-range utterances can reduce the frequency with which the classifier attempts to classify utterances that the classifier has not been trained to classify, thereby reducing the number of unpredictable or incorrect classification results that may be generated by attempting to classify utterances that fall outside the capabilities of the classifier, and thus improving utterance classification overall. The present technology can include a machine learning model (e.g., a neural network) that is trained to predict whether an utterance or message represents a resolvable classification (e.g., the type of task a skill robot is configured to perform, the intent associated with the skill robot) or an unresolvable classification. The machine learning model can be configured to apply a logit function to generate a first logit value that predicts whether the utterance corresponds to a resolvable classification of the intent, and use an enhanced logit value that predicts whether the utterance corresponds to an unresolvable classification of the intent. In some examples, the augmented logit value replaces the logit function or is determined independently of the logit function.

[0044] In some instances, the augmented logit value for the unresolvable classification includes one of: (i) a statistical value determined based on a set of logit values ​​generated from a training dataset; (ii) a bounded value selected from a range of values ​​defined by a first logarithm of the probability corresponding to the probability of the unresolvable classification, the first logarithm of the probability being constrained to the range of values ​​by a bounding function and weighted by the centroid of the distribution associated with the unresolvable classification; and (iii) a weighted value generated by a second logarithm of the probability corresponding to the probability of the unresolvable classification, the second logarithm of the probability being scaled by a scaling factor, bounded by a bounding function, and weighted by the centroid of the distribution associated with the unresolvable classification, (iv) a hyperparameter optimized value generated based on hyperparameter tuning of the machine learning model, or (v) a learned value dynamically adjusted during training of the machine learning model.

[0045] B. Robotics and Analytical Systems

[0046] A robot (also known as a skill, chatbot, conversational robot, or conversational robot) is a computer program that can perform a conversation with an end user. A robot can typically respond to natural language messages (e.g., questions or comments) through a messaging application that uses natural language messages. An enterprise can use one or more robot systems to communicate with end users through messaging applications. The messaging application (which can be referred to as a channel) can be a messaging application preferred by the end user that the end user has already installed and is familiar with. Therefore, the end user does not need to download and install a new application in order to chat with the robot system. The messaging application can include, for example, over-the-top (OTT) messaging channels (such as Facebook Messenger, Facebook WhatsApp, WeChat, Line, Kik, Telegram, Talk, Skype, Slack, or SMS), virtual personal assistants (such as Amazon Dot, Echo or Show, Google Home, Apple HomePod, etc.), native or hybrid extended mobile and web application extensions / responsive mobile applications or web applications with chat functionality, or voice-based input (such as a device or application with an interface using Siri, Microsoft Cortana, Google Voice, or other voice input for interaction).

[0047] In some examples, a robot system can be associated with a uniform resource identifier (URI). A URI can use a string of characters to identify a robot system. A URI can be used as a webhook for one or more messaging application systems. A URI can include, for example, a uniform resource locator (URL) or a uniform resource name (URN). A robot system can be designed to receive messages (e.g., hypertext transfer protocol (HTTP) post call messages) from a messaging application system. An HTTP post call message can point to a URI from a messaging application system. In some embodiments, the message can be different from an HTTP post call message. For example, a robot system can receive a message from a short message service (SMS). Although the discussion herein may refer to communications received by a robot system as messages, it should be understood that a message can be an HTTP post call message, an SMS message, or any other type of communication between two systems.

[0048] End users can interact with the robot system through conversational interactions (sometimes called conversational user interfaces (UIs)), just as people interact with each other. In some cases, the interaction can include the end user saying "Hello" to the robot and the robot responding with "Hi" and asking the end user how the robot can help. In some cases, the interaction can also be transactional, such as with a banking robot, such as transferring money from one account to another; informational, such as with an HR robot, such as inquiring about vacation balances; or retail, such as discussing returning a purchase or seeking technical support.

[0049] In some embodiments, the robot system can intelligently handle end-user interactions without interacting with the administrator or developer of the robot system. For example, an end user can send one or more messages to the robot system in order to achieve a desired goal. The message can include some content, such as text, emoticons, audio, images, video, or other methods of conveying the message. In some embodiments, the robot system can convert the content into a standardized form (e.g., a Representational State Transfer (REST) ​​call to an enterprise service with appropriate parameters) and generate a natural language response. The robot system can also prompt the end user for additional input parameters or request other additional information. In some embodiments, the robot system can also initiate communication with the end user rather than passively responding to the end user's utterances. Described herein are various techniques for identifying explicit calls to a robot system and determining the input of the called robot system. In some embodiments, explicit call analysis is performed by a master robot based on detecting a call name in an utterance. In response to detecting the call name, the utterance can be refined for input to a skill robot associated with the call name.

[0050] A conversation with a robot can follow a specific conversation flow that includes multiple states. The flow can define what will happen next based on input. In some embodiments, a state machine that includes user-defined states (e.g., end-user intent) and actions to be taken in or between states can be used to implement the robot system. The conversation can take different paths based on the end-user input, which may affect the decision made by the robot for that flow. For example, in each state, based on the end-user input or utterance, the robot can determine the end-user's intent in order to determine the next appropriate action to take. As used herein and in the context of utterances, the term "intent" refers to the intention of the user providing the utterance. For example, a user may intend for the robot to participate in a conversation for ordering pizza, so that the user's intent can be expressed by the utterance "Order pizza". User intent can point to a specific task that the user wants the chatbot to perform on the user's behalf. Therefore, utterances can be expressed as questions, commands, requests, etc. that reflect the user's intent. Intent can include a goal that the end-user wants to accomplish.

[0051] In the context of chatbot configuration, the term "intent" as used herein refers to configuration information used to map user utterances to specific tasks / actions or specific types of tasks / actions that the chatbot can perform. To distinguish the intent of an utterance (i.e., user intent) from the chatbot's intent, the latter is sometimes referred to herein as a "bot intent." A bot intent can include a set of one or more utterances associated with the intent. For example, an intent to order pizza can include various permutations of utterances expressing a desire to place an order for pizza. These associated utterances can be used to train the chatbot's intent classifier so that the intent classifier can subsequently determine whether an input utterance from the user matches the intent to order pizza. A bot intent can be associated with one or more dialog flows in a certain state that are used to initiate a conversation with the user. For example, the first message for an intent to order pizza might be the question "What kind of pizza would you like?" In addition to the associated utterances, a bot intent can further include named entities related to the intent. For example, an intent to order pizza can include variables or parameters for performing the task of ordering pizza, such as topping 1, topping 2, pizza type, pizza size, and number of pizzas. The values ​​of the entities are typically acquired through conversation with the user.

[0052] In some instances, an utterance is analyzed to determine whether it contains an invocation name for a skill bot. If no invocation name is found, the utterance is considered non-explicit and the process continues using an intent classifier (e.g., a trained model). If a invocation name is determined to be present, the utterance is considered explicit and the process continues to determine which parts of the utterance are associated with the invocation name. In instances where a trained model is invoked, the entire received utterance is provided as input to the intent classifier.

[0053] The intent classifier that receives the utterance can be the intent classifier of the master robot (e.g. Figure 2 242 in the intent classifier). The intent classifier can be a machine learning based or rule-based classifier that is trained using keyword-enriched data to determine whether the intent of the utterance matches a system intent (e.g., exit, help) or a specific skill robot. As explained herein, the intent analysis performed by the master robot can be limited to matching to a specific skill robot without determining which intent within that specific skill robot is the best match for the utterance. Thus, the intent classifier that receives the utterance can identify the specific skill robot to invoke. Alternatively, if the utterance represents a specific system intent (e.g., the utterance contains the word "exit" or "help"), the intent classifier that receives the utterance can identify that specific system intent to trigger a conversation between the master robot and the user based on a dialog flow configured for that specific system intent.

[0054] In an instance where there is a call name, one or more explicit call rules are applied to determine which parts of the discourse are associated with the call name. This determination can be based on an analysis of the sentence structure of the discourse using POS tags, dependency information and / or other extracted information received together with the discourse. For example, the part associated with the call name can be a noun phrase comprising the call name or a prepositional object corresponding to the call name. Any part associated with the call name as determined based on processing will be removed. Other parts (e.g., prepositions) that are not needed for conveying the meaning of the discourse can also be removed. Removing some parts of the discourse generates input for the skill robot associated with the call name. If any part of the received discourse still remains after removal, the remaining part forms a new discourse to be input to the skill robot, e.g., as a text string. Otherwise, if the received discourse is completely removed, the input can be an empty string. Thereafter, the skill robot associated with the call name is called, and the input generated is provided to the skill robot.

[0055] Upon receiving the generated input, the invoked skill bot processes the input, for example, by performing intent analysis using the skill bot's intent classifier, which has been trained to identify bot intents that match the user intent expressed in the input. Identification of a matching bot intent can cause the skill bot to perform a specific action or begin a conversation with the user according to the dialog flow associated with the matching bot intent. For example, if the input is an empty string, the conversation can begin in the default state defined for the dialog flow, such as a welcome message. Alternatively, if the input is not an empty string, the conversation can begin in some intermediate state, for example, because the input contains the value of an entity or some other information that the skill bot no longer needs to ask the user for because it has already received this information as part of the input. As another example, the skill bot may determine that it cannot process the input (e.g., because the confidence score for each bot intent configured for the skill bot is below a certain threshold). In this case, the skill bot can return the input to the master bot for processing (e.g., using the master bot's intent classifier for intent analysis), or the skill bot can prompt the user for clarification.

[0056] 1. Overall environment

[0057] Figure 1 is a simplified block diagram of an environment 100 incorporating a chatbot system according to some embodiments. The environment 100 includes a Digital Assistant Builder Platform (DABP) 102 that enables users of the DABP 102 to create and deploy digital assistants or chatbot systems. The DABP 102 can be used to create one or more digital assistants (or DAs) or chatbot systems. For example, Figure 1 As shown, a user 104 representing a particular business can use DABP 102 to create and deploy a digital assistant 106 for users of the particular business. For example, DABP 102 can be used by a bank to create one or more digital assistants for use by the bank's customers. Multiple businesses can use the same DABP 102 platform to create digital assistants. As another example, the owner of a restaurant (e.g., a pizza shop) can use DABP 102 to create and deploy a digital assistant that enables customers of the restaurant to order food (e.g., order pizza).

[0058] For purposes of this disclosure, a "digital assistant" is an entity that assists a user of the digital assistant in completing various tasks through natural language conversation. A digital assistant may be implemented using software alone (e.g., a digital assistant is a digital entity implemented using a program, code, or instructions executable by one or more processors), hardware, or a combination of hardware and software. A digital assistant may be embodied or implemented in various physical systems or devices, such as computers, mobile phones, watches, appliances, vehicles, and the like. A digital assistant is also sometimes referred to as a chatbot system. Therefore, for purposes of this disclosure, the terms digital assistant and chatbot system are interchangeable.

[0059] A digital assistant, such as digital assistant 106 built using DABP 102, can be used to perform various tasks via natural language-based conversations between the digital assistant and its user 108. As part of a conversation, the user can provide one or more user inputs 110 to the digital assistant 106 and obtain responses 112 in return from the digital assistant 106. The conversation can include one or more of the inputs 110 and the responses 112. Through these conversations, the user can request one or more tasks to be performed by the digital assistant, and in response, the digital assistant is configured to perform the user-requested tasks and respond to the user with an appropriate response.

[0060] User input 110 is typically in the form of a natural language and is referred to as an utterance. User utterance 110 can be in textual form, such as when a user types a sentence, question, text snippet, or even a single word and provides it as input to the digital assistant 106. In some embodiments, user utterance 110 can be in the form of audio input or speech, such as when a user speaks or pronounces something that is provided as input to the digital assistant 106. The utterance is typically in the form of a language spoken by the user 108. For example, the utterance can be in English or some other language. When the utterance is in speech form, the speech input is converted into an utterance in textual form in that particular language, and the text utterance is then processed by the digital assistant 106. Various speech-to-text processing techniques can be used to convert the speech or audio input into textual utterances, which are then processed by the digital assistant 106. In some embodiments, the speech-to-text conversion can be performed by the digital assistant 106 itself.

[0061] An utterance (which may be a text utterance or a voice utterance) may be a fragment, a sentence, multiple sentences, one or more words, one or more questions, a combination of the above types, and the like. The digital assistant 106 is configured to apply natural language understanding (NLU) technology to the utterance to understand the meaning of the user input. As part of the NLU processing for the utterance, the digital assistant 106 is configured to perform processing for understanding the meaning of the utterance, which involves identifying one or more intents and one or more entities corresponding to the utterance. After understanding the meaning of the utterance, the digital assistant 106 may perform one or more actions or operations in response to the understood meaning or intent. For the purposes of this disclosure, it is assumed that these utterances are text utterances that have been directly provided by the user 108 of the digital assistant 106, or are the result of converting the input utterance into text form. However, this is not intended to be limiting or restrictive in any way.

[0062] For example, the input of user 108 can request to order pizza by providing an utterance such as “I want to order a pizza”. After receiving such an utterance, the digital assistant 106 is configured to understand the meaning of the utterance and take appropriate action. Appropriate action may involve, for example, responding to the user with questions requesting user input regarding the type of pizza the user desires to order, the size of the pizza, and any toppings on the pizza. The responses provided by the digital assistant 106 may also be in natural language form, and typically in the same language as the input utterance. As part of generating these responses, the digital assistant 106 may perform natural language generation (NLG). In order for the user to order pizza via a conversation between the user and the digital assistant 106, the digital assistant may guide the user to provide all the necessary information for pizza ordering, and then order the pizza at the end of the conversation. The digital assistant 106 may end the conversation by outputting information indicating that the pizza has been ordered to the user.

[0063] At a conceptual level, the digital assistant 106 performs various processing in response to an utterance received from the user. In some embodiments, the processing involves a series of processing steps or a pipeline of processing steps, including, for example, understanding the meaning of the input utterance (sometimes referred to as natural language understanding (NLU), determining an action to be performed in response to the utterance, causing the action to be performed if appropriate, generating a response to be output to the user in response to the user utterance, outputting the response to the user, etc. NLU processing may include parsing the received input utterance to understand the structure and meaning of the utterance, refining and reformulating the utterance to develop a more understandable form (e.g., a logical form) or structure for the utterance. Generating a response may include using NLG techniques.

[0064] The NLU processing performed by a digital assistant (such as digital assistant 106) may include various NLP-related processing such as sentence grammatical analysis (e.g., tokenization, classification by inflectional form, identification of part-of-speech tags for sentences, identification of named entities in sentences, generation of dependency trees to represent sentence structure, division of sentences into clauses, analysis of separate clauses, resolution of references, execution of chunks, etc.). In some embodiments, the NLU processing or part thereof is performed by the digital assistant 106 itself. In some other embodiments, the digital assistant 106 may use other resources to perform part of the NLU processing. For example, the syntax and structure of the input utterance sentence can be identified by processing the sentence using a grammatical analyzer, a part-of-speech tagger, and / or a named entity recognizer. In one embodiment, for the English language, a grammatical analyzer, a part-of-speech tagger, and a named entity recognizer such as that provided by the Stanford Natural Language Processing (NLP) group are used to analyze sentence structure and syntax. These are provided as part of the Stanford CoreNLP toolkit.

[0065] Although the various examples provided in this disclosure show utterances in the English language, this is meant as an example only. In some embodiments, the digital assistant 106 is also capable of processing utterances in languages ​​other than English. The digital assistant 106 may provide subsystems (e.g., components that implement NLU functionality) that are configured to perform processing for different languages. These subsystems may be implemented as pluggable units that can be called from the NLU core server using service calls. This makes NLU processing flexible and extensible for each language, including allowing different processing orders. Language packs may be provided for individual languages, wherein the language packs may register a list of subsystems that can be served from the NLU core server.

[0066] Digital assistants (such as those described in the preceding text) can be made available through various channels, such as, but not limited to, via certain applications, via social media platforms, via various messaging services and applications, and other applications or channels. Figure 1 106) is available or accessible to its user 108. A single digital assistant can configure multiple channels for itself, so that a single digital assistant can run on and be accessed through different services at the same time.

[0067] A digital assistant or chatbot system typically includes or is associated with one or more skills. In some embodiments, these skills are separate chatbots (called skill bots) that are configured to interact with the user and complete specific types of tasks (such as tracking inventory, submitting time cards, creating expense reports, ordering food, checking bank accounts, making appointments, purchasing widgets, etc.). For example, Figure 1In the depicted embodiment, the digital assistant or chatbot system 106 includes skills 116 - 1 , 116 - 2 , etc. For purposes of this disclosure, the terms “skill” and “skills” are used synonymously with the terms “skill bot” and “skill bots,” respectively.

[0068] Each skill associated with the digital assistant helps the user of the digital assistant complete tasks through a conversation with the user, where the conversation can include a combination of text or audio input provided by the user and responses provided by the skill bot. These responses can be in the form of text or audio messages to the user and / or using simple user interface elements (e.g., a list of choices) presented to the user for selection.

[0069] There are various ways to associate or add skills or skill bots to a digital assistant. In some instances, a skill bot can be developed by an enterprise and then added to a digital assistant using DABP 102. In other instances, a skill bot can be developed and created using DABP 102 and then added to a digital assistant created using DABP 102. In still other instances, DABP 102 provides an online digital store (referred to as a "skill store") that offers a plurality of skills directed to a variety of tasks. The skills offered through the skill store can also be exposed to various cloud services. To add a skill to a digital assistant being generated using DABP 102, a user of DABP 102 can access the skill store via DABP 102, select the desired skill, and instruct that the selected skill be added to the digital assistant created using DABP 102. Skills from the skill store can be added to the digital assistant as is or in modified form (e.g., a user of DABP 102 can select and copy a particular skill bot provided by the skill store, customize or modify the selected skill bot, and then add the modified skill bot to a digital assistant created using DABP 102).

[0070] A digital assistant or chatbot system can be implemented using a variety of different architectures. For example, in some embodiments, a digital assistant created and deployed using DABP 102 can be implemented using a master robot / secondary (or child) robot paradigm or architecture. According to this paradigm, the digital assistant is implemented as a master robot that interacts with one or more secondary robots that are skill robots. For example, in Figure 1 In the depicted embodiment, the digital assistant 106 includes a master robot 114 and skill robots 116-1, 116-2, etc., which are secondary robots to the master robot 114. In some embodiments, the digital assistant 106 itself is considered to act as the master robot.

[0071] A digital assistant implemented according to a master-slave robot architecture enables users of the digital assistant to interact with multiple skills through a unified user interface (i.e., via a master robot). When a user engages with the digital assistant, the master robot receives the user input. The master robot then performs processing to determine the meaning of the user's input utterance. The master robot then determines whether the task requested by the user in the utterance can be handled by the master robot itself. Otherwise, the master robot selects an appropriate skill robot to handle the user request and routes the conversation to the selected skill robot. This enables users to engage in a conversation with the digital assistant through a common, single interface while still providing the ability to use multiple skill robots configured to perform specific tasks. For example, for a digital assistant developed for an enterprise, the digital assistant's master robot can be connected to a skill robot interface with specific functions, such as a CRM robot for performing functions related to customer relationship management (CRM), an ERP robot for performing functions related to enterprise resource planning (ERP), an HCM robot for performing functions related to human capital management (HCM), and so on. In this way, the end user or consumer of the digital assistant only needs to know how to access the digital assistant through the common master robot interface, and multiple skill robots are provided in the background to handle user requests.

[0072] In some embodiments, in a master robot / secondary robot infrastructure, the master robot is configured to understand a list of available skill robots. The master robot can access metadata identifying the various available skill robots and, for each skill robot, access the capabilities of the skill robot, including the tasks that can be performed by the skill robot. Upon receiving a user request in the form of an utterance, the master robot is configured to identify or predict a specific skill robot from the multiple available skill robots that can best serve or handle the user request. The master robot then routes the utterance (or portion of the utterance) to that specific skill robot for further processing. Thus, control flows from the master robot to the skill robots. The master robot can support multiple input channels and output channels.

[0073] Although Figure 1 The embodiment shown in FIG. 1 shows the digital assistant 106 including a master robot 114 and skill robots 116 - 1 , 116 - 2 , and 116 - 3 , but this is not intended to be limiting. The digital assistant may include various other components (e.g., other systems and subsystems) that provide the functionality of the digital assistant. These systems and subsystems may be implemented solely in software (e.g., code, instructions stored on a computer-readable medium and executable by one or more processors), solely in hardware, or in implementations that use a combination of software and hardware.

[0074] DABP 102 provides infrastructure and various services and features that enable users of DABP 102 to create digital assistants (including one or more skill bots associated with a digital assistant). In some instances, a skill bot can be created by cloning an existing skill bot, e.g., cloning a skill bot provided by a skill store. As previously mentioned, DABP 102 provides a skill store or skill catalog that provides a plurality of skill bots for performing various tasks. A user of DABP 102 can clone a skill bot from the skill store. The cloned skill bot can be modified or customized as needed. In some other instances, a user of DABP 102 creates a skill bot from scratch using tools and services provided by DABP 102. As previously mentioned, the skill store or skill catalog provided by DABP 102 can provide a plurality of skill bots for performing various tasks.

[0075] In some embodiments, at a high level, creating or customizing a skill bot involves the following steps:

[0076] (1) configuring settings for the new skill bot

[0077] (2) configuring one or more intents for the skill bot

[0078] (3) configuring one or more entities for the one or more intents

[0079] (4) training the skill bot

[0080] (5) creating a dialog flow for the skill bot

[0081] (6) adding custom components to the skill bot as needed

[0082] (7) testing and deploying the skill bot

[0083] Each of the above steps is briefly described below.

[0084] (1) configuring settings for the new skill bot - Various settings can be configured for a skill bot. For example, a skill bot designer can specify one or more invocation names for the skill bot being created. These invocation names can then be used by users of the digital assistant to explicitly invoke the skill bot. For example, a user can enter an invocation name in a user utterance to explicitly invoke the corresponding skill bot.

[0085] (2) Configuring the skill robot with one or more intents and associated example utterances - The skill robot designer specifies one or more intents (also called robot intents) for the skill robot being created. The skill robot is then trained based on these specified intents. These intents represent the categories or classifications that the skill robot is trained to infer for the input utterance. Upon receiving an utterance, the trained skill robot infers the intent of the utterance, where the inferred intent is selected from a predefined set of intents used to train the skill robot. The skill robot then takes appropriate actions in response to the utterance based on the intent inferred for the utterance. In some instances, the intent of the skill robot represents the tasks that the skill robot can perform for the user of the digital assistant. Each intent is given an intent identifier or intent name. For example, for a skill robot trained for a bank, the intents specified for the skill robot may include "CheckBalance", "TransferMoney", "DepositCheck", etc.

[0086] For each intent defined for a skill bot, the skill bot designer can also provide one or more example utterances that represent and illustrate the intent. These example utterances are intended to represent the utterances that a user might enter into the skill bot for that intent. For example, for the CheckBalance intent, example utterances might include “What’s my savings account balance?”, “How much is in my checking account?”, “How much money do I have in my account?”, etc. Thus, various permutations of typical user utterances can be designated as example utterances for an intent.

[0087] These intents and their associated example utterances serve as training data for training the skill bot. A variety of different training techniques can be used. As a result of this training, a machine learning model is generated that is configured to take an utterance as input and output an intent inferred by the machine learning model for that utterance. In some instances, the input utterance is provided to an intent analysis engine that is configured to use the trained model to predict or infer the intent of the input utterance. The skill bot can then take one or more actions based on the inferred intent.

[0088] (3) Configuring entities for one or more intents of a skill bot - In some instances, additional context may be needed to enable a skill bot to respond appropriately to a user utterance. For example, there may be situations where user input utterances resolve to the same intent in a skill bot. For example, in the example above, the utterances "What's my savings account balance?" and "How much is in my checking account?" both resolve to the same intent, CheckBalance, but these utterances are different requests asking for different things. To clarify such requests, one or more entities are added to the intent. Using the example of a banking skill bot, an entity called AccountType (which defines values ​​called "checking" and "saving") can enable the skill bot to parse the user request and respond appropriately. In the example above, although these utterances resolve to the same intent, the values ​​associated with the AccountType entity are different for the two utterances. This enables the skill bot to perform potentially different actions for the two utterances, even though the two utterances resolve to the same intent. One or more entities can be specified for certain intents configured for a skill bot. Thus, entities are used to add context to the intent itself. Entities help more fully describe intent and enable skill bots to complete user requests.

[0089] In some embodiments, there are two types of entities: (a) built-in entities provided by DABP 102; and (b) custom entities that can be specified by the skill bot designer. Built-in entities are general entities that can be used with various bots. Examples of built-in entities include, but are not limited to, entities related to time, date, address, number, email address, duration, recurring time period, currency, phone number, URL, etc. Custom entities are used for more customized applications. For example, for a banking skill, an AccountType entity can be defined by the skill bot designer to implement various banking transactions by checking the user's input for keywords such as checking account, savings, credit card, etc.

[0090] (4) Training a Skill Bot - A skill bot is configured to receive user input in the form of utterances, parse or otherwise process the received input and identify or select intents associated with the received user input. As indicated above, the skill bot must be trained for this purpose. In some embodiments, the skill bot is trained based on intents configured for the skill bot and example utterances associated with the intents (collectively referred to as training data) so that the skill bot can parse the user input utterance into one of the intents for which it is configured. In some embodiments, the skill bot uses a machine learning model that is trained using the training data and allows the skill bot to discern what the user said (or in some cases, is attempting to say). The DABP 102 provides a variety of different training techniques that can be used by skill bot designers to train skill bots, including various machine learning-based training techniques, rule-based training techniques and / or combinations thereof. In some embodiments, a portion (e.g., 80%) of the training data is used to train the skill bot model and another portion (e.g., the remaining 20%) is used to test or validate the model. Once trained, the trained model (sometimes also referred to as a trained skill bot) can be used to process user utterances and respond to user utterances. In some cases, a user's utterance may be a question that requires only a single answer and no additional conversation. To handle this, a Q&A intent can be defined for the skill bot. This enables the skill bot to output a response to the user's request without having to update the dialog definition. A Q&A intent is created in a similar manner to a regular intent. The dialog flow for a Q&A intent can be different from that for a regular intent.

[0091] (5) Creating a conversation flow for a skill bot - The conversation flow specified for a skill bot describes how the skill bot reacts when parsing different intents of the skill bot in response to received user input. The conversation flow defines the actions or behaviors that the skill bot will take, for example, how the skill bot responds to user utterances, how the skill bot prompts the user for input, and how the skill bot returns data. The conversation flow is like a flowchart that the skill bot follows. The skill bot designer specifies the conversation flow using a language such as markdown. In some embodiments, a version of YAML called OBotML can be used to specify the conversation flow for a skill bot. The conversation flow definition for the skill bot acts as a model for the conversation itself, which is a model that enables the skill bot designer to orchestrate the interactions between the skill bot and the user that the skill bot serves.

[0092] In some embodiments, the dialog flow definition for a skill bot consists of three parts:

[0093] (a) Context

[0094] (b) Default Transformation Section

[0095] (c) Status section

[0096] Context section - Skill bot designers can define variables used in conversation flows in the context section. Other variables that can be named in the context section include, but are not limited to, variables for error handling, variables for built-in or custom entities, and user variables that enable skill bots to recognize and save user preferences.

[0097] Default Transitions Section - A skill bot's transitions can be defined in either the Dialog Flow State section or the Default Transitions section. Transitions defined in the Default Transitions section act as fallbacks and are triggered when no applicable transition is defined within a state or when the conditions required to trigger a state transition are not met. The Default Transitions section can be used to define routes that allow a skill bot to gracefully handle unexpected user actions.

[0098] State Part - A dialog flow and its associated operations are defined as a sequence of transient states that manage the logic within the dialog flow. Each state node within a dialog flow definition names a component that provides the functionality required at that point in the dialog. Thus, states are structured around components. States contain properties specific to a component and define transitions to other states that are triggered after the component executes.

[0099] You can use the state section to handle special case scenarios. For example, you might sometimes want to give the user the option to temporarily let the first skill they're engaged with do something in a second skill within the digital assistant. For example, if the user is busy in a conversation with a shopping skill (e.g., the user has made some choices about a purchase), the user might want to jump to a banking skill (e.g., the user might want to make sure they have enough money for the purchase) and then return to the shopping skill to complete the user's order. To address this, an action in the first skill can be configured to initiate an interaction with a different second skill in the same digital assistant and then return to the original flow.

[0100] (6) Adding Custom Components to a Skill Bot - As described above, the states specified in the skill bot's conversation flow name the components that provide the required functionality for the states. Components enable the skill bot to perform functions. In some embodiments, DABP 102 provides a set of preconfigured components for performing a variety of functions. A skill bot designer can select one or more of these preconfigured components and associate them with a state in the skill bot's conversation flow. A skill bot designer can also use the tools provided by DABP 102 to create custom or new components and associate the custom components with one or more states in the skill bot's conversation flow.

[0101] (7) Testing and deploying skill robots - DABP 102 provides several features that enable skill robot designers to test skill robots that are under development. Skill robots can then be deployed and included in digital assistants.

[0102] While the above description describes how skill robots are created, similar techniques can also be used to create digital assistants (or host robots). At the host robot or digital assistant level, the digital assistant can be configured with built-in system intents. These built-in system intents are used to identify general tasks that the digital assistant itself (i.e., the host robot) can handle without invoking a skill robot associated with the digital assistant. Examples of system intents defined for the host robot include: (1) Exit: applicable when a user signals a desire to exit a current session or context in the digital assistant; (2) Help: applicable when a user requests help or direction; and (3) Unresolved Intent: applicable to user input that does not closely match the exit intent and the help intent. The digital assistant also stores information about one or more skill robots associated with the digital assistant. This information enables the host robot to select a particular skill robot for processing an utterance.

[0103] At the host robot or digital assistant level, when a user inputs a phrase or utterance to the digital assistant, the digital assistant is configured to perform processing to determine how to route the utterance and the related session. The digital assistant uses a routing model to determine this, which can be rule-based, AI-based, or a combination thereof. The digital assistant uses the routing model to determine whether a session corresponding to a user input utterance is to be routed to a particular skill for processing, to be processed by the digital assistant or host robot itself according to built-in system intents, or to be handled as a different state in a current session flow.

[0104] In some embodiments, as part of this processing, the digital assistant determines whether the user input utterance explicitly identifies a skill robot using its invocation name. If the invocation name is present in the user input, the invocation name is considered an explicit invocation of a skill robot corresponding to the invocation name. In this scenario, the digital assistant can route the user input to the explicitly invoked skill robot for further processing. In some embodiments, if no specific invocation or explicit invocation is present, the digital assistant evaluates the received user input utterance and calculates confidence scores for the system intents and skill robots associated with the digital assistant. The scores calculated for the skill robots or system intents indicate how likely it is that the user input represents the task that the skill robot is configured to perform or represents a system intent. Any system intent or skill robot whose associated calculated confidence score exceeds a threshold (e.g., a confidence threshold routing parameter) is selected as a candidate for further evaluation. The digital assistant then selects a specific system intent or skill robot from the identified candidates for further processing of the user input utterance. In some embodiments, after one or more skill robots are identified as candidates, the intents associated with those candidate skills are evaluated (according to an intent model for each skill), and a confidence score is determined for each intent. Typically, any intent with a confidence score exceeding a threshold (e.g., 70%) is considered a candidate intent. If a specific skill bot is selected, the user utterance is routed to that skill bot for further processing. If a system intent is selected, the main bot itself performs one or more actions based on the selected system intent.

[0105] 2. Components of the main robot system

[0106] Figure 2 2 is a simplified block diagram of a master robot (MB) system 200 according to some embodiments. MB system 200 can be implemented solely in software, solely in hardware, or in a combination of hardware and software. MB system 200 includes a pre-processing subsystem 210, a multiple intent subsystem (MIS) 220, an explicit invocation subsystem (EIS) 230, a skill robot invoker 240, and data storage 250. Figure 2 The MB system 200 depicted in FIG is merely an example of a component arrangement in a master robot. Those skilled in the art will recognize many possible variations, alternatives, and modifications. For example, in some embodiments, the MB system 200 may have more Figure 2 More or fewer systems or components than those shown may be used, may combine two or more subsystems, or may have a different configuration or arrangement of subsystems.

[0107] The pre-processing subsystem 210 receives an utterance "A" 202 from a user and processes the utterance through a language detector 212 and a language parser 214. As indicated above, the utterance can be provided in various ways, including audio or text. The utterance 202 can be a sentence fragment, a complete sentence, multiple sentences, etc. The utterance 202 can include punctuation. For example, if the utterance 202 is provided as audio, the pre-processing subsystem 210 can convert the audio to text using a speech-to-text converter (not shown) that inserts punctuation (e.g., commas, semicolons, periods, etc.) into the resulting text.

[0108] Language detector 212 detects the language of utterance 202 based on the text of utterance 202. The way utterance 202 is processed depends on the language, as each language has its own grammar and semantics. Differences between languages ​​are taken into account when analyzing the syntax and structure of the utterance.

[0109] The language parser 214 parses the utterance 202 to extract part-of-speech (POS) tags for each linguistic unit (e.g., word) in the utterance 202. POS tags include, for example, noun (NN), pronoun (PN), verb (VB), etc. The language parser 214 may also tokenize the linguistic units of the utterance 202 (e.g., converting each word into a separate token) and classify words by inflectional form. A lemma is a primary form of a group of words as represented in a dictionary (e.g., "run" is a lemma for run, runs, ran, running, etc.). Other types of preprocessing that the language parser 214 may perform include chunking of compound expressions, e.g., combining "credit" and "card" into a single expression "credit_card". The language parser 214 may also identify relationships between words in the utterance 202. For example, in some embodiments, the language parser 214 generates a dependency tree that indicates which part of the utterance (e.g., a specific noun) is a direct object, which part of the utterance is a preposition, etc. The results of the processing performed by the language parser 214 form the extracted information 205 and are provided as input to the MIS 220 along with the utterance 202 itself.

[0110] As indicated above, utterance 202 may include more than one sentence. For the purposes of detecting multiple intents and explicit calls, utterance 202 may be treated as a single unit even if it includes multiple sentences. However, in some embodiments, pre-processing may be performed, for example by pre-processing subsystem 210, to identify individual sentences from the multiple sentences for use in multiple intent analysis and explicit call analysis. Generally, MIS 220 and EIS 230 produce substantially the same results regardless of whether utterance 202 is processed at the level of individual sentences or as a single unit including multiple sentences.

[0111] The MIS 220 determines whether the utterance 202 represents multiple intents. Although the MIS 220 can detect the presence of multiple intents in the utterance 202, the processing performed by the MIS 220 does not involve determining whether the intent of the utterance 202 matches any intent that has been configured for the robot. Instead, the process of determining whether the intent of the utterance 202 matches the robot's intent can be performed by the intent classifier 242 of the MB system 200 or by the intent classifier of the skill robot (e.g., Figure 3 The processing performed by MIS 220 assumes that there is a bot (e.g., a specific skill bot or the master bot itself) that can process utterance 202. Therefore, the processing performed by MIS 220 does not need to know which bots are in the chatbot system (e.g., the identities of the skill bots registered with the master bot) or what intents have been configured for a particular bot.

[0112] To determine that an utterance 202 includes multiple intents, the MIS 220 applies one or more rules from a set of rules 252 in the data store 250. The rules applied to the utterance 202 depend on the language of the utterance 202 and may include sentence patterns that indicate the presence of multiple intents. For example, a sentence pattern may include a coordinating conjunction that connects two parts of a sentence (e.g., a conjunction), where the two parts correspond to different intents. If the utterance 202 matches the sentence pattern, it can be inferred that the utterance 202 represents multiple intents. It should be noted that an utterance with multiple intents does not necessarily have different intents (e.g., intents directed to different robots or to different intents within the same robot). Instead, an utterance can have different instances of the same intent, such as "Place a pizza order using payment account X, then place a pizza order using payment account Y."

[0113] As part of determining that the utterance 202 represents multiple intents, the MIS 220 also determines which portion of the utterance 202 is associated with each intent. The MIS 220 constructs a new utterance for separate processing for each intent represented in the utterance containing multiple intents to replace the original utterance (e.g., utterance "B" 206 and utterance "C" 208, as shown in FIG. Figure 2). Thus, the original utterance 202 can be split into two or more separate utterances that are processed one at a time. The MIS 220 uses the extracted information 205 and / or based on an analysis of the utterance 202 itself to determine which of the two or more utterances should be processed first. For example, the MIS 220 can determine that the utterance 202 contains a marker word that indicates that a particular intent should be processed first. The newly formed utterance corresponding to the particular intent (e.g., one of the utterances 206 or 208) will be sent first for further processing by the EIS 230. After the conversation triggered by the first utterance has ended (or has been temporarily paused), the next highest priority utterance (e.g., the other of the utterances 206 or 208) can then be sent to the EIS 230 for processing.

[0114] The EIS 230 determines whether the utterance it receives (e.g., utterance 206 or utterance 208) contains a call name for a skill robot. In some embodiments, each skill robot in the chatbot system is assigned a unique call name that distinguishes the skill robot from other skill robots in the chatbot system. A list of call names can be stored in the data store 250 as part of the skill robot information 254. When an utterance contains a word that matches the call name, the utterance is considered an explicit call. If the robot is not explicitly called, the utterance received by the EIS 230 is considered a non-explicit call utterance 234 and is input to the intent classifier of the master robot (e.g., intent classifier 242) to determine which robot to use to process the utterance. In some instances, the intent classifier 342 will determine that the master robot should process the non-explicit call utterance. In other instances, the intent classifier 242 will determine the skill robot to which the utterance is routed for processing.

[0115] The explicit call functionality provided by EIS 230 has several advantages. It can reduce the amount of processing that the master robot must perform. For example, when an explicit call is present, the master robot may not have to perform any intent classification analysis (e.g., using intent classifier 242), or may have to perform a simplified intent classification analysis to select a skill robot. Thus, explicit call analysis can enable the selection of a specific skill robot without resorting to intent classification analysis.

[0116] Furthermore, there may be cases where the functionality of multiple skill robots overlaps. This can occur, for example, if two skill robots handle overlapping intents or are very close to each other. In this case, it can be difficult for the master robot to identify which of the multiple skill robots to select based solely on intent classification analysis. In this scenario, explicit invocation eliminates ambiguity regarding the specific skill robot to use.

[0117] In addition to determining whether an utterance is an explicit call, EIS 230 is also responsible for determining whether any portion of the utterance should be used as input to the explicitly called skill bot. Specifically, EIS 230 may determine whether a portion of the utterance is irrelevant to the call. EIS 230 may perform this determination by analyzing the utterance and / or analyzing extracted information 205. Instead of sending the entire utterance received by EIS 230, EIS 230 may send the portion of the utterance that is irrelevant to the call to the called skill bot. In some instances, the input to the called skill bot is simply formed by removing any portion of the utterance that is associated with the call. For example, "I want to order pizza using Pizza Bot" can be shortened to "I want to order pizza" because "using Pizza Bot" is related to calling the Pizza Bot but not to any processing to be performed by the Pizza Bot. In some instances, EIS 230 may reformat the portion to be sent to the called bot, for example, to form a complete sentence. Thus, EIS 230 not only determines that there is an explicit call, but also determines what to send to the skill bot when there is an explicit call. In some instances, there may not be any text to be input to the invoked bot. For example, if the utterance is "Pizza Bot," EIS 230 may determine that the pizza bot is being invoked, but there is no text to be processed by the pizza bot. In such a scenario, EIS 230 may indicate to skill bot invoker 240 that there is no content to send.

[0118] The skill bot invoker 240 invokes a skill bot in various ways. For example, the skill bot invoker 240 may invoke the bot in response to receiving an indication 235 that a particular skill bot has been selected as a result of an explicit invocation. The indication 235 may be sent by the EIS 230 along with the input from the explicitly invoked skill bot. In this scenario, the skill bot invoker 240 hands control of the conversation to the explicitly invoked skill bot. The explicitly invoked skill bot will determine an appropriate response to the input from the EIS 230 by treating the input as an independent utterance. For example, the response may be to perform a specific action or to start a new conversation in a specific state, where the initial state of the new conversation depends on the input sent from the EIS 230.

[0119] Another way in which the skill robot invoker 240 can invoke skill robots is through implicit invocation using an intent classifier 242. The intent classifier 242 can be trained using machine learning and / or rule-based training techniques to determine the likelihood that an utterance represents a task for which a particular skill robot is configured to perform. The intent classifier 242 is trained on different classes, one for each skill robot. For example, whenever a new skill robot is registered with the host robot, a list of example utterances associated with the new skill robot can be used to train the intent classifier 242 to determine the likelihood that a particular utterance represents a task that the new skill robot can perform. The parameters that result from this training (e.g., a set of parameter values for a machine learning model) can be stored as part of the skill robot information 254.

[0120] In some embodiments, the intent classifier 242 is implemented using a machine learning model, as described in further detail herein. Training of the machine learning model can involve inputting at least a subset of utterances from example utterances associated with various skill robots to generate, as output of the machine learning model, an inference as to which robot is the correct robot for handling any particular training utterance. For each training utterance, an indication of the correct robot for the training utterance can be provided as ground truth information. The behavior of the machine learning model can then be adapted (e.g., through backpropagation) to minimize the difference between the generated inference and the ground truth information.

[0121] In some embodiments, the intent classifier 242 determines, for each skill robot registered with the host robot, a confidence score indicating the likelihood that the skill robot can handle an utterance (e.g., a non-explicitly invoked utterance 234 received from the EIS 230). The intent classifier 242 can also determine a confidence score for each system-level intent that has been configured (e.g., help, exit). If a particular confidence score satisfies one or more conditions, then the skill robot invoker 240 will invoke the robot associated with that particular confidence score. For example, it can be required that a threshold confidence score value be satisfied. Thus, the output 245 of the intent classifier 242 is either an identification of a system intent or an identification of a particular skill robot. In some embodiments, in addition to satisfying a threshold confidence score value, a confidence score must exceed the next highest confidence score by a winning margin. Such a condition will enable routing to a particular skill robot when multiple skill robots each have a confidence score that exceeds the threshold confidence score value.

[0122] After identifying the robot based on the evaluation of the confidence score, the skill robot invoker 240 transfers processing to the identified robot. In the case of system intent, the identified robot is the master robot. Otherwise, the identified robot is a skill robot. Further, the skill robot invoker 240 determines what to provide as input 247 to the identified robot. As indicated above, in the case of an explicit invocation, the input 247 can be based on a portion of the utterance that is not associated with the invocation, or the input 247 can be nothing (e.g., an empty string). In the case of an implicit invocation, the input 247 can be the entire utterance.

[0123] The data store 250 includes one or more computing devices that store data used by the various subsystems of the master robot system 200. As explained above, the data store 250 includes rules 252 and skill bot information 254. The rules 252 include, for example, rules for determining when an utterance represents multiple intents by the MIS 220 and how to split utterances that represent multiple intents. The rules 252 further include rules for determining which parts of an utterance that explicitly calls a skill bot to be sent to the skill bot by the EIS 230. The skill bot information 254 includes the invocation names of the skill bots in the chatbot system, for example, a list of the invocation names of all skill bots registered with a particular master robot. The skill bot information 254 may also include information used by the intent classifier 242 to determine the confidence score for each skill bot in the chatbot system, for example, parameters of a machine learning model.

[0124] 3. Components of the Skill Robot System

[0125] Figure 3 is a simplified block diagram of a skill robot system 300 according to some embodiments. The skill robot system 300 is a computing system that can be implemented in software only, hardware only, or a combination of hardware and software. Figure 1 In some embodiments of the embodiments depicted in , the skill robot system 300 can be used to implement one or more skill robots within a digital assistant.

[0126] The skill robot system 300 includes an MIS 310, an intent classifier 320, and a conversation manager 330. The MIS 310 is similar to Figure 2The MIS 220 in FIG. 310 and provides similar functionality, including being operable to use rules 352 in the data store 350 to determine: (1) whether an utterance represents multiple intents, and if so, (2) how to split the utterance into separate utterances for each of the multiple intents. In some embodiments, the rules applied by the MIS 310 for detecting multiple intents and for splitting utterances are the same as the rules applied by the MIS 220. The MIS 310 receives the utterance 302 and the extracted information 304. The extracted information 304 is similar to Figure 2 The extracted information 205 in the skill robot system 300 can be generated using the language parser 214 or a language parser local to the skill robot system 300.

[0127] The intent classifier 320 can be combined with the above Figure 2 In some embodiments, the intent classifier 320 is trained in a similar manner to the intent classifier 242 discussed above and described in further detail herein. For example, in some embodiments, the intent classifier 320 is implemented using a machine learning model. The machine learning model of the intent classifier 320 is trained using at least a subset of example utterances associated with a particular skill bot as training utterances for a particular skill bot. The ground truth for each training utterance will be the particular bot intent associated with the training utterance.

[0128] The utterance 302 may be received directly from the user or provided by the master robot. When the utterance 302 is provided by the master robot (e.g., as a Figure 2 ), MIS 310 may be bypassed to avoid duplicating processing already performed by MIS 220. However, if utterance 302 is received directly from the user (e.g., during a conversation occurring after being routed to a skill bot), MIS 310 may process utterance 302 to determine whether utterance 302 represents multiple intents. If so, MIS 310 applies one or more rules to split utterance 302 into separate utterances for each intent, such as utterance "D" 306 and utterance "E" 308. If utterance 302 does not represent multiple intents, MIS 310 forwards utterance 302 to intent classifier 320 for intent classification without splitting utterance 302.

[0129] The intent classifier 320 is configured to match received utterances (e.g., utterances 306 or 308) with intents associated with the skill robot system 300. As explained above, a skill robot can be configured with one or more intents, each including at least one example utterance associated with the intent and used to train the classifier. Figure 2In an embodiment, the intent classifier 242 of the master robot system 200 is trained to determine confidence scores for individual skill robots and confidence scores for system intents. Similarly, the intent classifier 320 can be trained to determine a confidence score for each intent associated with the skill robot system 300. The classification performed by the intent classifier 242 is at the robot level, while the classification performed by the intent classifier 320 is at the intent level and is therefore more fine-grained. The intent classifier 320 has access to intent information 354. For each intent associated with the skill robot system 300, the intent information 354 includes a list of utterances that represent the intent and explain the meaning of the intent and are typically associated with tasks that can be performed by the intent. The intent information 354 can further include parameters generated as a result of training on the utterance list.

[0130] The conversation manager 330 receives, as output from the intent classifier 320, an indication 322 of a particular intent that the intent classifier 320 identifies as the best match to the utterance input to the intent classifier 320. In some instances, the intent classifier 320 may not determine any match. For example, if the utterance points to a system intent or an intent of a different skill bot, the confidence score calculated by the intent classifier 320 may fall below a threshold confidence score value. When this occurs, the skill bot system 300 may hand the utterance off to the master bot for processing, e.g., routing to a different skill bot. However, if the intent classifier 320 is successful in identifying the intent within the skill bot, the conversation manager 330 will initiate a conversation with the user.

[0131] The session initiated by the session manager 330 is a session specific to the intent identified by the intent classifier 320. For example, the session manager 330 can be implemented using a state machine configured to execute a dialog flow for the identified intent. The state machine can include a default starting state (e.g., when the intent is invoked without any additional input) and one or more additional states, each of which is associated with an action to be performed by the skill bot (e.g., executing a purchase transaction) and / or a dialog to be presented to the user (e.g., a question, a response). Thus, the session manager 330 can determine an action / dialog 335 upon receiving an indication 322 that an intent was recognized, and can determine additional actions or dialogs in response to subsequent utterances received during the session.

[0132] The data storage 350 includes one or more computing devices that store data used by the various subsystems of the skill robot system 300. Figure 3 As depicted, data store 350 may include rules 352 and intent information 354. In some embodiments, data store 350 may be integrated into the data store of a host robot or digital assistant, e.g. Figure 2 Data storage 250 in.

[0133] 4. Scheme for classifying utterances using trained intent classifiers

[0134] Figure 4 The diagram shows a method configured to train and utilize a classifier (e.g., about Figure 2 and Figure 3 242 or 320) of the chatbot system 400. Figure 4 As shown, the text classification performed by the chatbot system 400 in this example includes various stages: a machine learning model training stage 410; a skill robot invocation stage 415, which is used to determine the likelihood that an utterance represents a task that a particular skill robot is configured to perform; and an intent prediction stage 420, which is used to classify an utterance into one or more intents. The machine learning model training stage 410 establishes and trains one or more machine learning models 425a-425n ('n' represents any natural number) for use in other stages (the machine learning models may be referred to herein individually as machine learning models 425 or collectively as machine learning models 425). For example, the machine learning models 425 may include a model for determining the likelihood that an utterance represents a task that a particular skill robot is configured to perform, another model for predicting intent based on the utterances of a first type of skill robot, and another model for predicting intent based on the utterances of a second type of skill robot. Still other types of machine learning models may be implemented in other examples according to the present disclosure.

[0135] The machine learning model 425 can be a machine learning ("ML") model, such as a convolutional neural network ("CNN") (e.g., an initial neural network, a residual neural network ("Resnet")); or a recurrent neural network, such as a long short-term memory ("LSTM") model or a gated recurrent unit ("GRU") model; other variants of a deep neural network ("DNN") (e.g., a multi-label n binary DNN classifier or a multi-class DNN classifier for single intent classification. The machine learning model 425 can also be any other suitable ML model trained for natural language processing, such as a naive Bayes classifier, a linear classifier, a support vector machine, a bagging model such as a random forest model, a boosting model, or a similar model. Model), shallow neural networks, or a combination of one or more of these techniques—for example, CNN-HMM or MCNN (Multi-Scale Convolutional Neural Network). The chatbot system 400 can employ the same type of machine learning model or different types of machine learning models to determine the likelihood of a particular skill bot being configured to perform a task, to predict intent based on utterances of a first type of skill bot, and to predict intent based on utterances of a second type of skill bot. Still other types of machine learning models can be implemented in other examples according to the present disclosure.

[0136] To train the various machine learning models 425, the training phase 410 consists of three main components: dataset preparation 430, feature engineering 435, and model training 440. Dataset preparation 430 includes the process of loading data assets 445, splitting the data assets 445 into training and validation sets 445a-n so that the system can train and test the machine learning models 425, and performing basic preprocessing. Data assets 445 may include at least a subset of utterances from example utterances associated with various skill bots. As indicated above, utterances can be provided in various ways, including audio or text. Utterances can be sentence fragments, complete sentences, multiple sentences, etc. For example, if the utterances are provided as audio, data preparation 430 can convert the audio to text using a speech-to-text converter (not shown) that inserts punctuation (e.g., commas, semicolons, periods, etc.) into the resulting text. In some instances, the example utterances are provided by the client or customer. In other instances, the example utterances are automatically generated from a library of previous utterances (e.g., identifying utterances from the library that are specific to the skill the chatbot is to learn). The data assets 445 of the machine learning model 425 can include input text or audio (or input features of the text or audio frames) and labels 450 corresponding to the input text or audio (or input features) as a matrix or table of values. For example, for each training utterance, an indication of the correct robot for the training utterance can be provided as ground truth information for the label 450. The behavior of the machine learning model 425 can then be adapted (e.g., by backpropagation) to minimize the difference between the generated inferences and the ground truth information. Alternatively, for a particular skill robot, the machine learning model 425 can be trained using at least a subset of example utterances associated with the particular skill robot as training utterances. The ground truth information for the label 450 for each training utterance will be the particular robot intent associated with the training utterance.

[0137] In various embodiments, feature engineering 435 includes transforming data assets 445 into feature vectors and / or creating new features using data assets 445. Feature vectors can include count vectors as features, term frequency-inverse document frequency (TF-IDF) vectors as features (e.g., at the word level, n-gram level, or character level), word embeddings as features, text / NLP features, topic models as features, or a combination thereof. A count vector is a matrix representation of data assets 445, where each row represents an utterance, each column represents a word from the utterance, and each cell represents a frequency count of a particular word in the utterance. TF-IDF scores represent the relative importance of a word in an utterance. Word embeddings are a form of representing words and utterances using dense vector representations. The position of a word within the vector space is learned from the text and is based on the words surrounding the word when it is used. Text / NLP-based features can include the number of words in an utterance, the number of characters in an utterance, average word density, the number of punctuation marks, the number of capital letters, the number of title words, the frequency distribution of part-of-speech tags (e.g., nouns and verbs), or any combination thereof. Topic modeling is a technique for identifying groups of words (called topics) from a collection of utterances that contain the best information about the utterance.

[0138] In various embodiments, model training 440 includes training a classifier using the feature vectors and / or new features created in feature engineering 435. In some instances, the training process includes an iterative operation to find a set of parameters for the machine learning model 425 that minimizes or maximizes a cost function for the machine learning model 425 (e.g., minimizes a loss function or an error function). Each iteration can involve finding a set of parameters for the machine learning model 425 such that the value of the cost function using this set of parameters is less than or greater than the value of the cost function using another set of parameters in the previous iteration. A cost function can be constructed to measure the difference between the output predicted using the machine learning model 425 and the labels 450 contained in the data assets 445. Once the set of parameters is identified, the machine learning model 425 has been trained and can be used for predictions as designed.

[0139] In various embodiments, the training phase 410 further includes a logit determination 455. The logit determination 455 is configured to apply a logit function to the probability that the utterance is associated with a parsable classification. To determine the probability that the utterance is associated with an unparsable classification, the logit determination 455 may use an enhanced logit value instead of a logit function. The enhanced logit value may be processed (e.g., through an activation function) to determine a classification of the utterance, including a classification of the utterance as corresponding to having an unparsed intent. The use of the enhanced logit value by the logit determination 455 may improve the performance of the model and the chatbot, particularly in predicting whether the utterance is out of range or out of domain. Although the logit determination 455 is Figure 4 4 is shown as a sub-process of the training phase, but in some embodiments, such as when the enhanced logit value corresponding to the unresolvable classification is a learned value that can be dynamically adjusted during the training of the intent classifier, the logit determination 455 can form part of the model training 440. Additionally and / or alternatively, the enhanced logit value can be tuned by optimizing the hyperparameters of the intent classifier. Implementation details of the logit determination 455 are further described below.

[0140] In addition to data assets 445, labels 450, feature vectors and / or new features, other techniques and information can be employed to refine the training process of the machine learning model 425. For example, feature vectors and / or new features can be combined together to help improve the accuracy of the classifier or model. Additionally or alternatively, hyperparameters can be adjusted or optimized, for example, multiple parameters such as tree length, leaves, network parameters, etc. can be fine-tuned to obtain a best fit model. Although the training mechanisms described herein are primarily focused on training the machine learning model 425. These training mechanisms can also be used to fine-tune existing machine learning models 425 trained from other data assets. For example, in some cases, the machine learning model 425 may have been pre-trained using utterances specific to another skill robot. In these cases, the data assets 445 can be used to retrain the machine learning model 425, as discussed herein.

[0141] The machine learning model training phase 410 outputs trained machine learning models 425, including a task machine learning model 460 and an intent machine learning model 465. The task machine learning model 460 can be used in the skill bot invocation phase 415 to determine the likelihood that an utterance represents a task 470 that a particular skill bot is configured to perform, and the intent machine learning model 465 can be used in the intent prediction phase 420 to classify the utterance into one or more intents 475. In some instances, the skill bot invocation phase 415 and the intent prediction phase 420 can be performed independently using separate models in some examples. For example, the trained intent machine learning model 465 can be used in the intent prediction phase 420 to predict the intent of a skill bot without first identifying the skill bot in the skill bot invocation phase 415. Similarly, the task machine learning model 460 can be used in the skill bot invocation phase 415 to predict a task or skill bot to use for an utterance without first identifying the intent of the utterance in the intent prediction phase 420.

[0142] Alternatively, the skill bot invocation phase 415 and the intent prediction phase 420 can be performed sequentially, with one phase using the output of another phase as input, or with one phase being invoked in a specific manner based on the output of another phase for a particular skill bot. For example, for given text data 405, a skill bot invoker can invoke a skill bot via an implicit call using the skill bot invocation phase 415 and the task machine learning model 460. The task machine learning model 460 can be trained using machine learning and / or rule-based training techniques to determine the likelihood that an utterance represents a task that a particular skill bot 470 is configured to perform. Then, for the identified or invoked skill bot and given text data 405, the intent prediction phase 420 and the intent machine learning model 465 can be used to match received utterances (e.g., utterances within a given data asset 445) with intents 475 associated with the skill bot. As explained herein, a skill bot can be configured with one or more intents, each intent including at least one example utterance associated with the intent and used to train a classifier. In some embodiments, the skill robot invocation stage 415 and the task machine learning model 460 used in the master robot system are trained to determine the confidence score of each skill robot and the confidence score of the system intent. Similarly, the intent prediction stage 420 and the intent machine learning model 465 can be trained to determine the confidence score of each intent associated with the skill robot system. The classification performed by the skill robot invocation stage 415 and the task machine learning model 460 is at the robot level, while the classification performed by the intent prediction stage 420 and the intent machine learning model 465 is at the intent level and is therefore more fine-grained.

[0143] C.Logit function

[0144] Figure 5 A schematic diagram of an example neural network 500 is illustrated in accordance with some embodiments. Neural network 500 can be a machine learning model trained by a training system and implemented by a chatbot system, wherein the machine learning model is trained to predict whether an utterance corresponds to a particular intent classification. In some examples, neural network 500 includes an input layer 502, a hidden layer 504, and an output layer 506. In some examples, neural network 500 includes multiple hidden layers, wherein hidden layer 504 corresponds to the final hidden layer of the neural network.

[0145] The input layer 502 can receive input data or a representation of the input data (e.g., an n-dimensional array of values ​​representing an utterance) and apply one or more learned parameters to the input data to generate a set of outputs (e.g., a set of numerical values). The set of outputs can be processed by the hidden layer 504.

[0146] The hidden layer 504 can include one or more learned parameters that transform the set of outputs to a different feature space in which input data points from different classifications are better separated.

[0147] The output layer 506 can include a classification layer that maps the output from the hidden layer to logit values, where each value corresponds to a particular classification (e.g., a resolvable classification, a non-resolvable classification). In some instances, the output layer 506 includes an activation function for constraining the logit values to a set of probability values that sum to one. Thus, the logit values generated by the classification layer (logit function) for each classification can then be processed by the activation function of the output layer (also referred to herein as an “activation layer”) to predict a classification for an utterance.

[0148] As part of predicting a single intent from a set of output values, the machine learning model can employ a logit function (short for “logistic regression function”) in the network layers of the neural network 500. The output layer 506 can include a logit function for transforming intermediate outputs (e.g., probability values that predict whether an utterance corresponds to a particular classification) to logit values. In certain instances, the logit function is a logarithm of odds that receives as input a probability value corresponding to zero and one for a particular classification and outputs a logit value in an unbounded range between negative infinity to positive infinity. The logit function can be used to normalize each of the set of intermediate outputs such that the resulting set of logit values can express a symmetric unimodal probability distribution over the predicted output classifications. In mathematical terms, the logit function is defined as where p is the probability that the input corresponds to a particular classification. The probability p can be set in a range between (0, 1). The output logit can correspond to a logit value in a range between (-∞, +∞). In this way, the machine learning model can employ a logit function such that a subsequent activation function can predict the most likely output (e.g., intent) from other less likely outputs of the classifier.

[0149] As part of training the machine learning model (e.g., the intent classifier 320 of Figure 3 ), the model can learn a centroid for each classification, where the centroid is part of the logit function. In particular, the centroid can be used to weight the logit function, for example, as part of filtering the classifier output to separate possible intents from an utterance. In mathematical terms, the equation related to the logit function can be: logit i = f(x) * W i where x is the input to the model, f(x) is a set of transformations (e.g., highway network function), and Wi is the centroid of intent "i", which is used as a weighting factor for the model output for that intent. Thus, for an identified set of intents that an intent classifier has been trained to recognize, the centroid plays a role in classifying an utterance as belonging to one particular intent and not to others, for example, by using the centroid as the location for measuring distances between adjacent clusters.

[0150] The logit values ​​generated for each of the predicted output classes can then be processed through an activation function to map the probability represented by the logit value to a probability distribution over all predicted output classes. As an example of an activation function, a softmax function can be used to normalize the output of the network (e.g., the logit value corresponding to the order_pizza class) to a probability distribution over all predicted output classes (e.g., order_pizza, unresolved intent).

[0151] Before applying softmax, the logit values ​​can be negative, or greater than one; and may not sum to 1. In the case of applying softmax, each output will be in the interval (0,1), and the outputs will sum to 1. In addition, larger input logit values ​​will correspond to larger probabilities. In functional terms, the softmax function is expressed as: i=1:K and z is a set of K real numbers (z1:z K ). In some cases, the softmax function can be weighted by a basis factor b, creating a probability distribution that is more concentrated around the location of the maximum input value. In this case, the softmax function is Where β is a real number. In the case of intent classification, z i Can be set to f(x)*w i .

[0152] For a set of utterances, a resolved intent can match any of the identified intents for which an intent classifier can be trained. By definition, an unresolved intent can describe those utterances that the classifier cannot confidently place in any intent category. As an illustrative example, a skill robot trained to process pizza orders can be trained to classify utterances into one of three intents: "order_pizza", "cancel_pizza", and "unresolvedIntent". For example, utterances that intend to order a new pizza or modify an existing order can be classified under "order_pizza". In contrast, utterances that intend to cancel an existing order can be classified under "cancel_pizza". Finally, utterances that intend other things can be classified under "unresolvedIntent". For example, "what is the weather tomorrow" has nothing to do with pizza orders and should be classified under "unresolvedIntent". Therefore, in this example, "unresolvedIntent" is defined in a negative sense relative to "order_pizza" and "cancel_pizza". Discourse that does not match the latter conforms to the former.

[0153] Therefore, unresolved intent can be defined to cover utterances that do not fit into the set of identified intents. In this way, for unresolvable classification, the classifier output can be decentralized rather than aggregated. While the examples described herein focus on intent prediction, it should be understood that other classification tasks can be handled similarly. For example, a chatbot system can include a classifier to resolve "unresolved" utterances at other levels (such as skill bot invocations).

[0154] D. Techniques for classifying utterances as having unparsed intent using augmented logit

[0155] Applying the concept of centroid as a weighting factor in the logit function can be effective in many instances. However, for the “unresolved intent” classification, the effectiveness of using the centroid is greatly reduced. For a defined intent (such as “order_pizza”), utterances can cluster around the centroid. However, “unresolved_intent” is defined as the negative classification of all utterances outside a set of categories, which may show a relative lack of clustering. Therefore, learning the centroid may introduce errors in logistic regression through inaccurate weighting factors.

[0156] As an example, for the pizza skill bot described above, while utterances with ordering intent or canceling intent may cluster around the corresponding centroids, utterances ranging from weather to finance to asking about a specific person may fall into "unresolved_intent". Similarly, to direct inquiries to specific skill bots, the classifier can output the probability that the utterance invokes a specific skill bot or the probability that the utterance does not invoke any of the skill bots that the system has been trained to classify it. In each of these cases, the centroid of the unresolved classification may not be largely representative of the group characteristics, in part because the definition of the "unresolved" classification includes any utterance that does not map to a learned intent, skill, domain, or scope.

[0157] To overcome this and other problems, various embodiments are directed to techniques for using enhanced logit values ​​for unresolved intents in order to improve classification of utterances as unresolved intents. For example, a machine learning model (e.g., Figure 3 The intent classifier 320) can implement enhanced logit values ​​for unresolved intent classification. For example, instead of using the centroid as a weighting factor to calculate the logit value, the enhanced logit value for the unresolvable classification can be determined by using a modified logit function to include a trainable scaler parameter. Additionally or alternatively, the enhanced logit value for the unresolvable classification can be optimized via hyperparameter tuning of the machine learning model. In some instances, if the training data is insufficient to indicate a resolvable classification, the intent classifier classifies the output as an unresolvable classification. In effect, instead of determining whether an utterance invokes an unresolvable classification, the intent classifier can determine whether an utterance invokes a resolvable classification.

[0158] Wherever it is possible to classify an utterance (e.g., intent, scope, skill, etc.), an automated process that can be integrated into the chatbot system can be used to determine the augmented logit value, as in the case of Figure 1 、 Figure 2 and Figure 3 Advantageously, models and chatbots perform better when classifying out-of-range utterances using augmented logit values, at least in part because the augmented logit values ​​do not rely on the centroids of unresolvable clusters, which may not reflect the true clustering in the model output. Furthermore, because augmentation is applied automatically, customers or clients experience fewer misleading queries from the chatbot system.

[0159] As further explained below, the augmented logit value used to classify an utterance as an unparseable classification can be based on one of the following: (i) a statistical value determined based on a set of logit values ​​generated from a training dataset; (ii) a bounded value selected from a range of values ​​defined by a first logarithm of the probability corresponding to the probability of the unparseable classification, the first logarithm of the probability being constrained to the range of values ​​by a bounding function and weighted by the centroid of the distribution associated with the unparseable classification; (iii) a weighted value generated by a second logarithm of the probability corresponding to the probability of the unparseable classification, the second logarithm of the probability being scaled by a scaling factor, bounded by a bounding function, and weighted by the centroid of the distribution associated with the unparseable classification; (iv) a hyperparameter optimized value generated based on hyperparameter tuning of a machine learning model; or (v) a learned value dynamically adjusted during training of the machine learning model. Additionally or alternatively, a rule-based approach can be implemented as part of determining the augmented logit value. The rule-based approach can include a logical operator that identifies an augmented logit value for the unparseable classification from a set of augmented logit values. For example, augmented logit values ​​may be retrieved from a database or lookup table based on a query value generated by applying a set of rules to an utterance.

[0160] 1. Statistical values

[0161] In some embodiments, the enhanced logit value is a statistical value that replaces the logit function when determining whether an utterance corresponds to an out-of-range or out-of-domain classification. The logit value may include a value selected from a range of values. Reference function expression logit i =f(x)*W i , the function f(x)*W can be replaced by a predetermined number "A" that is not analytically classifiable i To set logit i (i.e., logit i =A). Such a configuration provides a probability function based on fixed values: In some examples, the statistic is set to 0.3, 0.5, 0.7, 0.9, 1.0, or any higher number. The statistic is determined such that the corresponding probability value indicates that the utterance will be classified as unresolved (eg, unresolved skill or unresolved intent).

[0162] In some embodiments, the statistic is based on the probability that an out-of-domain or out-of-range utterance in the training utterance set maps to an unparseable category. Determining an appropriate statistic can be Figure 4 , where the number may depend in part on the training dataset (e.g., data assets 445) applied in the training phase 410.

[0163] Figure 6A flow diagram illustrating an example process 600 for determining a statistic representing an augmented logit value for predicting whether an utterance corresponds to an unparseable classification is shown in accordance with some embodiments. Figure 6 The depicted processes may be implemented in software (e.g., code, instructions, programs) executed by one or more processing units (e.g., processors, cores) of the corresponding system, hardware, or a combination thereof. The software may be stored on a non-transitory storage medium (e.g., a memory device). Figure 6 The methods presented and described below are intended to be illustrative and not limiting. Figure 6 The various processing steps are depicted as occurring in a particular sequence or order, but this is not intended to be limiting. In some embodiments, the steps may be performed in a different order or some steps may be performed in parallel. In some embodiments, as in Figures 1 to 4 In the depicted embodiment, Figure 6 The depicted processing may be performed by a training subsystem to map probability values ​​corresponding to unresolvable classifications to augmented logit values ​​for one or more machine learning models (e.g., intent classifiers 242 or 320). The training subsystem may be a data processing system (e.g., Figure 4 The system may be part of the chatbot system 400 described herein, or a component of another system configured to train and deploy machine learning models.

[0164] At 605, the training subsystem receives a training dataset. The training dataset may include a set of utterances or messages. Each utterance in the set of utterances or messages is associated with a training label, wherein the training label identifies a predicted intent classification for the utterance. The training dataset may also include utterances or messages that are outside the scope of a specific task that the corresponding skill bot or chatbot is configured to perform. For example, when the skill bot is configured to process food orders for a pizza restaurant, an out-of-scope utterance may be a message asking about the weather. In some instances, each of the out-of-scope utterances or messages in the training dataset is associated with a training label, wherein the training label identifies the utterance or message as an unparseable classification.

[0165] At 610, the training subsystem trains a machine learning model using the training dataset to predict whether an utterance or message represents a task that the skill bot is configured to perform or to match an utterance or message with an intent associated with the skill bot. One or more parameters of the machine learning model can be learned to minimize a loss between a predicted output generated by the machine learning model and an expected output indicated by a training label for the corresponding utterance. Training of the machine learning model can be performed until the loss reaches a minimum error threshold.

[0166] At 615, the training subsystem identifies a set of training logit values. Each training logit value in the set of training logit values ​​corresponds to an out-of-range utterance in the training dataset. Each training logit value can be generated based on inputting the corresponding out-of-range utterance into the machine learning model. To obtain the training logit values, one or more helper functions (e.g., a get_layer function) can be utilized to capture output generated from one or more layers of the machine learning model.

[0167] At 620, the training subsystem determines a statistical value representing the set of training logit values. For example, the statistical value can be the median of the set of training logit values. The median can be used in response to determining that the statistical distribution of the set of training logit values ​​corresponds to a skewed distribution. In some examples, a mean and a median are determined for the set of training logit values. If the difference between the mean and the median is statistically significant, the median is selected as the statistical value representing the set of training logit values.

[0168] At 625, the training subsystem sets the statistical value to the augmented logit value for predicting whether the utterance corresponds to the unparseable category. The augmented logit value can replace the existing logit function of the machine learning model that was originally used to predict whether the utterance corresponds to the unparseable category. In some instances, the existing logit function continues to be used for prediction of utterances corresponding to one of the parsable categories (e.g., order_pizza, cancel_pizza).

[0169] At 630, the training subsystem deploys the trained machine learning model to a chatbot system (e.g., as part of a skill bot), wherein the trained machine learning model includes a statistic set as an augmented logit value for predicting whether an utterance corresponds to an unparseable classification. Thereafter, process 600 terminates.

[0170] By using statistics and removing the influence of the centroid from the unresolved category, the performance of the machine learning model in accurately predicting whether an utterance invokes the resolvable or unresolved category can be significantly improved. In the illustrative example, the pizza skill described above includes three intents: "order_pizza", "cancel_pizza", and "unresolvedIntent". The utterance "what is the weather tomorrow" has a weighting factor w based on the centroid. i (For example, logit i =F(x)*w i) computed logit values ​​(0.3, 0.01, 0.01) for the three intents. In this case, the centroid weighted relationship is used to determine the three values ​​of the logit vector, and the softmax function is used to convert the three values ​​of the logit vector into probabilities (0.4, 0.3, 0.3). In this example, "order_pizza" is the most likely outcome, with a confidence score of 40%. However, it should be clear that the query about the weather does not express the intent to order a pizza. The classification in this example is inappropriate (the correct classification would be out-of-domain unresolvedIntent), in part because the unresolved classification has a relatively low logit value of 0.01.

[0171] Continuing from this example, the enhanced logit value can be a statistic equal to 0.4. The statistic can be applied while maintaining the centroid-weighted logit value for the resolvable classification. Therefore, the logit values ​​can be redefined as (0.3, 0.01, 0.4). Applying the softmax function again, the logit values ​​are converted into probabilities (0.35, 0.26, 0.39). In this case, the result of the softmax function reflects an accurate classification of "unresolvedIntent" with a confidence score of 39%.

[0172] 2. Bounded Values

[0173] Determining augmented logit values ​​for unresolvable categories can be complicated by the unbounded nature of the standard logit function. As described above, the logit function outputs logit values ​​that range between negative infinity and positive infinity. Therefore, the unbounded range introduces complications when determining augmented logit values. For example, logit values ​​that range between negative infinity and positive infinity can result in logit values ​​that are scattered across a very wide range of values, which can make optimizing augmented logit values ​​difficult.

[0174] Thus, the enhanced logit value can be a bounded value within a range specified by the modified logit function. The modified logit function can be modified to constrain the logit value to a range that includes a lower limit and an upper limit. In some instances, the modified logit function includes a bounding function. The bounding function can include a trigonometric function that generates an output within a range between negative one and one, such as a sine function or a cosine function. For example, the logit function can be defined as logit i =cosine(f(x)*W i), where the minimum is -1 and the maximum is 1. An enhanced logit value can then be selected for the unresolved classification from the bounded range between -1 and 1. Continuing from this example, the logit function for the order_pizza classification and the cancel_pizza classification can use the logit i = cosine(f(x)*W i ), and a bounded value of 0.4 can be selected for the unresolvedIntent classification.

[0175] Figure 7 A flowchart illustrating an example process 700 for modifying a logit function to determine an enhanced logit value for predicting whether an utterance corresponds to an unresolved classification within a specified range, in accordance with some embodiments, is shown. Figure 7 The depicted processes can be implemented in software (e.g., code, instructions, programs) executed by one or more processing units (e.g., processors, cores) of the respective systems, hardware, or combinations thereof. The software can be stored on a non-transitory storage medium (e.g., on a memory device). Figure 7 The methods presented and described below are intended to be illustrative and not limiting. Although Figure 7 Various processing steps are depicted as occurring in a particular sequence or order, this is not intended to be limiting. In some embodiments, these steps are performed by a training subsystem to map probability values corresponding to unresolved classifications to enhanced logit values of one or more machine learning models (e.g., intent classifiers 242 or 320). The training subsystem can be part of a data processing system (e.g., with respect to the chatbot system 400 described below) or a component of another system configured to train and deploy machine learning models. Figure 4 The chatbot system 400 described below) or a component of another system configured to train and deploy machine learning models.

[0176] At 705, the training subsystem initializes a machine learning model. The machine learning model can be a convolutional neural network (“CNN”) (e.g., an initial neural network, a residual neural network (“Resnet”)); or a recurrent neural network (e.g., a long short-term memory (“LSTM”) model or a gated recurrent unit (“GRU”) model); other variants of deep neural networks (“DNN”) (e.g., a multi-label n-ary DNN classifier or a multi-class DNN classifier for single-intent classification. The machine learning model 425 can also be any other suitable ML model trained for natural language processing, such as a Naive Bayes classifier, a linear classifier, a support vector machine, a bagging model such as a random forest model, a boosting model, a shallow neural network, or a combination of one or more of such techniques— e.g., a CNN-HMM or a MCNN (multi-scale convolutional neural network).

[0177] Initializing a machine learning model can include defining the number of layers, the type of each layer (e.g., fully connected neural network, convolutional neural network), and the type of activation function for each layer (e.g., sigmoid, ReLU, softmax).

[0178] At 710, the training subsystem retrieves the logit function from the final layer of the machine learning model. For example, the training system may select the final layer from a set of layers in a fully connected neural network. The logit function of the final layer may then be accessed for modification.

[0179] At 715, the training subsystem modifies the logit function by adding a bounding function. The bounding function can include a trigonometric function that generates an output between negative one and one, such as a sine function or a cosine function. In some examples, the bounding function is selected from one of the following formulas: (i) sin(f(x)); (ii) cos(f(x)); (iii) tan(f(x)). -1 (f(x)); (iv) and (v) Where f(x) is the logit function. Additionally or alternatively, the logit function may include the centroid W i , where i corresponds to the corresponding predicted class.

[0180] At 720, the training subsystem trains the machine learning model to generate enhanced logit values ​​based on the range specified by the modified logit function. Thus, the training subsystem can train the machine learning model to generate enhanced logit values ​​by identifying statistics from the training dataset that predict whether an utterance corresponds to an unparseable classification (e.g., an unparsed skill or an unparsed intent).

[0181] At 725, the training subsystem deploys the trained machine learning model to the chatbot system (e.g., as part of a skill bot), wherein the trained machine learning model includes the modified logit function. Thereafter, process 700 terminates.

[0182] 3. Weighted value

[0183] The enhanced logit value for predicting out-of-range utterances may be a weighted value generated by a modified logit function that is weighted by a scaling factor. In some instances, the scaling factor (also referred to herein as a “scaler”) may be expressed as a logit function for class “i”. i = scaler*cosine(f(x)*W iThe value of the scaling factor for an unresolvable class (e.g., unresolvedIntent) (e.g., 2) can be different from the value of the scaling factor for a resolvable class (e.g., order_pizza, cancel_pizza) (e.g., 1). Including the scaling factor in the logit function can widen the gap between the probability of the predicted intent and the probabilities of other intents, thereby allowing the predicted intent of the unresolvable class to be better distinguished from the other intents of the resolvable class.

[0184] Figure 8 Shown is a flowchart 800 illustrating an example process for adding a scaling factor to a logit function to determine an enhanced logit value for predicting whether an utterance corresponds to an unparseable classification, in accordance with some embodiments. Figure 8 The depicted processes may be implemented in software (e.g., code, instructions, programs) executed by one or more processing units (e.g., processors, cores) of the corresponding system, hardware, or a combination thereof. The software may be stored on a non-transitory storage medium (e.g., a memory device). Figure 8 The methods presented and described below are intended to be illustrative and not limiting. Figure 8 The various processing steps are depicted as occurring in a particular sequence or order, but this is not intended to be limiting. In some embodiments, these steps are performed by a training subsystem to map probability values ​​corresponding to unresolvable classifications to augmented logit values ​​for one or more machine learning models (e.g., intent classifier 242 or 320). The training subsystem can be a data processing system (e.g., Figure 4 The system may be part of the chatbot system 400 described herein, or a component of another system configured to train and deploy machine learning models.

[0185] At 805, the training subsystem initializes the machine learning model. The machine learning model can be a convolutional neural network ("CNN") (e.g., an initial neural network, a residual neural network ("Resnet")); or a recurrent neural network (e.g., a long short-term memory ("LSTM") model or a gated recurrent unit ("GRU") model); or other variants of a deep neural network ("DNN") (e.g., a multi-label n-binary DNN classifier or a multi-class DNN classifier for single intent classification). The machine learning model 425 can also be any other suitable ML model trained for natural language processing, such as a naive Bayes classifier, a linear classifier, a support vector machine, a bagging model such as a random forest model, a boosting model, a shallow neural network, or a combination of one or more of these techniques - for example, a CNN-HMM or an MCNN (multi-scale convolutional neural network).

[0186] Initializing a machine learning model can include defining the number of layers, the type of each layer (e.g., fully connected neural network, convolutional neural network), and the type of activation function for each layer (e.g., sigmoid, ReLU, softmax).

[0187] At 810, the training subsystem retrieves the logit function from the final layer of the machine learning model. For example, the training system may select the final layer from a set of layers in a fully connected neural network. The logit function of the final layer may then be accessed for modification.

[0188] At 815, the training subsystem modifies the logit function by adding a scaling factor. In some instances, different initial values ​​of the scaling factors are assigned based on the predicted classification of the corresponding utterance. For example, the value of the scaling factor for an unresolvable classification (e.g., unresolvedIntent) (e.g., 2) may be different from the value of the scaling factor for a resolvable classification (e.g., order_pizza, cancel_pizza) (e.g., 1). A first scaling factor of a logit function that generates a probability indicating whether an utterance corresponds to an unresolvable classification is assigned a larger value than a second scaling factor of another logit function that generates a probability indicating whether an utterance corresponds to one of the resolvable classifications. Additionally or alternatively, the logit function may be further modified by adding a bounding function (e.g., a cosine function) to the logit function to which the scaling factor is added.

[0189] At 820, the training subsystem trains the machine learning model to generate augmented logit values ​​based on the scaling factor assigned to the unresolvable class. In some instances, the scaling factor for the unresolvable class is a learned parameter of the machine learning model. Thus, the training subsystem can train the machine learning model to adjust the scaling factor corresponding to the unresolvable class. Additionally or alternatively, the scaling factor can be a hard-coded number that can remain static during training and deployment of the machine learning model.

[0190] At 825, the training subsystem deploys the trained machine learning model to the chatbot system (e.g., as part of a skill bot), wherein the trained machine learning model includes the modified logit function. Thereafter, process 800 terminates.

[0191] 4. Hyperparameter Tuning

[0192] The enhanced logit value used to predict whether an utterance corresponds to an unparseable classification can be a hyperparameter optimized value determined based on hyperparameter tuning of the machine learning model. In some embodiments, hyperparameter tuning allows meta-training of the enhanced logit value as part of logit determination 455. Tunable hyperparameters of the machine learning model can include a learning rate, a number of epochs, a momentum, a regularization constant, a number of layers of the machine learning model, and a number of weights in the machine learning model. Hyperparameter tuning can include a process of determining an optimal combination of the above hyperparameters that allows the intent classifier to determine the enhanced logit value for the unparseable classification.

[0193] Figure 9 Shown is a flowchart 900 illustrating an example process for determining augmented logit values ​​for predicting whether an utterance corresponds to an unparseable classification using hyperparameter tuning in accordance with some embodiments. Figure 9 The depicted processes may be implemented in software (e.g., code, instructions, programs) executed by one or more processing units (e.g., processors, cores) of the corresponding system, hardware, or a combination thereof. The software may be stored on a non-transitory storage medium (e.g., a memory device). Figure 9 The methods presented and described below are intended to be illustrative and not limiting. Figure 9 The various processing steps are depicted as occurring in a particular sequence or order, but this is not intended to be limiting. In some embodiments, the steps may be performed in a different order or some steps may be performed in parallel. In some embodiments, as in Figures 1 to 4 In the depicted embodiment, Figure 9 The depicted processing may be performed by a training subsystem to map probability values ​​corresponding to unresolvable classifications to augmented logit values ​​for one or more machine learning models (e.g., intent classifiers 242 or 320). The training subsystem may be a data processing system (e.g., Figure 4 The intent classifier of the chatbot system 400 described above, or a component of another system configured to train and deploy machine learning models.

[0194] At 905, the training subsystem receives a training dataset. The training dataset may include a set of utterances or messages. Each utterance in the set of utterances or messages is associated with a training label, wherein the training label identifies a predicted intent classification for the utterance. The training dataset may also include utterances or messages that are outside the scope of a specific task that the corresponding skill robot or chatbot is configured to perform. For example, when the skill robot is configured to process food orders for a pizza restaurant, the out-of-scope utterance may be a message asking about the weather. In some instances, each of the out-of-scope utterances or messages in the training dataset is associated with a training label, wherein the training label identifies the utterance or message as an unparseable classification.

[0195] At 910, the training subsystem trains a machine learning model using the training dataset to predict whether an utterance or message represents a task that the skill bot is configured to perform or to match an utterance or message with an intent associated with the skill bot. One or more parameters of the machine learning model can be learned to minimize a loss between a predicted output generated by the machine learning model and an expected output indicated by a training label for the corresponding utterance. Training of the machine learning model can be performed until the loss reaches a minimum error threshold.

[0196] At 915, the training subsystem identifies a set of training logit values. Each training logit value in the set of training logit values ​​corresponds to an out-of-range utterance in the training dataset. Each training logit value can be generated based on inputting the corresponding out-of-range utterance into the machine learning model. To obtain the training logit values, one or more helper functions (e.g., a get_layer function) can be utilized to capture output generated from one or more layers of the machine learning model.

[0197] At 920, the training subsystem determines a statistical value representing the set of training logit values. For example, the statistical value can be the median of the set of training logit values. The median can be used in response to determining that the statistical distribution of the set of training logit values ​​corresponds to a skewed distribution. In some examples, a mean and a median are determined for the set of training logit values. If the difference between the mean and the median is statistically significant, the median is selected as the statistical value representing the set of training logit values.

[0198] At 925, the training subsystem adjusts one or more hyperparameters of the machine learning model to optimize the statistical value. Training of the machine learning model can be iterated using the adjusted hyperparameters, where the hyperparameter combination that provides the best performance for predicting the unparseable classification can be selected. In some embodiments, the best performance corresponds to a model configuration of the machine learning model that produces a minimum loss value between a predicted output generated by the machine learning model and an expected output indicated by a training label for the corresponding out-of-range utterance in the training dataset.

[0199] Various hyperparameter optimization techniques can be used to optimize statistical values. In some instances, random search techniques are used for hyperparameter tuning. Random search techniques can include generating a grid of possible hyperparameter values. Each search iteration selects a random combination of hyperparameters from this grid, and the performance of each iteration can be recorded. Additionally or alternatively, grid search techniques can be used for hyperparameter tuning. Grid search techniques can include generating a grid of possible hyperparameter values. Each search iteration selects a specific order of hyperparameter combinations. The performance of each iteration can be recorded. The hyperparameter combination that provides the best performance can be selected.

[0200] Other techniques for hyperparameter tuning can be considered, including but not limited to Bayesian optimization algorithms, tree-based Parzen estimators (TPEs), HyperBand, population-based training (PBT), and Bayesian optimization and HyperBand (BOHB).

[0201] At 930, the training subsystem sets the optimized statistical value as the enhanced logit value for the unresolvable classification. The enhanced logit value can replace the existing logit function of the machine learning model that was originally used to predict whether the utterance corresponds to the unresolvable classification. In some instances, the existing logit function continues to be used for the prediction of the utterance corresponding to one of the resolvable classifications (e.g., order_pizza, cancel_pizza). At 935, the training subsystem deploys the trained machine learning model to the chatbot system (e.g., as part of a skill bot), wherein the trained machine learning model includes the statistical value set as the enhanced logit value for predicting whether the utterance corresponds to the unresolvable classification. Thereafter, process 900 terminates.

[0202] 5. Learning Value

[0203] The augmented logit value used to predict whether an utterance corresponds to an unparseable classification can be a learned value that can be dynamically adjusted during the training of the machine learning model. The model parameters can be learned during the training phase of the machine learning model. In fact, when training with additional training data (e.g., out-of-range utterances), the augmented logit value can be optimized.

[0204] Various techniques may be performed to generate and adjust learned values. Figure 10 Shown is a flowchart 1000 illustrating an example process of using learned values ​​as augmented logit values ​​for predicting whether an utterance corresponds to an unparseable classification, in accordance with some embodiments. Figure 10The depicted processes may be implemented in software (e.g., code, instructions, programs) executed by one or more processing units (e.g., processors, cores) of the corresponding system, hardware, or a combination thereof. The software may be stored on a non-transitory storage medium (e.g., a memory device). Figure 10 The methods presented and described below are intended to be illustrative and not limiting. Figure 10 The various processing steps are depicted as occurring in a particular sequence or order, but this is not intended to be limiting. In some embodiments, these steps are performed by a training subsystem to map probability values ​​corresponding to unresolvable classifications to augmented logit values ​​for one or more machine learning models (e.g., intent classifier 242 or 320). The training subsystem can be a data processing system (e.g., Figure 4 The system may be part of the chatbot system 400 described herein, or a component of another system configured to train and deploy machine learning models.

[0205] At 1005, the training subsystem receives a training dataset. The training dataset may include a set of utterances or messages. Each utterance in the set of utterances or messages is associated with a training label, wherein the training label identifies a predicted intent classification for the utterance. The training dataset may also include utterances or messages that are outside the scope of a specific task that the corresponding skill robot or chatbot is configured to perform. For example, when the skill robot is configured to process food orders for a pizza restaurant, the out-of-scope utterance may be a message asking about the weather. In some instances, each of the out-of-scope utterances or messages in the training dataset is associated with a training label, wherein the training label identifies the utterance or message as an unparseable classification.

[0206] At 1010, the training subsystem performs batch balancing of the training dataset to generate an augmented training dataset that includes augmented copies of out-of-range utterances. This step may be optional based on the amount of out-of-range utterances available in the training dataset. For example, the training subsystem may generate replicas or augmented copies of training data corresponding to unresolvable categories (e.g., out-of-range utterances). Data augmentation techniques may include back-translating utterances in the training dataset, performing synonym replacement on one or more tokens of an utterance, randomly inserting tokens into an utterance, swapping two tokens of an utterance, and randomly deleting one or more tokens of an utterance.

[0207] At 1015, the training subsystem initializes the machine learning model. The machine learning model can be a convolutional neural network ("CNN") (e.g., an initial neural network, a residual neural network ("Resnet")); or a recurrent neural network (e.g., a long short-term memory ("LSTM") model or a gated recurrent unit ("GRU") model); or other variants of a deep neural network ("DNN") (e.g., a multi-label n-binary DNN classifier or a multi-class DNN classifier for single intent classification). The machine learning model 425 can also be any other suitable ML model trained for natural language processing, such as a naive Bayes classifier, a linear classifier, a support vector machine, a bagging model such as a random forest model, a boosting model, a shallow neural network, or a combination of one or more of these techniques - for example, a CNN-HMM or an MCNN (multi-scale convolutional neural network).

[0208] Initializing a machine learning model can include defining the number of layers, the type of each layer (e.g., fully connected neural network, convolutional neural network), and the type of activation function for each layer (e.g., sigmoid, ReLU, softmax).

[0209] At 1020, the training subsystem trains a machine learning model using the training dataset or the augmented training dataset to predict whether an utterance or message represents a task that the skill robot is configured to perform or to match an utterance or message to an intent associated with the skill robot. One or more parameters of the machine learning model can be learned to minimize a loss between a predicted output generated by the machine learning model and an expected output indicated by a training label for the corresponding utterance. Training of the machine learning model can be performed until the loss reaches a minimum error threshold. In particular, the machine learning model can be trained using the augmented training dataset so that the machine learning model can generate a probability of whether a given utterance corresponds to an unparseable intent.

[0210] In some instances, to reduce overfitting of a machine learning model due to a training dataset or an augmented training dataset, the training subsystem can use regularization to adjust one or more parameter weights. For example, regularization of model parameters can include Gaussian / L2 regularization, in which a small portion of the machine learning model weights is removed at each training iteration. Such an approach can improve the accuracy of the augmented logit values ​​with a relatively low increase in loss, thereby reducing overfitting.

[0211] In some instances, the performance of the trained machine learning model is evaluated by cross-validation. For example, the trained machine learning model can be evaluated using a k-fold cross-validation technique, wherein the k-fold cross-validation technique includes: (i) shuffling the training data (e.g., utterances) in the augmented training dataset; (ii) splitting the training dataset into k subsets; (iii) using each of the (k-1) subsets to train a corresponding machine learning model; (iv) using the reserved k1 subsets to evaluate the trained machine learning model; and (v) using different reserved k subsets to evaluate the trained machine learning model. i Repeat steps (iii) and (iv) for the subset.

[0212] At 1025, the training subsystem deploys the trained machine learning model to a chatbot system (e.g., as part of a skill bot) to generate an enhanced logit value for predicting whether an utterance or message corresponds to an unparseable classification. Because the machine learning model is trained using an augmented training dataset, the enhanced logit value can be used to accurately determine the probability value of an out-of-range utterance being associated with an unparseable classification. Thereafter, process 1000 terminates.

[0213] 6. Example of determining the value of augmented logit

[0214] As described above, machine learning techniques can be used to dynamically adjust the enhanced logit values ​​for unresolved classifications based on factors including, but not limited to, the size of the data or the number of intents. For example, the scaler can be varied by classification (e.g., order_pizza, unresolvedIntent), where weight decay is added to limit spurious optimization of the scaler value. In another example, the logit can be dynamically modified within predetermined boundaries. For unresolved intent classification, the logit values ​​can be learned as part of the machine learning model training, but the logit values ​​can be constrained to be within a range (e.g., from 0.0=>0.2).

[0215] In an illustrative example of a machine learning model that uses logit values ​​for unresolvable intent classification, a fixed logit value is applied to the unresolvable classification and the trained model is tested on two standard out-of-domain test sets. The data presented in Table 1 below show the improvement in out-of-domain classification when logit values ​​are applied without using a centroid weighting method. For example, applying a predetermined logit value improves the accuracy of the trained classifier in identifying out-of-domain utterances by up to 9% (compare the isMatch values ​​in rows 5 and 1) and improves recall across time periods by up to 8% (compare the e2e_ood_recall values ​​in rows 5 and 1).

[0216] Table 1: Out-of-domain testing of trained machine learning models

[0217]

[0218] Additionally, the experimental results do not reflect a corresponding or concurrent drop in performance for accurately classifying in-domain utterances, as shown in Table 2 below. For example, the in-domain test does not show a significant drop in accuracy between rows 3 and 5 (corresponding to applying a predetermined logit value of 0.5) compared to the cosine-bounded softmax function weighted by the centroid of other intent classes.

[0219] Table 2: In-domain testing of trained machine learning models

[0220]

[0221] E. Process for training a machine learning model that implements augmented logit values ​​for out-of-scope utterance classification

[0222] Figure 11 is a flow diagram illustrating a process 1100 for training a machine learning model that implements augmented logit values ​​for unparseable classifications, according to some embodiments. Figure 11 The depicted processes may be implemented in software (e.g., code, instructions, programs) executed by one or more processing units (e.g., processors, cores) of the corresponding system, hardware, or a combination thereof. The software may be stored on a non-transitory storage medium (e.g., a memory device). Figure 11 The methods presented and described below are intended to be illustrative and not limiting. Figure 11 The various processing steps are depicted as occurring in a particular sequence or order, but this is not intended to be limiting. In some embodiments, the steps may be performed in a different order or some steps may be performed in parallel. In some embodiments, as in Figures 1 to 4 In the depicted embodiment, Figure 11 The depicted processing may be performed by a training subsystem to map probability values ​​corresponding to unresolvable classifications to augmented logit values ​​of one or more machine learning models (e.g., intent classifiers 242 or 320, or machine learning model 425). The training subsystem may be a data processing system (e.g., Figure 4 The system may be part of the chatbot system 400 described herein, or a component of another system configured to train and deploy machine learning models.

[0223] At 1105, the training subsystem receives a training data set. The training data set may include a set of utterances or messages. Each utterance in the set of utterances or messages is associated with a training label, wherein the training label identifies a predicted intent classification for the utterance. In some instances, the utterances include text data converted from speech input (e.g., speech utterances), wherein the speech input can be converted into utterances in text form in the particular language and the text utterances can then be processed. The training data set may also include utterances or messages that are outside the scope of the specific task that the corresponding skill robot or chatbot is configured to perform. For example, when the skill robot is configured to process food orders for a pizza restaurant, the out-of-scope utterance may be a message asking about the weather. In some instances, each of the out-of-scope utterances or messages in the training data set is associated with a training label, wherein the training label identifies the utterance or message as an unparseable classification.

[0224] At 1110, the training subsystem initializes the machine learning model. The machine learning model may include a series of network layers, wherein a final network layer in the series of layers includes a logit function that transforms a first probability of a resolvable class into a first real number representing a first logit value and transforms a second probability of an unresolvable class into a second real number representing a second logit value. Thus, the logit function may be the logarithm of the probability of transforming the probability value of the corresponding class into a real number that fits into a probability distribution (e.g., a logit normal distribution). In some instances, the logit function is weighted by the centroid of the distribution associated with the corresponding class.

[0225] The machine learning model can be a convolutional neural network (“CNN”) (e.g., an inception neural network, a residual neural network (“Resnet”)); or a recurrent neural network (e.g., a long short-term memory (“LSTM”) model or a gated recurrent unit (“GRU”) model); or other variants of a deep neural network (“DNN”) (e.g., a multi-label n binary DNN classifier or a multi-class DNN classifier for single intent classification. The machine learning model 425 can also be any other suitable ML model trained for natural language processing, such as a naive Bayes classifier, a linear classifier, a support vector machine, a bagging model such as a random forest model, a boosting model, a shallow neural network, or a combination of one or more of such techniques - for example, a CNN-HMM or an MCNN (multi-scale convolutional neural network).

[0226] Initializing a machine learning model can include defining the number of layers, the type of each layer (e.g., fully connected neural network, convolutional neural network), and the type of activation function for each layer (e.g., sigmoid, ReLU, softmax).

[0227] At 1115, the training subsystem trains the machine learning model using the training dataset to predict whether an utterance or message represents a task that the skill bot is configured to perform or matches the utterance or message to an intent associated with the skill bot. In some instances, the machine learning model is trained to determine a first probability of a resolvable classification and a second probability of a non-resolvable classification. The first probability of a resolvable classification can be generated by a first output channel of a final network layer of the machine learning model. The second probability of a non-resolvable classification can be generated by a second output channel of the final network layer of the machine learning model. In some instances, the machine learning model determines a probability of each of two or more resolvable classifications (e.g., order_pizza, cancel_pizza).

[0228] One or more parameters of the machine learning model can be learned to minimize a loss between a predicted output generated by the machine learning model and an expected output indicated by a training label of a corresponding utterance. Training of the machine learning model can be performed until the loss reaches a minimum error threshold. In particular, the machine learning model can be trained using the augmented training dataset such that the machine learning model can generate a probability of whether a given utterance corresponds to a non-resolvable intent.

[0229] At 1120, the training subsystem replaces the logit function associated with the non-resolvable classification with an enhanced logit value. For the resolvable classifications, the logit function can continue to be used. Thus, the trained machine learning model can apply the logit function to the probabilities of the resolvable classifications to generate corresponding logit values.

[0230] For the non-resolvable classification, the logit function is replaced with an enhanced logit value. In some instances, the enhanced logit value is a different real number determined independently of the logit function used for the resolvable classifications. The enhanced logit value can include one of: (i) a statistical value determined based on a set of logit values generated from the training dataset, (ii) a bounded value selected from a range of values defined by a first logarithm of an odds corresponding to the probability of the non-resolvable classification, the first logarithm of the odds being constrained to the range of values by a clipping function and weighted by a centroid of a distribution associated with the non-resolvable classification, (iii) a weighted value generated by a second logarithm of the odds corresponding to the probability of the non-resolvable classification, the second logarithm of the odds being constrained to the range of values by a clipping function, scaled by a scaling factor, and weighted by the centroid of the distribution associated with the non-resolvable classification, (iv) a hyperparameter optimization value generated based on hyperparameter tuning of the machine learning model, or (v) a learning value adjusted during training of the machine learning model.

[0231] At 1125, the trained machine learning model with the augmented logit value can be deployed within a chatbot system (e.g., as part of a skill bot) to predict whether an utterance or message represents a task that the skill bot is configured to perform, match an utterance or message to an intent associated with the skill bot, or predict whether an utterance or message corresponds to an unparseable classification. Thereafter, process 1100 terminates.

[0232] F. Procedure for Classifying Out-of-Range Utterances Using Augmented Logit Values

[0233] Figure 12 is a flow diagram illustrating a process 1200 for classifying an utterance into an unparseable classification using augmented logit values, in accordance with some embodiments. Figure 12 The depicted processes may be implemented in software (e.g., code, instructions, programs) executed by one or more processing units (e.g., processors, cores) of the corresponding system, hardware, or a combination thereof. The software may be stored on a non-transitory storage medium (e.g., a memory device). Figure 12 The methods presented and described below are intended to be illustrative and not limiting. Figure 12 The various processing steps are depicted as occurring in a particular sequence or order, but this is not intended to be limiting. In some embodiments, the steps may be performed in a different order or some steps may be performed in parallel. In some embodiments, as in Figures 1 to 4 In the embodiment shown, Figure 12 The depicted process can be performed by a chatbot or skill bot system that can implement a machine learning model and augmented logit values ​​to predict whether an utterance corresponds to an unparseable classification. In this particular example process, the chatbot system performs the classification.

[0234] At 1205, the chatbot system receives an utterance generated by a user interacting with the chatbot system. An utterance can refer to a set of words (e.g., one or more sentences) exchanged during a conversation with the chatbot. In some instances, an utterance includes text data converted from voice input (e.g., a voice utterance), wherein the voice input can be converted into a textual form of the utterance in the particular language, and the textual utterance can then be processed.

[0235] At 1210, the chatbot system inputs the utterance into a machine learning model. The machine learning model can include a series of network layers, wherein a final network layer in the series of layers includes a logit function that transforms a first probability of a parsable class into a first real number representing a first logit value and transforms a second probability of an unparsable class into a second real number representing a second logit value.

[0236] The machine learning model may perform operations 1215, 1220, and 1225 to predict whether the utterance corresponds to an unparseable classification. At 1215, the machine learning model determines a first probability of a parsable classification and a second probability of an unparseable classification. The first probability of the parsable classification may be generated by a first output channel of a final network layer of the machine learning model. The second probability of the unparseable classification may be generated by a second output channel of the final network layer of the machine learning model. In some instances, the machine learning model determines a probability for each of two or more parsable classifications (e.g., order_pizza, cancel_pizza).

[0237] At 1220, the machine learning model maps a first probability of the resolvable class to a first logit value using a logit function. The logit function can be a logarithm of the odds corresponding to the first probability of the resolvable class. The first logarithm of the odds can be weighted by a centroid of a distribution associated with the resolvable class.

[0238] At 1225, the machine learning model maps a second probability of the unresolvable class to a second logit value, wherein the second logit value is determined independently of the logit function used for the resolvable class. In some instances, the second logit value comprises: (i) a statistical value determined based on a set of logit values ​​generated from a training dataset, (ii) a bounded value selected from a range of values ​​defined by a first logarithm of the probability corresponding to the probability of the unresolvable class, the first logarithm of the probability being constrained to the range of values ​​by a bounding function and weighted by the centroid of the distribution associated with the unresolvable class, (iii) a weighted value generated by a second logarithm of the probability corresponding to the probability of the unresolvable class, the second logarithm of the probability being constrained to the range of values ​​by a bounding function, scaled by a scaling factor, and weighted by the centroid of the distribution associated with the unresolvable class, (iv) a hyperparameter optimized value generated based on hyperparameter tuning of the machine learning model, or (v) a learned value adjusted during training of the machine learning model. The first logit value and the second logit value are provided back to the chatbot system for classification.

[0239] At 1230, the chatbot system classifies the utterance as a parsable class or an unparsable class based on the first logit value and the second logit value. In some instances, the activation function of the machine learning model (e.g., a softmax function) is applied to: (i) the first logit value to determine the probability that the utterance corresponds to the parsable class within the multinomial distribution; and (ii) the second logit value to determine the probability that the utterance corresponds to the unparsable class within the multinomial distribution. Based on the determined probabilities, the chatbot system can classify the utterance. Thereafter, process 1200 terminates.

[0240] G. Prescriptive Systems

[0241] Figure 13 Depicted is a simplified diagram of a distributed system 1300. In the illustrated example, distributed system 1300 includes one or more client computing devices 1302, 1304, 1306, and 1308 coupled to a server 1312 via one or more communication networks 1310. Client computing devices 1302, 1304, 1306, and 1308 may be configured to execute one or more applications.

[0242] In various examples, server 1312 can be adapted to run one or more services or software applications that implement one or more embodiments described in the present disclosure. In some examples, server 1312 can also provide other services or software applications that can include non-virtual environments and virtual environments. In some examples, these services can be provided to users of client computing devices 1302, 1304, 1306, and / or 1308 as web-based services or cloud services (such as under a software-as-a-service (SaaS) model). Users operating client computing devices 1302, 1304, 1306, and / or 1308 can then utilize one or more client applications to interact with server 1312 to utilize the services provided by these components.

[0243] exist Figure 13 In the depicted configuration, the server 1312 may include one or more components 1318, 1320, and 1322 that implement the functions performed by the server 1312. These components may include software components that may be executed by one or more processors, hardware components, or a combination thereof. It should be understood that a variety of different system configurations are possible that may differ from the distributed system 1300. Thus, Figure 13 The illustrated example is one example of a distributed system for implementing the example system and is not intended to be limiting.

[0244] A user may use client computing devices 1302, 1304, 1306, and / or 1308 to execute one or more applications, models, or chatbots that may generate one or more events or models that may then be implemented or served according to the teachings of the present disclosure. The client device may provide an interface that enables a user of the client device to interact with the client device. The client device may also output information to the user via the interface. Although Figure 13 Only four client computing devices are depicted, but any number of client computing devices can be supported.

[0245] Client devices can include various types of computing systems, such as portable handheld devices, general-purpose computers such as personal computers and laptops, workstation computers, wearable devices, gaming systems, thin clients, various messaging devices, sensors or other sensing devices, etc. These computing devices can run various types and versions of software applications and operating systems (e.g., Microsoft Apple or UNIX-like operating systems, Linux or Linux-like operating systems (such as Google Chrome TM OS), including various mobile operating systems (e.g., Microsoft Windows Windows Android TM 、 Portable handheld devices may include cellular phones, smart phones (e.g., ), tablet computers (e.g. ), personal digital assistants (PDAs), etc. Wearable devices may include Google Head-mounted displays and other devices. Gaming systems can include various handheld gaming devices, Internet-enabled gaming devices (e.g., with or without Microsoft Gesture Input Devices Game consoles, Sony system, by Various game systems provided by and others), etc. The client device may be capable of executing various different applications, such as various Internet-related applications, communication applications (e.g., email applications, short message service (SMS) applications), and may use various communication protocols.

[0246] The one or more networks 1310 may be any type of network familiar to those skilled in the art that can support data communications using any of a variety of available protocols, including but not limited to TCP / IP (Transmission Control Protocol / Internet Protocol), SNA (Systems Network Architecture), IPX (Internet Packet Exchange), By way of example only, the one or more networks 1310 can be a local area network (LAN), an Ethernet-based network, a token ring, a wide area network (WAN), the Internet, a virtual network, a virtual private network (VPN), an intranet, an extranet, a public switched telephone network (PSTN), an infrared network, a wireless network (e.g., according to the Institute of Electrical and Electronics Engineers (IEEE) 1002.11 protocol suite, and / or any other wireless protocol) and / or any combination of these networks and / or other networks.

[0247] The server 1312 may be composed of one or more general-purpose computers, dedicated server computers (including, by way of example, PC (personal computer) servers, Servers 1312 may include servers, mid-range servers, mainframe computers, rack servers, etc.), server farms, server clusters, or any other suitable arrangement and / or combination. Server 1312 may include one or more virtual machines running virtual operating systems or other computing architectures involving virtualization. For example, logical storage devices may be virtualized to maintain one or more flexible pools of virtual storage devices for the servers. In various examples, server 1312 may be adapted to run one or more services or software applications that provide the functionality described in the foregoing disclosure.

[0248] The computing systems in the servers 1312 may run one or more operating systems, including any of those discussed above and any commercially available server operating systems. The servers 1312 may also run any of a variety of additional server applications and / or middle-tier applications, including HTTP (Hypertext Transport Protocol) servers, FTP (File Transfer Protocol) servers, CGI (Common Gateway Interface) servers, Servers, database servers, etc. Exemplary database servers include but are not limited to (International Business Machines Corporation) and others.

[0249] In some implementations, server 1312 may include one or more applications to analyze and consolidate data feeds and / or event updates received from users of client computing devices 1302, 1304, 1306, and 1308. By way of example, data feeds and / or event updates may include, but are not limited to: feed, The server 1312 may also include one or more applications to display the data feeds and / or real-time events via one or more display devices of the client computing devices 1302 , 1304 , 1306 , and 1308 .

[0250] The distributed system 1300 may also include one or more data repositories 1314, 1316. In some examples, these data repositories may be used to store data and other information. For example, one or more of the data repositories 1314, 1316 may be used to store information (such as information about chatbot performance or generated models) for use by the chatbot used by the server 1312 when performing various functions according to various embodiments. The data repositories 1314, 1316 may reside in various locations. For example, the data repository used by the server 1312 may be local to the server 1312 or may be remote from the server 1312 and communicate with the server 1312 via a network-based or dedicated connection. The data repositories 1314, 1316 may be of different types. In some examples, the data repository used by the server 1312 may be a database, for example, such as an Oracle database. and relational databases such as databases provided by other suppliers. One or more of these databases can be adapted to respond to SQL format commands to implement storage, update and retrieval of data to and from the database.

[0251] In some examples, one or more of data repositories 1314, 1316 may also be used by applications to store application data. The data repositories used by applications may be of different types, such as a key-value store, an object store, or a general-purpose storage store backed by a file system.

[0252] In some examples, the functionality described in this disclosure can be provided as a service via a cloud environment. Figure 14 is a simplified block diagram of a cloud-based system environment in which various services may be provided as cloud services according to certain examples. Figure 14 In the depicted example, cloud infrastructure system 1402 can provide one or more cloud services that can be requested by users using one or more client computers 1404, 1406, and 1408. Cloud infrastructure system 1402 can comprise one or more computers and / or servers that can include those described above for server 1312. The computers in cloud infrastructure system 1402 can be organized as general-purpose computers, specialized server computers, server farms, server clusters, or any other appropriate arrangement and / or combination.

[0253] One or more networks 1410 can facilitate communications and exchange of data between clients 1404, 1406, and 1408 and cloud infrastructure system 1402. One or more networks 1410 can include one or more networks. Networks can be the same or different types. One or more networks 1410 can support one or more communication protocols, including wired and / or wireless protocols, to facilitate communications.

[0254] Figure 14 The depicted example is merely an example of cloud infrastructure system and is not intended to suggest any limitation as to the scope of use or functionality of the cloud infrastructure system. It is to be understood that the cloud infrastructure system 1402 can have more or fewer components than shown, can combine two or more components, or can have a different configuration or arrangement of components. For example, although shown as a single system, cloud infrastructure system 1402 can include two or more servers, computers, or systems that are functioning as a single server, computer, or system. For example, the Figure 14 The depicted example is merely an example of cloud infrastructure system and is not intended to suggest any limitation as to the scope of use or functionality of the cloud infrastructure system. It is to be understood that the cloud infrastructure system 1402 can have more or fewer components than shown, can combine two or more components, or can have a different configuration or arrangement of components. For example, although shown as a single system, cloud infrastructure system 1402 can include two or more servers, computers, or systems that are functioning as a single server, computer, or system. For example, the Figure 14 Although three client computing devices are depicted, in alternative examples any number of client computing devices can be supported.

[0255] The term cloud service is generally used to refer to a service that is made available to users by a service provider's system (e.g., cloud infrastructure system 1402) over the Internet or other communication network. Generally, in a public cloud environment, servers and systems that make up the cloud service provider's system are different from the customer's own on-premises servers and systems. The cloud service provider's system is managed by the cloud service provider. Customers do not manage or control the cloud service provider's system. Customers are provided with cloud services by the service provider's tens or hundreds of servers or systems that are geographically distributed. For example, customers can purchase cloud storage from an online storage provider and access the purchased storage, without the customer having to purchase a separate license, support, or hardware and software resources for the storage. For example, the cloud service provider's system can host an application, and users can order and use the application on-demand via the Internet, without the users having to purchase infrastructure resources for executing the application. Cloud services are designed to provide easy, scalable access to applications, resources and services. Multiple providers offer cloud services. For example, Oracle Provides a variety of cloud services such as middleware services, database services, Java cloud services and other services.

[0256] In some examples, cloud infrastructure system 1402 can provide one or more cloud services using different models, such as software as a service (SaaS), platform as a service (PaaS), infrastructure as a service (IaaS), and other models (including hybrid service models). Cloud infrastructure system 1402 can include a set of applications, middleware, databases, and other resources that enable the provision of various cloud services.

[0257] The SaaS model enables applications or software to be delivered to customers as a service over a communication network such as the Internet, without the customer having to purchase the hardware or software for the underlying application. For example, the SaaS model can be used to provide customers with access to on-demand applications hosted by a cloud infrastructure system 1402. Oracle Examples of SaaS services provided include, but are not limited to, various services for human resource / capital management, customer relationship management (CRM), enterprise resource planning (ERP), supply chain management (SCM), enterprise performance management (EPM), analytical services, social applications, etc.

[0258] The IaaS model is commonly used to provide infrastructure resources (e.g., servers, storage, hardware, and networking resources) as cloud services to customers to provide elastic computing and storage capabilities. Provides various IaaS services.

[0259] The PaaS model is typically used to provide platform and environment resources as a service that enable customers to develop, run, and manage applications and services without having to purchase, build, or maintain such resources. Examples of PaaS services provided include, but are not limited to, Oracle Java Cloud Service (JCS), Oracle Database Cloud Service (DBCS), Data Management Cloud Service, various application development solution services, etc.

[0260] Cloud services are typically provided on an on-demand, self-service basis, based on subscriptions, in an elastically scalable, reliable, highly available, and secure manner. For example, a customer can order one or more services provided by cloud infrastructure system 1402 via a subscription order. Cloud infrastructure system 1402 then performs processing to provide the services requested in the customer's subscription order. For example, a user can use speech to request that the cloud infrastructure system take an action (e.g., an intent) as described above and / or provide services to a chatbot system as described herein. Cloud infrastructure system 1402 can be configured to provide one or even multiple cloud services.

[0261] Cloud infrastructure system 1402 can provide cloud services via different deployment models. In a public cloud model, cloud infrastructure system 1402 can be owned by a third party cloud service provider and the cloud services are provided to any general public customer, where the customer can be an individual or an enterprise. In some other examples, under a private cloud model, cloud infrastructure system 1402 can operate within an organization (e.g., within an enterprise organization) and the services are provided to customers within the organization. For example, the customers can be various departments of an enterprise such as a human resources department, a payroll department, or even individuals within the enterprise. In some other examples, under a community cloud model, cloud infrastructure system 1402 and the services provided can be shared by multiple organizations within a related community. Various other models can also be used, such as a hybrid of the models mentioned above.

[0262] Client computing devices 1404, 1406, and 1408 can be different types of (e.g., as Figure 13 depicted in FIG. 13) and can be capable of operating one or more client applications. Users can use the client devices to interact with cloud infrastructure system 1402, such as to request services provided by cloud infrastructure system 1402. For example, a user can use a client device to request information or actions from a chatbot as described in the present disclosure.

[0263] In some examples, processing performed by cloud infrastructure system 1402 to provide services can involve model training and deployment. This analysis can involve using, analyzing, and manipulating datasets to train and deploy one or more models. The analysis can be performed by one or more processors, processing data possibly in parallel, performing simulations using the data, etc. For example, big data analysis can be performed by cloud infrastructure system 1402 for generating and training one or more models for a chatbot system. Data used for this analysis can include structured data (e.g., data stored in a database or structured according to a structured model) and / or unstructured data (e.g., data blocks (binary large objects)).

[0264] As depicted in the examples in Figure 14 Cloud infrastructure system 1402 can include infrastructure resources 1430 that are used to facilitate the provisioning of various cloud services provided by cloud infrastructure system 1402. Infrastructure resources 1430 can include, for example, processing resources, storage or memory resources, networking resources, etc. In some examples, storage virtual machines that can be used to service storage requests from applications can be part of cloud infrastructure system 1402. In other examples, the storage virtual machines can be part of a different system.

[0265] In certain examples, to facilitate efficient provisioning of these resources to support the various cloud services offered by cloud infrastructure system 1402 to different customers, resources can be bound into resource groups or resource modules (also referred to as "pods"). Each resource module or pod can include a pre-integrated and optimized combination of one or more types of resources. In certain examples, different pods can be provisioned in advance for different types of cloud services. For example, a first set of pods can be provisioned for database services, a second set of pods (which can include a different combination of resources than the pods in the first set of pods) can be provisioned for Java services, and so on. For some services, the resources allocated for provisioning the services can be shared between services.

[0266] Cloud infrastructure system 1402 itself can internally use services 1432 that are shared by different components of cloud infrastructure system 1402 and that facilitate the provisioning of services by cloud infrastructure system 1402. These internally shared services can include, but are not limited to, security and identity services, integration services, enterprise repository services, enterprise manager services, virus scanning and whitelist services, high availability, backup and recovery services, services for enabling cloud support, email services, notification services, file transfer services, and the like.

[0267] Cloud infrastructure system 1402 can include a plurality of subsystems. These subsystems can be implemented in software or hardware, or combination of software and hardware. As Figure 14 As depicted, the subsystems can include a user interface subsystem 1412 that enables users or customers of cloud infrastructure system 1402 to interact with cloud infrastructure system 1402. User interface subsystem 1412 can include a variety of different interfaces, such as a web interface 1414, an online store interface 1416 in which cloud services offered by cloud infrastructure system 1402 are advertised and consumers can purchase (e.g., subscribe to), and other interfaces 1418. For example, a customer can use a client device to request (service request 1434) one or more services offered by cloud infrastructure system 1402 using one or more of interfaces 1414, 1416, and 1418. For example, a customer can access an online store, browse cloud services offered by cloud infrastructure system 1402, and place a subscription order for one or more services offered by cloud infrastructure system 1402 that the customer wishes to subscribe to. The service request can include information identifying the customer and the one or more services that the customer desires to subscribe to. For example, a customer can place a subscription order for services offered by cloud infrastructure system 1402. As part of the order, the customer can provide information identifying a chatbot system for which the services are to be provided and optionally one or more credentials for the chatbot system.

[0268] In certain examples (such as Figure 14In the depicted example, cloud infrastructure system 1402 may include an order management subsystem (OMS) 1420 configured to process new orders. As part of this processing, OMS 1420 may be configured to: create an account for the customer (if one has not already been created); receive billing and / or billing information from the customer to be used to bill the customer for providing the requested services to the customer; verify the customer information; after verification, book the order for the customer; and orchestrate various workflows to prepare the order for fulfillment.

[0269] Once properly authenticated, the OMS 1420 may then invoke the Order Provisioning Subsystem (OPS) 1424, which is configured to provision resources for the order (including processing resources, memory resources, and networking resources). Provisioning may include allocating resources to the order and configuring the resources to facilitate the services requested by the customer order. The manner in which resources are provisioned for the order and the type of resources provisioned may depend on the type of cloud service the customer has ordered. For example, according to one workflow, the OPS 1424 may be configured to determine the specific cloud service being requested and identify the number of groups that may have been pre-configured for that specific cloud service. The number of groups allocated for the order may depend on the size / volume / tier / scope of the requested service. For example, the number of groups to be allocated may be determined based on the number of users supported by the service, the duration of the service being requested, etc. The allocated groups may then be customized for the specific requesting customer to provide the requested service.

[0270] In some examples, the setup phase processing described above can be performed as part of the provisioning process by cloud infrastructure system 1402. Cloud infrastructure system 1402 can generate an application ID and select a storage virtual machine for the application from storage virtual machines provided by cloud infrastructure system 1402 itself or from storage virtual machines provided by systems other than cloud infrastructure system 1402.

[0271] The cloud infrastructure system 1402 can send a response or notification 1444 to the requesting customer to indicate when the requested service is now ready for use. In some instances, information (e.g., a link) can be sent to the customer that enables the customer to begin using and taking advantage of the benefits of the requested service. In some examples, for the customer requesting the service, the response can include a chatbot system ID generated by the cloud infrastructure system 1402 and information identifying the chatbot system selected by the cloud infrastructure system 1402 for the chatbot system corresponding to the chatbot system ID.

[0272] Cloud infrastructure system 1402 can provide services to multiple customers. For each customer, cloud infrastructure system 1402 is responsible for managing information related to one or more subscription orders received from the customer, maintaining customer data associated with the order, and providing the requested services to the customer. Cloud infrastructure system 1402 can also collect usage statistics regarding the customer's use of the subscribed services. For example, statistics can be collected regarding the amount of storage used, the amount of data transferred, the number of users, and the amount of system uptime and downtime. This usage information can be used to bill the customer. Billing can be performed, for example, on a monthly basis.

[0273] Cloud infrastructure system 1402 can provide services to multiple customers concurrently. Cloud infrastructure system 1402 can store information specific to these customers (possibly including proprietary information). In some examples, cloud infrastructure system 1402 includes an identity management subsystem (IMS) 1428 configured to manage customer information and provide separation of managed information so that information related to one customer cannot be accessed by another customer. IMS 1428 can be configured to provide various security-related services, such as identity services, information access management, authentication and authorization services, and services for managing customer identities and roles and related functions.

[0274] Figure 15 An example of a computer system 1500 is shown. In some examples, the computer system 1500 can be used to implement any digital assistant or chatbot system in a distributed environment and the various servers and computer systems described above. Figure 15 As shown, computer system 1500 includes various subsystems, including a processing subsystem 1504 that communicates with a number of other subsystems via a bus subsystem 1502. These other subsystems may include a processing acceleration unit 1506, an I / O subsystem 1508, a storage subsystem 1518, and a communication subsystem 1524. The storage subsystem 1518 may include non-transitory computer-readable storage media, including storage media 1522 and system memory 1510.

[0275] Bus subsystem 1502 provides an mechanism for letting the various components and subsystems of computer system 1500 interact with each other as expected. While bus subsystem 1502 is illustrated as a single bus, alternative implementations of bus subsystem 1502 can utilize multiple buses. Bus subsystem 1502 can be any of various types of bus structures including a memory bus or memory controller, a peripheral bus, a local bus using any of a variety of bus architectures, etc. Such architectures can include Industry Standard Architecture (ISA) bus, Micro Channel Architecture (MCA) bus, Enhanced ISA (EISA) bus, Video Electronics Standards Association (VESA) local bus, and Peripheral Component Interconnect (PCI) bus (which can be implemented as a Mezzanine bus manufactured to the IEEE P1386.1 standard), etc.

[0276] Processing subsystem 1504 controls the operation of computer system 1500 and can include one or more processors, application specific integrated circuits (ASICs), or field programmable gate arrays (FPGAs). A processor can include a single core or multiple cores. Processing resources of computer system 1500 can be organized into one or more processing units 1532, 1534, etc. A processing unit can include one or more processors, one or more cores from the same or different processors, combinations of cores and processors, or other combinations of cores and processors. In some examples, processing subsystem 1504 can include one or more special purpose co-processors such as graphics processors, digital signal processors (DSPs), etc. In some examples, some or all of the processing units of processing subsystem 1504 can be implemented using custom circuitry such as application specific integrated circuits (ASICs) or field programmable gate arrays (FPGAs).

[0277] In some examples, processing units in processing subsystem 1504 can execute instructions stored in system memory 1510 or on computer readable storage media 1522. In various examples, processing units can execute various programs or code instructions and can maintain multiple simultaneously executing programs or processes. Some or all of the program code to be executed can reside on system memory 1510 and / or computer readable storage media 1522 (potentially including on one or more storage devices) at any given time. Processing subsystem 1504, through suitable programming, can provide various functionalities described above. In examples where computer system 1500 executes one or more virtual machines, one or more processing units can be assigned to each virtual machine.

[0278] In some examples, a processing acceleration unit 1506 may optionally be provided for performing custom processing or for offloading some of the processing performed by processing subsystem 1504 , thereby speeding up the overall processing performed by computer system 1500 .

[0279] The I / O subsystem 1508 may include devices and mechanisms for inputting information to the computer system 1500 and / or for outputting information from or via the computer system 1500. In general, use of the term input device is intended to include all possible types of devices and mechanisms for inputting information to the computer system 1500. User interface input devices may include, for example, a keyboard, a pointing device such as a mouse or trackball, a touchpad or touch screen incorporated into a display, a scroll wheel, a click wheel, a dial, buttons, switches, a keypad, an audio input device with a voice command recognition system, a microphone, and other types of input devices. User interface input devices may also include motion sensing and / or gesture recognition devices, such as Microsoft Windows®, which enable a user to control and interact with the input device. Motion sensor, Microsoft 360 game controllers, devices that provide an interface for receiving input using gestures and spoken commands. User interface input devices may also include eye gesture recognition devices, such as devices that detect eye activity from the user (e.g., a "wink" when taking a picture and / or making a menu selection) and translate eye gestures into input devices (e.g., Google Assistant). ) In addition, the user interface input device may include an input device that enables the user to communicate with the voice recognition system (e.g., Navigator) interactive voice recognition sensing device.

[0280] Other examples of user interface input devices include, but are not limited to, three-dimensional (3D) mice, joysticks or pointing rods, game controllers and graphic tablets, and audio / visual devices (such as speakers, digital cameras, digital video cameras, portable media players, webcams, image scanners, fingerprint scanners, barcode readers 3D scanners, 3D printers, laser rangefinders, and eye tracking devices). In addition, user interface input devices can include, for example, medical imaging input devices, such as computed tomography, magnetic resonance imaging, positron emission tomography, and medical ultrasound equipment. User interface input devices can also include, for example, audio input devices, such as MIDI keyboards, digital musical instruments, etc.

[0281] In general, the term output device is used to include all possible types of devices and mechanisms for outputting information from the computer system 1500 to a user or another computer. User interface output devices may include a display subsystem, indicator lights, or non-visual displays such as audio output devices. A display subsystem may be a cathode ray tube (CRT), a flat panel device (such as one using a liquid crystal display (LCD) or plasma display), a projection device, a touch screen, etc. For example, user interface output devices may include, but are not limited to, various display devices that visually convey text, graphics, and audio / video information, such as monitors, printers, speakers, headphones, car navigation systems, plotters, voice output devices, and modems.

[0282] The storage subsystem 1518 provides a repository or data storage for storing information and data used by the computer system 1500. The storage subsystem 1518 provides a tangible, non-transitory, computer-readable storage medium for storing basic programming and data constructs that provide some example functionality. The storage subsystem 1518 can store software (e.g., programs, code modules, instructions) that, when executed by the processing subsystem 1504, provide the functionality described above. The software can be executed by one or more processing units of the processing subsystem 1504. The storage subsystem 1518 can also provide authentication according to the teachings of the present disclosure.

[0283] The storage subsystem 1518 may include one or more non-transitory memory devices, including volatile memory devices and non-volatile memory devices. Figure 15 As shown, the storage subsystem 1518 includes system memory 1510 and computer-readable storage media 1522. The system memory 1510 may include multiple memories, including volatile main random access memory (RAM) for storing instructions and data during program execution and non-volatile read-only memory (ROM) or flash memory in which fixed instructions are stored. In some embodiments, the basic input / output system (BIOS), which contains basic routines such as those that help transfer information between elements within the computer system 1500 during startup, may typically be stored in ROM. RAM typically contains data and / or program modules currently being operated on and executed by the processing subsystem 1504. In some embodiments, the system memory 1510 may include a variety of different types of memory, such as static random access memory (SRAM), dynamic random access memory (DRAM), etc.

[0284] By way of example and not limitation, Figure 15As depicted, system memory 1510 may load executing applications 1512 (which may include various applications such as a web browser, a middle-tier application, a relational database management system (RDBMS), etc.), program data 1514, and an operating system 1516. By way of example, operating system 1516 may include various versions of Microsoft Apple and / or Linux operating systems, various commercially available or UNIX-like operating systems (including but not limited to various GNU / Linux operating systems, Google OS, etc.) and / or such as iOS, Telephone, OS, OS, OS operating system and other mobile operating systems, etc.

[0285] Computer-readable storage media 1522 can store programming and data structures that provide some example functionality. Computer-readable media 1522 can provide storage for computer-readable instructions, data structures, program modules, and other data for computer system 1500. Software (programs, code modules, instructions) that provide the functionality described above when executed by processing subsystem 1504 can be stored in storage subsystem 1518. By way of example, computer-readable storage media 1522 can include hard drives, magnetic disk drives, optical disk drives (e.g., CD ROMs, DVDs, Blu-ray discs, etc.). Computer readable storage media 1522 may include but is not limited to Drives, flash memory cards, universal serial bus (USB) flash memory drives, secure digital (SD) cards, DVD disks, digital video tapes, etc. The computer-readable storage medium 1522 may also include solid-state drives (SSDs) based on non-volatile memory (e.g., flash memory-based SSDs, enterprise-class flash memory drives, solid-state ROMs, etc.), SSDs based on volatile memory such as solid-state RAM, dynamic RAM, static RAM, etc., DRAM-based SSDs, magnetoresistive RAM (MRAM) SSDs, and hybrid SSDs using a combination of DRAM and flash memory-based SSDs.

[0286] In some examples, storage subsystem 1518 may also include a computer-readable storage medium reader 1520 that may be further connected to computer-readable storage medium 1522. Reader 1520 may receive data from a storage device such as a disk, flash memory drive, or the like and is configured to read data from the storage device.

[0287] In some examples, computer system 1500 can support virtualization technology, including but not limited to virtualization of processing and memory resources. For example, computer system 1500 can provide support for executing one or more virtual machines. In some examples, computer system 1500 can execute programs such as a hypervisor that facilitates the configuration and management of virtual machines. Each virtual machine can be allocated with memory, computing (e.g., processor, core), I / O and networking resources. Each virtual machine typically runs independently of other virtual machines. A virtual machine typically runs its own operating system, which can be the same as or different from the operating system executed by other virtual machines executed by computer system 1500. Therefore, multiple operating systems can potentially be run simultaneously by computer system 1500.

[0288] The communication subsystem 1524 provides an interface to other computer systems and networks. The communication subsystem 1524 is used as an interface for receiving data from other systems and transmitting data from the computer system 1500 to other systems. For example, the communication subsystem 1524 can enable the computer system 1500 to establish a communication channel to one or more client devices via the Internet for receiving information from the client devices and sending information to the client devices. For example, when the computer system 1500 is used to implement Figure 1 When the robot system 120 is depicted, the communication subsystem can be used to communicate with the chat robot system selected for the application.

[0289] The communication subsystem 1524 can support both wired communication protocols and / or wireless communication protocols. In some examples, the communication subsystem 1524 can include a radio frequency (RF) transceiver component for accessing a wireless voice and / or data network (e.g., using cellular phone technology, advanced data network technologies such as 3G, 4G, or EDGE (Enhanced Data Rates for Global Evolution), WiFi (IEEE 802.XX Family Standard), or other mobile communication technologies, or any combination thereof), a global positioning system (GPS) receiver component, and / or other components. In some examples, in addition to or in lieu of a wireless interface, the communication subsystem 1524 can provide wired network connectivity (e.g., Ethernet).

[0290] The communication subsystem 1524 can receive and transmit various forms of data. In some examples, the communication subsystem 1524 can receive input communications in the form of structured and / or unstructured data feeds 1526, event streams 1528, event updates 1530, etc., among other forms. For example, the communication subsystem 1524 can be configured to receive (or send) data feeds 1526 in real time from users of social media networks and / or other communication services, such as feed, Updates, web feeds (such as Rich Site Summary (RSS) feeds), and / or real-time updates from one or more third-party information sources.

[0291] In some examples, the communication subsystem 1524 can be configured to receive data in the form of a continuous data stream, which can include an event stream 1528 of real-time events (which may be continuous or unbounded in nature and may not have an explicit end) and / or event updates 1530. Examples of applications that generate continuous data can include, for example, sensor data applications, financial tickers, network performance measurement tools (e.g., network monitoring and traffic management applications), clickstream analysis tools, automobile traffic monitoring, etc.

[0292] The communication subsystem 1524 can also be configured to transmit data from the computer system 1500 to other computer systems or networks. The data can be transmitted in various forms, such as structured and / or unstructured data feeds 1526, event streams 1528, event updates 1530, etc., to one or more databases that can communicate with one or more streaming data sources coupled to the computer system 1500.

[0293] Computer system 1500 can be of various types, including a handheld portable device (e.g., Cellular phones, Computing tablets, PDAs), wearable devices (e.g., Google head-mounted display), personal computer, workstation, mainframe, kiosk, server rack, or any other data processing system. Due to the ever-changing nature of computers and networks, Figure 15 The description of the depicted computer system 1500 is intended to be a specific example only. Figure 15 Many other configurations of more or fewer components of the depicted systems are possible.Based on the disclosure and the teachings provided herein, it should be understood that there are other ways and / or methods to implement the various examples.

[0294] Although specific examples have been described, various modifications, changes, alternative constructions, and equivalents are possible. The examples are not limited to operations within certain specific data processing environments, but are free to operate within multiple data processing environments. In addition, although certain examples have been described using a specific series of transactions and steps, it should be apparent to those skilled in the art that this is not intended to be restrictive. Although some flowcharts describe operations as sequential processes, many operations can be performed in parallel or simultaneously. In addition, the order of operations can be rearranged. The process may have additional steps not included in the figure. The various features and aspects of the examples described above can be used individually or in combination.

[0295] Furthermore, although certain examples have been described using a particular combination of hardware and software, it should be appreciated that other combinations of hardware and software are possible. Certain examples may be implemented solely in hardware, solely in software, or using a combination thereof. The various processes described herein may be implemented in any combination on the same processor or on different processors.

[0296] Where a device, system, component or module is described as being configured to perform certain operations or functions, such configuration may be accomplished, for example, by designing electronic circuits to perform the operations, by programming programmable electronic circuits (such as microprocessors) to perform the operations (e.g., by executing computer instructions or code), or by a processor or core programmed to execute code or instructions stored on a non-transitory memory medium, or any combination thereof. Processes may communicate using a variety of techniques, including but not limited to conventional techniques for inter-process communication, and different pairs of processes may use different techniques, or the same pair of processes may use different techniques at different times.

[0297] Specific details are provided in this disclosure to provide a thorough understanding of the examples. However, the examples can be practiced without these specific details. For example, well-known circuits, processes, algorithms, structures, and techniques have been shown without unnecessary details to avoid obscuring the examples. This description provides only illustrative examples and is not intended to limit the scope, applicability, or configuration of other examples. Rather, the previous description of the examples will provide those skilled in the art with an enabling description for implementing the various examples. Various changes can be made to the functions and arrangements of the elements.

[0298] The present specification and drawings should, therefore, be viewed in an illustrative rather than a restrictive sense. However, it will be apparent that additions, subtractions, deletions, and other modifications and changes may be made thereto without departing from the broader spirit and scope set forth in the claims. Therefore, while specific examples have been described, these examples are not intended to be limiting. Various modifications and equivalents are within the scope of the appended claims.

[0299] In the foregoing description, various aspects of the present disclosure have been described with reference to specific examples of the present disclosure, but those skilled in the art will recognize that the present disclosure is not limited thereto. The various features and aspects of the disclosure described above may be used individually or in combination. Further, without departing from the broader spirit and scope of the description, the examples may be utilized in any number of environments and application environments beyond those described herein. Therefore, the description and drawings are to be considered illustrative rather than restrictive.

[0300] In the foregoing description, for the purpose of illustration, the method is described in a specific order. It should be understood that in an alternative example, the method can be performed in an order different from the described order. It should also be understood that the method described above can be performed by hardware components or can be embodied in a sequence of machine-executable instructions, which can be used to make a machine (for example, a general or special processor or logic circuit programmed with instructions) perform the method. These machine-executable instructions can be stored on one or more machine-readable media (for example, CD-ROM or other types of optical disks, floppy disks, ROM, RAM, EPROM, EEPROM, magnetic cards or optical cards, flash memories or other types of machine-readable media suitable for storing electronic instructions). Alternatively, the method can be performed by a combination of hardware and software.

[0301] Where a component is described as being configured to perform certain operations, such configuration may be accomplished, for example, by designing electronic circuits or other hardware to perform the operations, by programming programmable electronic circuits (e.g., a microprocessor or other suitable electronic circuits) to perform the operations, or any combination thereof.

[0302] While illustrative examples of the present application have been described in detail herein, it should be understood that the inventive concepts may be otherwise variously embodied and employed, and that the appended claims are intended to be interpreted to encompass such variations, except to the extent limited by the prior art.

Claims

1. A method for classifying utterances, comprising: receiving, by a chatbot system, an utterance generated by a user interacting with the chatbot system, wherein the utterance comprises text data converted from speech input of the user; Inputting, by the chatbot system, the utterance into a machine learning model comprising a series of network layers, wherein a final network layer in the series of network layers comprises a logit function that transforms a first probability of a resolvable class into a first real number representing a first logit value and transforms a second probability of an unresolvable class into a second real number representing a second logit value; determining, by the machine learning model, the first probability of the resolvable class and the second probability of the unresolvable class; mapping, by the machine learning model, the first probability of the resolvable class to a first logit value using the logit function, wherein the logit function for mapping the first probability is a logarithm of the odds corresponding to the first probability of the resolvable class, the logarithm of the odds weighted by a centroid of a distribution associated with the resolvable class; the second probability of the unresolvable classification being mapped to an augmented logit value by the machine learning model, wherein the augmented logit value is a third real number determined independently of the logit function used to map the first probability, wherein the augmented logit value comprises: (i) a statistical value determined based on a set of logit values ​​generated from a training data set; (ii) a bounded value selected from a range of values ​​defined by a first logarithm of the probability corresponding to the second probability of the unresolvable classification, the first logarithm of the probability being constrained to the range of values ​​by a bounding function and weighted by a centroid of a distribution associated with the unresolvable classification; (iii) a weighted value generated by a second logarithm of the probability corresponding to the second probability of the unresolvable classification, the second logarithm of the probability being constrained to the range of values ​​by the bounding function, scaled by a scaling factor, and weighted by the centroid of the distribution associated with the unresolvable classification; (iv) a hyperparameter optimized value generated based on hyperparameter tuning of the machine learning model; or (v) a learned value adjusted during training of the machine learning model; and The chatbot system classifies the utterance into the parsable category or the unparsable category based on the first logit value and the enhanced logit value.

2. The method of claim 1, further comprising: The chatbot system responds to the user based on classifying the utterance into the parsable classification or the unparsable classification.

3. The method according to claim 1, wherein The resolvable classifications are skills or intents that are within the domain and scope, and the unresolvable classifications are skills or intents that are outside the domain or scope.

4. A method as claimed in any preceding claim, wherein The enhanced logit value is the statistical value determined based on the set of logit values ​​generated from the training data set, and wherein determining the statistical value comprises: accessing a subset of the training dataset, wherein the subset of the training dataset comprises a subset of utterances, and wherein each utterance in the subset of utterances is associated with the unparseable classification; generating a set of training logit values, wherein each training logit value in the set of training logit values ​​is generated by applying the machine learning model to a corresponding utterance in the subset of utterances; determining the statistical value, wherein the statistical value represents the set of training logit values; and Set the statistic to the enhanced logit value.

5. The method according to claim 4, wherein: The statistic is the median of the set of training logit values.

6. The method of claim 4, wherein: The statistical value is the mean of the set of training logit values.

7. The method according to any one of claims 1 to 3, wherein The augmented logit value is the bounded value, and wherein the logit function is constrained to the range of values ​​by the bounding function.

8. The method according to any one of claims 1 to 3, wherein The enhanced logit value is the weighted value, and wherein the logit function is constrained to the range of values ​​by the bounding function and scaled by the scaling factor.

9. The method of claim 8, wherein: A first value is assigned to the scaling factor of the logit function and a second value is assigned to the scaling factor of a second logarithm of the odds corresponding to the second probability of the unresolvable classification, and wherein the second value is greater than the first value.

10. The method according to any one of claims 1 to 3, wherein The enhanced logit value is the hyperparameter optimized value, and wherein determining the hyperparameter optimized value comprises: accessing a subset of the training dataset, wherein the subset of the training dataset comprises a subset of utterances, and wherein each utterance in the subset of utterances is associated with the unparseable classification; generating a set of training logit values, wherein each training logit value in the set of training logit values ​​is generated by applying the machine learning model to a corresponding utterance in the subset of utterances; determining the statistical value, wherein the statistical value represents the set of training logit values; adjusting one or more hyperparameters of the machine learning model to generate optimized statistical values; and The optimized statistical value is set to the enhanced logit value.

11. A system for classifying utterances, comprising: one or more data processors; as well as A non-transitory computer-readable storage medium containing instructions that, when executed on the one or more data processors, cause the one or more data processors to perform operations comprising: receiving an utterance generated by a user interacting with a chatbot system, wherein the utterance comprises text data converted from speech input of the user; inputting the utterance into a machine learning model comprising a series of network layers, wherein a final network layer in the series of network layers comprises a logit function that transforms a first probability of a resolvable class into a first real number representing a first logit value and transforms a second probability of an unresolvable class into a second real number representing a second logit value; determining, by the machine learning model, the first probability of the resolvable class and the second probability of the unresolvable class; mapping, by the machine learning model, the first probability of the resolvable class to a first logit value using the logit function, wherein the logit function for mapping the first probability is a logarithm of the odds corresponding to the first probability of the resolvable class, the logarithm of the odds weighted by a centroid of a distribution associated with the resolvable class; the second probability of the unresolvable classification being mapped to an augmented logit value by the machine learning model, wherein the augmented logit value is a third real number determined independently of the logit function used to map the first probability, wherein the augmented logit value comprises: (i) a statistical value determined based on a set of logit values ​​generated from a training data set; (ii) a bounded value selected from a range of values ​​defined by a first logarithm of the probability corresponding to the second probability of the unresolvable classification, the first logarithm of the probability being constrained to the range of values ​​by a bounding function and weighted by a centroid of a distribution associated with the unresolvable classification; (iii) a weighted value generated by a second logarithm of the probability corresponding to the second probability of the unresolvable classification, the second logarithm of the probability being constrained to the range of values ​​by the bounding function, scaled by a scaling factor, and weighted by the centroid of the distribution associated with the unresolvable classification; (iv) a hyperparameter optimized value generated based on hyperparameter tuning of the machine learning model; or (v) a learned value adjusted during training of the machine learning model; and The utterance is classified into the parsable category or the unparsable category based on the first logit value and the enhanced logit value.

12. The system of claim 11, wherein: The instructions further cause the one or more data processors to perform operations comprising: The user is responded to based on classifying the utterance as either the parsable classification or the unparsable classification.

13. The system of claim 11, wherein: The resolvable classifications are skills or intents that are within the domain and scope, and the unresolvable classifications are skills or intents that are outside the domain or scope.

14. The system of any one of claims 11 to 13, wherein: The enhanced logit value is the statistical value determined based on the set of logit values ​​generated from the training data set, and wherein determining the statistical value comprises: accessing a subset of the training dataset, wherein the subset of the training dataset comprises a subset of utterances, and wherein each utterance in the subset of utterances is associated with the unparseable classification; generating a set of training logit values, wherein each training logit value in the set of training logit values ​​is generated by applying the machine learning model to a corresponding utterance in the subset of utterances; determining the statistical value, wherein the statistical value represents the set of training logit values; and Set the statistic to the enhanced logit value.

15. The system of claim 14, wherein: The statistic is the median of the set of training logit values.

16. The system of claim 14, wherein: The statistical value is the mean of the set of training logit values.

17. The system of any one of claims 11 to 13, wherein: The augmented logit value is the bounded value, and wherein the logit function is constrained to the range of values ​​by the bounding function.

18. The system of any one of claims 11 to 13, wherein: The enhanced logit value is the weighted value, and wherein the logit function is constrained to the range of values ​​by the bounding function and scaled by the scaling factor.

19. The system of claim 18, wherein: A first value is assigned to the scaling factor of the logit function and a second value is assigned to the scaling factor of a second logarithm of the odds corresponding to the second probability of the unresolvable classification, and wherein the second value is greater than the first value.

20. The system of any one of claims 11 to 13, wherein: The enhanced logit value is the hyperparameter optimized value, and wherein determining the hyperparameter optimized value comprises: accessing a subset of the training dataset, wherein the subset of the training dataset comprises a subset of utterances, and wherein each utterance in the subset of utterances is associated with the unparseable classification; generating a set of training logit values, wherein each training logit value in the set of training logit values ​​is generated by applying the machine learning model to a corresponding utterance in the subset of utterances; determining the statistical value, wherein the statistical value represents the set of training logit values; adjusting one or more hyperparameters of the machine learning model to generate optimized statistical values; and The optimized statistical value is set to the enhanced logit value.

21. A computer program product tangibly embodied in a non-transitory machine-readable storage medium, the computer program product comprising instructions configured to cause one or more data processors to perform operations comprising: receiving an utterance generated by a user interacting with the chatbot system, wherein The utterance comprises text data converted from voice input of the user; inputting the utterance into a machine learning model comprising a series of network layers, wherein a final network layer in the series of network layers comprises a logit function that transforms a first probability of a resolvable class into a first real number representing a first logit value and transforms a second probability of an unresolvable class into a second real number representing a second logit value; determining, by the machine learning model, the first probability of the resolvable class and the second probability of the unresolvable class; mapping, by the machine learning model, the first probability of the resolvable class to a first logit value using the logit function, wherein the logit function for mapping the first probability is a logarithm of the odds corresponding to the first probability of the resolvable class, the logarithm of the odds weighted by a centroid of a distribution associated with the resolvable class; the second probability of the unresolvable classification being mapped to an augmented logit value by the machine learning model, wherein the augmented logit value is a third real number determined independently of the logit function used to map the first probability, wherein the augmented logit value comprises: (i) a statistical value determined based on a set of logit values ​​generated from a training data set; (ii) a bounded value selected from a range of values ​​defined by a first logarithm of the probability corresponding to the second probability of the unresolvable classification, the first logarithm of the probability being constrained to the range of values ​​by a bounding function and weighted by a centroid of a distribution associated with the unresolvable classification; (iii) a weighted value generated by a second logarithm of the probability corresponding to the second probability of the unresolvable classification, the second logarithm of the probability being constrained to the range of values ​​by the bounding function, scaled by a scaling factor, and weighted by the centroid of the distribution associated with the unresolvable classification; (iv) a hyperparameter optimized value generated based on hyperparameter tuning of the machine learning model; or (v) a learned value adjusted during training of the machine learning model; and The utterance is classified into the parsable category or the unparsable category based on the first logit value and the enhanced logit value.

22. The computer program product of claim 21, wherein: The instructions further cause the one or more data processors to perform operations comprising: The user is responded to based on classifying the utterance as either the parsable classification or the unparsable classification.

23. The computer program product of claim 21, wherein: The resolvable classifications are skills or intents that are within the domain and scope, and the unresolvable classifications are skills or intents that are outside the domain or scope.

24. The computer program product of any one of claims 21 to 23, wherein: The enhanced logit value is the statistical value determined based on the set of logit values ​​generated from the training data set, and wherein determining the statistical value comprises: accessing a subset of the training dataset, wherein the subset of the training dataset comprises a subset of utterances, and wherein each utterance in the subset of utterances is associated with the unparseable classification; generating a set of training logit values, wherein each training logit value in the set of training logit values ​​is generated by applying the machine learning model to a corresponding utterance in the subset of utterances; determining the statistical value, wherein the statistical value represents the set of training logit values; and Set the statistic to the enhanced logit value.

25. The computer program product of claim 24, wherein: The statistic is the median of the set of training logit values.

26. The computer program product of claim 24, wherein: The statistical value is the mean of the set of training logit values.

27. The computer program product of any one of claims 21 to 23, wherein: The augmented logit value is the bounded value, and wherein the logit function is constrained to the range of values ​​by the bounding function.

28. The computer program product of any one of claims 21 to 23, wherein: The enhanced logit value is the weighted value, and wherein the logit function is constrained to the value range by the bounding function and scaled by the scaling factor.

29. The computer program product of claim 28, wherein: A first value is assigned to the scaling factor of the logit function and a second value is assigned to the scaling factor of a second logarithm of the odds corresponding to the second probability of the unresolvable classification, and wherein the second value is greater than the first value.

30. The computer program product of any one of claims 21 to 23, wherein: The enhanced logit value is the hyperparameter optimized value, and wherein determining the hyperparameter optimized value comprises: accessing a subset of the training dataset, wherein the subset of the training dataset comprises a subset of utterances, and wherein each utterance in the subset of utterances is associated with the unparseable classification; generating a set of training logit values, wherein each training logit value in the set of training logit values ​​is generated by applying the machine learning model to a corresponding utterance in the subset of utterances; determining the statistical value, wherein the statistical value represents the set of training logit values; adjusting one or more hyperparameters of the machine learning model to generate optimized statistical values; and The optimized statistical value is set to the enhanced logit value.

31. A model training method comprising: receiving, by a training subsystem, a training dataset, wherein the training dataset comprises a plurality of utterances generated by a user interacting with a chatbot system, wherein at least one utterance of the plurality of utterances comprises text data converted from speech input of the user; accessing, by the training subsystem, a machine learning model comprising a series of network layers, wherein a final network layer in the series of network layers comprises a logit function that transforms a first probability of a resolvable class into a first real number representing a first logit value and transforms a second probability of an unresolvable class into a second real number representing a second logit value; The training subsystem uses the training data set to train the machine learning model, so that the machine learning model: determining the first probability of the resolvable class and the second probability of the unresolvable class; and mapping the first probability of the resolvable class to a first logit value using the logit function, wherein the logit function for mapping the first probability is a logarithm of the odds corresponding to the first probability of the resolvable class, the logarithm of the odds weighted by a centroid of a distribution associated with the resolvable class; The training subsystem replaces the logit function with an enhanced logit value, so that the second probability of the unresolvable classification is mapped to the enhanced logit value, wherein: The enhanced logit value is a third real number determined independently of the logit function used to map the first probability; and The augmented logit value comprises: (i) a statistical value determined based on a set of logit values ​​generated from the training data set; (ii) a bounded value selected from a range of values ​​defined by a first logarithm of the probability corresponding to the second probability of the unresolvable classification, the first logarithm of the probability being constrained to the range of values ​​by a bounding function and weighted by the centroid of the distribution associated with the unresolvable classification; (iii) a weighted value generated by a second logarithm of the probability corresponding to the second probability of the unresolvable classification, the second logarithm of the probability being constrained to the range of values ​​by the bounding function, scaled by a scaling factor, and weighted by the centroid of the distribution associated with the unresolvable classification; (iv) a hyperparameter optimized value generated based on hyperparameter tuning of the machine learning model; or (v) a learned value adjusted during training of the machine learning model; and The trained machine learning model with the enhanced logit values ​​is deployed by the training subsystem.

32. The model training method of claim 31, further comprising: generating an augmented training dataset from the training dataset, wherein the augmented training dataset comprises transforming one or more transcripts of a particular utterance from the plurality of utterances, wherein the particular utterance is associated with a training label identifying the particular utterance as being associated with the unparseable classification; and The machine learning model is trained using the augmented training dataset.

33. The model training method according to claim 32, wherein: Transforming the one or more copies of the particular utterance includes performing one or more of the following: (i) back-translating the one or more copies of the particular utterance; (ii) performing synonym replacement on one or more tokens of the one or more copies of the particular utterance; (iii) randomly inserting tokens into the one or more copies of the particular utterance; (iv) swapping between two tokens of the one or more copies of the particular utterance; or (v) randomly deleting one or more tokens of the one or more copies of the particular utterance.

34. The model training method according to any one of claims 31 to 33, wherein: The enhanced logit value is the statistical value determined based on the set of logit values ​​generated from the training dataset, and wherein training the machine learning model further comprises: accessing a subset of the training dataset, wherein the subset of the training dataset comprises a subset of the plurality of utterances, and wherein each utterance in the subset of utterances is associated with the unparseable classification; generating a set of training logit values, wherein each training logit value in the set of training logit values ​​is generated by applying the machine learning model to a corresponding utterance from the subset of utterances; determining the statistical value, wherein the statistical value represents the set of training logit values; and Set the statistic to the enhanced logit value.

35. The model training method according to claim 34, wherein: The statistic is the median of the set of training logit values.

36. The model training method according to claim 34, wherein: The statistical value is the mean of the set of training logit values.

37. The model training method according to any one of claims 31 to 33, wherein: The augmented logit value is the bounded value, and wherein the logit function is constrained to the range of values ​​by the bounding function.

38. The model training method according to any one of claims 31 to 33, wherein: The enhanced logit value is the weighted value, and wherein the logit function is constrained to the range of values ​​by the bounding function and scaled by the scaling factor.

39. The model training method according to claim 38, wherein: A first value is assigned to the scaling factor of the logit function and a second value is assigned to the scaling factor of a second logarithm of the odds corresponding to the second probability of the unresolvable classification, and wherein the second value is greater than the first value.

40. The model training method according to claim 38, wherein: Training the machine learning model further includes adjusting the scaling factor for the unparseable classification.

41. The model training method according to any one of claims 31 to 33, wherein: The enhanced logit value is the hyperparameter optimized value, and wherein training the machine learning model further comprises: accessing a subset of the training dataset, wherein the subset of the training dataset comprises a subset of utterances, and wherein each utterance in the subset of utterances is associated with the unparseable classification; generating a set of training logit values, wherein each training logit value in the set of training logit values ​​is generated by applying the machine learning model to a corresponding utterance in the subset of utterances; determining the statistical value, wherein the statistical value represents the set of training logit values; adjusting one or more hyperparameters of the machine learning model to generate optimized statistical values; and The optimized statistical value is set to the enhanced logit value.

42. A model training system comprising: one or more data processors; as well as A non-transitory computer-readable storage medium containing instructions that, when executed on the one or more data processors, cause the one or more data processors to perform operations comprising: receiving a training dataset, wherein the training dataset comprises a plurality of utterances generated by a user interacting with a chatbot system, wherein at least one utterance of the plurality of utterances comprises text data converted from speech input of the user; accessing a machine learning model comprising a series of network layers, wherein a final network layer in the series of network layers comprises a logit function that transforms a first probability of a resolvable class into a first real number representing a first logit value and transforms a second probability of an unresolvable class into a second real number representing a second logit value; The machine learning model is trained using the training data set so that the machine learning model: determining the first probability of the resolvable class and the second probability of the unresolvable class; and mapping the first probability of the resolvable class to a first logit value using the logit function, wherein the logit function for mapping the first probability is a logarithm of the odds corresponding to the first probability of the resolvable class, the logarithm of the odds weighted by a centroid of a distribution associated with the resolvable class; The logit function is replaced by an enhanced logit value, so that the second probability of the unresolvable classification is mapped to the enhanced logit value, wherein: The enhanced logit value is a third real number determined independently of the logit function used to map the first probability; and The augmented logit value comprises: (i) a statistical value determined based on a set of logit values ​​generated from the training data set; (ii) a bounded value selected from a range of values ​​defined by a first logarithm of the probability corresponding to the second probability of the unresolvable classification, the first logarithm of the probability being constrained to the range of values ​​by a bounding function and weighted by the centroid of the distribution associated with the unresolvable classification; (iii) a weighted value generated by a second logarithm of the probability corresponding to the second probability of the unresolvable classification, the second logarithm of the probability being constrained to the range of values ​​by the bounding function, scaled by a scaling factor, and weighted by the centroid of the distribution associated with the unresolvable classification; (iv) a hyperparameter optimized value generated based on hyperparameter tuning of the machine learning model; or (v) a learned value adjusted during training of the machine learning model; and Deploying the trained machine learning model with the augmented logit values.

43. The model training system of claim 42, wherein: The instructions further cause the one or more data processors to perform operations comprising: generating an augmented training dataset from the training dataset, wherein the augmented training dataset comprises transforming one or more transcripts of a particular utterance from the plurality of utterances, wherein the particular utterance is associated with a training label identifying the particular utterance as being associated with the unparseable classification; and The machine learning model is trained using the augmented training dataset.

44. The model training system of claim 43, wherein: Transforming the one or more copies of the particular utterance includes performing one or more of the following: (i) back-translating the one or more copies of the particular utterance; (ii) performing synonym replacement on one or more tokens of the one or more copies of the particular utterance; (iii) randomly inserting tokens into the one or more copies of the particular utterance; (iv) swapping between two tokens of the one or more copies of the particular utterance; or (v) randomly deleting one or more tokens of the one or more copies of the particular utterance.

45. The model training system according to any one of claims 42 to 44, wherein: The enhanced logit value is the statistical value determined based on the set of logit values ​​generated from the training dataset, and wherein training the machine learning model further comprises: accessing a subset of the training dataset, wherein the subset of the training dataset comprises a subset of the plurality of utterances, and wherein each utterance in the subset of utterances is associated with the unparseable classification; generating a set of training logit values, wherein each training logit value in the set of training logit values ​​is generated by applying the machine learning model to a corresponding utterance from the subset of utterances; determining the statistical value, wherein the statistical value represents the set of training logit values; and Set the statistic to the enhanced logit value.

46. ​​The model training system of claim 45, wherein: The statistic is the median of the set of training logit values.

47. The model training system of claim 45, wherein: The statistical value is the mean of the set of training logit values.

48. The model training system according to any one of claims 42 to 44, wherein: The augmented logit value is the bounded value, and wherein the logit function is constrained to the range of values ​​by the bounding function.

49. The model training system according to any one of claims 42 to 44, wherein: The enhanced logit value is the weighted value, and wherein the logit function is constrained to the range of values ​​by the bounding function and scaled by the scaling factor.

50. The model training system of claim 49, wherein: A first value is assigned to the scaling factor of the logit function and a second value is assigned to the scaling factor of a second logarithm of the odds corresponding to the second probability of the unresolvable classification, and wherein the second value is greater than the first value.

51. The model training system of claim 49, wherein: Training the machine learning model further includes adjusting the scaling factor for the unparseable classification.

52. The model training system according to any one of claims 42 to 44, wherein: The enhanced logit value is the hyperparameter optimized value, and wherein training the machine learning model further comprises: accessing a subset of the training dataset, wherein the subset of the training dataset comprises a subset of utterances, and wherein each utterance in the subset of utterances is associated with the unparseable classification; generating a set of training logit values, wherein each training logit value in the set of training logit values ​​is generated by applying the machine learning model to a corresponding utterance in the subset of utterances; determining the statistical value, wherein the statistical value represents the set of training logit values; adjusting one or more hyperparameters of the machine learning model to generate optimized statistical values; and The optimized statistical value is set to the enhanced logit value.

53. A computer program product tangibly embodied in a non-transitory machine-readable storage medium, the computer program product comprising instructions configured to cause one or more data processors to perform operations comprising: Receive a training dataset, where The training dataset includes a plurality of utterances generated by a user interacting with a chatbot system, wherein at least one utterance of the plurality of utterances includes text data converted from speech input of the user; accessing a machine learning model comprising a series of network layers, wherein a final network layer in the series of network layers comprises a logit function that transforms a first probability of a resolvable class into a first real number representing a first logit value and transforms a second probability of an unresolvable class into a second real number representing a second logit value; The machine learning model is trained using the training data set so that the machine learning model: determining the first probability of the resolvable class and the second probability of the unresolvable class; and mapping the first probability of the resolvable class to a first logit value using the logit function, wherein the logit function for mapping the first probability is a logarithm of the odds corresponding to the first probability of the resolvable class, the logarithm of the odds weighted by a centroid of a distribution associated with the resolvable class; The logit function is replaced by an enhanced logit value, so that the second probability of the unresolvable classification is mapped to the enhanced logit value, wherein: The enhanced logit value is a third real number determined independently of the logit function used to map the first probability; and The augmented logit value comprises: (i) a statistical value determined based on a set of logit values ​​generated from the training data set; (ii) a bounded value selected from a range of values ​​defined by a first logarithm of the probability corresponding to the second probability of the unresolvable classification, the first logarithm of the probability being constrained to the range of values ​​by a bounding function and weighted by the centroid of the distribution associated with the unresolvable classification; (iii) a weighted value generated by a second logarithm of the probability corresponding to the second probability of the unresolvable classification, the second logarithm of the probability being constrained to the range of values ​​by the bounding function, scaled by a scaling factor, and weighted by the centroid of the distribution associated with the unresolvable classification; (iv) a hyperparameter optimized value generated based on hyperparameter tuning of the machine learning model; or (v) a learned value adjusted during training of the machine learning model; and Deploying the trained machine learning model with the augmented logit values.

54. The computer program product of claim 53, wherein: The instructions further cause the one or more data processors to perform operations comprising: generating an augmented training dataset from the training dataset, wherein the augmented training dataset comprises transforming one or more transcripts of a particular utterance from the plurality of utterances, wherein the particular utterance is associated with a training label identifying the particular utterance as being associated with the unparseable classification; and The machine learning model is trained using the augmented training dataset.

55. The computer program product of claim 54, wherein: Transforming the one or more copies of the particular utterance includes performing one or more of the following: (i) back-translating the one or more copies of the particular utterance; (ii) performing synonym replacement on one or more tokens of the one or more copies of the particular utterance; (iii) randomly inserting tokens into the one or more copies of the particular utterance; (iv) swapping between two tokens of the one or more copies of the particular utterance; or (v) randomly deleting one or more tokens of the one or more copies of the particular utterance.

56. A computer program product as claimed in any one of claims 53 to 55, wherein: The enhanced logit value is the statistical value determined based on the set of logit values ​​generated from the training dataset, and wherein training the machine learning model further comprises: accessing a subset of the training dataset, wherein the subset of the training dataset comprises a subset of the plurality of utterances, and wherein each utterance in the subset of utterances is associated with the unparseable classification; generating a set of training logit values, wherein each training logit value in the set of training logit values ​​is generated by applying the machine learning model to a corresponding utterance from the subset of utterances; determining the statistical value, wherein the statistical value represents the set of training logit values; and Set the statistic to the enhanced logit value.

57. The computer program product of claim 56, wherein: The statistic is the median of the set of training logit values.

58. The computer program product of claim 56, wherein: The statistical value is the mean of the set of training logit values.

59. The computer program product of any one of claims 53 to 55, wherein: The augmented logit value is the bounded value, and wherein the logit function is constrained to the range of values ​​by the bounding function.

60. The computer program product of any one of claims 53 to 55, wherein: The enhanced logit value is the weighted value, and wherein the logit function is constrained to the range of values ​​by the bounding function and scaled by the scaling factor.

61. The computer program product of claim 60, wherein: A first value is assigned to the scaling factor of the logit function and a second value is assigned to the scaling factor of a second logarithm of the odds corresponding to the second probability of the unresolvable classification, and wherein the second value is greater than the first value.

62. The computer program product of claim 60, wherein: Training the machine learning model further includes adjusting the scaling factor for the unparseable classification.

63. The computer program product of any one of claims 53 to 55, wherein: The enhanced logit value is the hyperparameter optimized value, and wherein training the machine learning model further comprises: accessing a subset of the training dataset, wherein the subset of the training dataset comprises a subset of utterances, and wherein each utterance in the subset of utterances is associated with the unparseable classification; generating a set of training logit values, wherein each training logit value in the set of training logit values ​​is generated by applying the machine learning model to a corresponding utterance in the subset of utterances; determining the statistical value, wherein the statistical value represents the set of training logit values; adjusting one or more hyperparameters of the machine learning model to generate optimized statistical values; and The optimized statistical value is set to the enhanced logit value.

Images