Enhanced logits for natural language processing

Abstract

To provide a chatbot system which can use a logit value enhanced for classifying speech and messages input to a chatbot system in natural language processing.SOLUTION: A chatbot system receives speech of a user, and inputs the speech into a machine learning model including a series of network layers. A final network layer of the series of the network layers. The machine learning model calculates a first probability for a class which can be solved and a second probability for a class which cannot be solved to use a logit function, maps the first probability for the class which can be solved to a first logit value and maps the second probability for the class which cannot be solved to an enhanced logit value, and, based on the first logit value and the enhanced logit value, classifies speech into the class which can be solved and the class which cannot be solved.SELECTED DRAWING: Figure 12

Description

[Technical Field]

[0001] Reference to related applications This application claims the interests of U.S. Provisional Patent Application 63 / 119,449 (filed November 30, 2020) and U.S. Non-Provisional Patent Application 17 / 456,687 (filed November 29, 2021). Each of the above-mentioned applications is incorporated herein by reference in its entirety for all purposes.

[0002] Technical field This disclosure relates in general to chatbot systems, and more specifically to techniques for using enhanced logit values ​​to classify utterances and messages input to chatbot systems in natural language processing. [Background technology]

[0003] background Many users worldwide are on instant messaging or chat platforms to get immediate responses. Organizations often use these instant messaging or chat platforms to engage in live conversations with customers (or end users). However, hiring service personnel to engage in live communication with customers or end users can be very expensive for organizations. Chatbots or bots began to be developed, especially on the internet, to simulate conversations with end users. End users can communicate with bots through messaging apps that they have already installed and are using. Intelligent bots, generally driven by artificial intelligence (AI), can communicate more intelligently and contextually in live conversations, and thus enable more natural conversations between bots and end users for an improved conversational experience. Instead of end users learning a fixed set of keywords or commands that the bot knows how to respond to, intelligent bots can understand the end user's intent based on the end user's natural language utterances and respond accordingly. [Overview of the project]

[0004] overview A technique is provided for using enhanced logit values ​​to classify utterances and messages input to a chatbot system in natural language processing. One method may involve the chatbot system receiving utterances generated by users interacting with the chatbot system and inputting those utterances into a machine learning model comprising a series of network layers. The utterances may include text data converted from speech input by the user. The final network layer of the series of network layers may include a logit function that converts a first probability for a resolvable class into a first real number representing a first logit value, and a second probability for an unresolvable class into a second real number representing a second logit value.

[0005] This method also includes a machine learning model that calculates a first probability for solvable classes and a second probability for unsolvable classes. The machine learning model can use a logit function to map the first probability for solvable classes to a first logit value. The logit function for mapping the first probability can be the logarithm of the odds corresponding to the first probability for solvable classes, and the logarithm of the odds is weighted by the centroid of the distribution associated with the solvable classes.

[0006] This method may also include the machine learning model mapping a second probability for unsolvable classes to an enhanced logit value. The enhanced logit value may be a third real number obtained independently of the logit function used to map the first probability. The enhanced logit value is selected from a range of values ​​defined by (i) a statistic obtained based on a set of logit values ​​generated from the training dataset, and (ii) the logarithm of the first odds corresponding to the second probability for unsolvable classes, where the logarithm of the first odds is bounded by a bounding function. (iii) a bounded value constrained to a certain range of values ​​and weighted by the centroid of the distribution associated with the unsolvable class, (iv) a weighted value generated by the logarithm of a second odds corresponding to a second probability for the unsolvable class, the logarithm of the second odds constrained to a certain range of values ​​by a bounding function, scaled by a scaling coefficient, and weighted by the centroid of the distribution associated with the unsolvable class, (iv) a hyperparameter-optimized value generated based on the hyperparameter tuning of the machine learning model, or (v) a learned value adjusted during the training of the machine learning model. The method may also include the chatbot system classifying utterances as either resolvable or unsolvable classes based on the first logit value and the enhanced logit value.

[0007] Techniques are also provided for training machine learning models that use enhanced logit values ​​to classify utterances and messages. One method may include a training subsystem receiving a training dataset. The training dataset may include multiple utterances generated by a user interacting with a chatbot system. At least one of the multiple utterances may include text data converted from the user's voice input. The method may also include the training subsystem accessing a machine learning model that includes a set of network layers. The final network layer of the set of network layers includes a logit function that converts a first probability for a resolvable class into a first real number representing a first logit value, and a second probability for an unresolvable class into a second real number representing a second logit value.

[0008] The method may also include a training subsystem that trains the machine learning model on a training dataset such that the machine learning model (i) calculates a first probability for solvable classes and a second probability for unsolvable classes, and (ii) uses a logit function to map the first probability for solvable classes to a first logit value. The logit function for mapping the first probability can be the logarithm of the odds corresponding to the first probability for solvable classes, and the logarithm of the odds is weighted by the centroid of the distribution associated with the solvable classes.

[0009] The method may also include replacing the logit function with an enhanced logit value so that the training subsystem maps the second probability for unsolvable classes to an enhanced logit value. The enhanced logit value may be a third real number obtained independently of the logit function used to map the first probability. The enhanced logit value is a bounded value selected from a range of values ​​defined by (i) a statistic obtained based on a set of logit values ​​generated from the training dataset, (ii) a bounded value defined by the logarithm of the first odds corresponding to the second probability for unsolvable classes, where the logarithm of the first odds is constrained to a range of values ​​by a bounding function and weighted by the centroid of the distribution associated with the unsolvable classes, and (iii) a second logarithm of the odds corresponding to the second probability for unsolvable classes, where the logarithm of the second odds is constrained to a range of values ​​by a bounding function, scaled by a scaling factor and weighted by the centroid of the distribution associated with the unsolvable classes. The logit values ​​may also include (iv) weighted values, hyperparameter-optimized values ​​generated based on hyperparameter tuning of the machine learning model, or (v) learned values ​​that are adjusted during training of the machine learning model. The method may also include the training subsystem deploying the trained machine learning model using the enhanced logit values.

[0010] In various embodiments, a system is provided that includes one or more data processors and a non-temporary computer-readable storage medium containing instructions that, when executed on one or more data processors, cause one or more data processors to perform some or all of the methods disclosed herein.

[0011] In various embodiments, a computer program product is provided which includes instructions that are tangibly embodied in a non-temporary machine-readable storage medium and configured to cause one or more data processors to perform some or all of the methods disclosed herein.

[0012] The technologies described above and below can be implemented in several ways and in several situations. Some exemplary implementations and situations are provided with reference to the following drawings, as will be described in more detail below. However, the following implementations and situations are only a small number of many.

Brief Description of the Drawings

[0013] [Figure 1] It is a simplified block diagram of a distributed environment incorporating an exemplary embodiment. [Figure 2] It is a simplified block diagram of a computing system that realizes a master bot according to an embodiment. [Figure 3] It is a simplified block diagram of a computing system that realizes a skill bot according to an embodiment. [Figure 4] It is a simplified block diagram of a chatbot training and deployment system according to various embodiments. [Figure 5] It shows a schematic diagram of an exemplary neural network according to some embodiments. [Figure 6] It shows a flowchart of an exemplary process for obtaining a statistical value representing a boosted logit value to predict whether an utterance corresponds to a class that cannot be resolved according to some embodiments. [Figure 7] It shows a flowchart of an exemplary process for modifying a logit function to obtain a boosted logit value within a specified range to predict whether an utterance corresponds to a class that cannot be resolved according to some embodiments. [Figure 8] It shows a flowchart of an exemplary process for adding a scaling factor to a logit function to obtain a boosted logit value to predict whether an utterance corresponds to a class that cannot be resolved according to some embodiments. [Figure 9]A flowchart showing an exemplary process for using hyperparameter tuning to obtain a boosted logit value for predicting whether to handle utterances for classes that cannot be handled, according to some embodiments. [Figure 10] A flowchart showing an exemplary process for using a learned value as a boosted logit value for predicting whether to handle utterances for classes that cannot be handled, according to some embodiments. [Figure 11] A flowchart showing a process for training a machine learning model that realizes a boosted logit value for classes that cannot be handled, according to some embodiments. [Figure 12] A flowchart showing a process for using a boosted logit value for classifying utterances as classes that cannot be handled, according to some embodiments. [Figure 13] A schematic diagram of a distributed system for implementing various embodiments. [Figure 14] A simplified block diagram of one or more components of a system environment in which services provided by one or more components of the system of the embodiments can be provided as cloud services, according to various embodiments. [Figure 15] A diagram showing an exemplary computer system that can be used to implement various embodiments.

Best Mode for Carrying Out the Invention

[0014] Detailed Description In the following description, certain details are included for illustrative purposes to facilitate a full understanding of a particular embodiment. However, it will be apparent that various embodiments can be implemented without these specific details. Figures and descriptions are not intended to be limiting. The term “exemplary” is used here to mean “provided as an example, case, or illustration.” Any embodiment or design described herein as “exemplary” should not necessarily be construed as being preferable or advantageous over other embodiments or designs.

[0015] A. Overview 1. Intent A digital assistant is an artificial intelligence-driven interface that helps users accomplish various tasks in natural language conversation. For each digital assistant, customers can assemble one or more skills. Skills (also referred to here as chatbots, bots, or skillbots) are individual bots that focus on specific types of tasks, such as tracking inventory, submitting time cards, and creating expense reports. When an end user interacts with a digital assistant, the digital assistant evaluates the end user's input and routes the conversation to the appropriate chatbot and from the appropriate chat. Digital assistants can be made available to end users through various channels such as Facebook Messenger, Skype Mobile Messenger, or Short Message Service (SMS). Channels allow chats to travel from the end user to the digital assistant and its various chatbots on various messaging platforms. Channels may also support user agent escalation, event-activated conversations, and testing.

[0016] Intents enable a chatbot to understand what a user wants the chatbot to do. An intent consists of a permutation of typical user requests and statements, also referred to as utterances (e.g., get account balance, make purchase). As used here, an utterance or message can refer to a set of words (e.g., one or more sentences) exchanged during a conversation with the chatbot. An utterance can be text converted from voice input entered by the user through the user interface. Intents may be created by providing a name that indicates some user action (e.g., order a pizza) and compiling a set of real-life user statements or utterances that are commonly associated with triggering that action. Since the chatbot's cognition is derived from these intents, each intent may be created from a robust dataset (1 to over 20 utterances) and may vary to allow the chatbot to interpret ambiguous user input. A rich set of utterances enables the chatbot to understand what a user wants when it receives messages that mean the same thing but are expressed differently, such as "ignore this order" or "cancel delivery!" Collectively, intents and the utterances belonging to them constitute a training corpus for chat. By training a model using this corpus, customers can use that model as a reference to resolve end-user input into a single intent. It can be converted into a tool. Customers can improve their chat cognitive agility through a cycle of intent testing and intent training.

[0017] However, building a chatbot that can determine an end-user's intent based on user utterances is a challenging task, partly due to the nuances and ambiguities of natural language, as well as the dimensions of the input space (e.g., possible user utterances) and the size of the output space (number of intents). Exemplary examples of this difficulty arise from the characteristics of natural language, such as employing euphemisms, synonyms, or ungrammatical language usage to express intents. For example, an utterance might express the intent to order a pizza without explicitly mentioning pizza, order, or delivery. For instance, in the everyday language of a particular region, "pizza" is called "pie." These tendencies in natural language, such as inaccuracy or variability, create uncertainty and introduce reliability as a parameter for predicting intent, for example, through keyword inclusion, as opposed to the explicit indication of intent. Therefore, it may be necessary to train, monitor, debug, and retrain chatbots to improve their performance and the user experience they provide. Conventional systems provide training systems for training and retraining machine learning models for digital assistants or chatbots in spoken language understanding (SLU) and natural language processing (NLP).

[0018] 2. Determine intent using machine learning models. In one or more respects, a chatbot system can provide utterances as input to a neural network model that uses a logistic regression function to map its outputs to a probability distribution. For classification, for example, ordering a set of outputs in a probability distribution allows for prediction of the intent invoked in the utterance. Accurate prediction, in turn, allows the chatbot to interact accurately with the end user. In this sense, accuracy depends at least in part on mapping the outputs of the neural network classifier to a probability distribution.

[0019] Logit values ​​are calculated based on the input to map the output of a neural network machine learning model to a probability distribution. Logit values, also called "logits," are values ​​output by the logit function of the network layer of the machine learning model. Logit values ​​can represent the odds that an utterance corresponds to a specific class. The logit function is the logarithm of the odds for a specific class (e.g., the order_pizza intent class, the unresolved class). Yes, the output of the machine learning model is converted into a corresponding logit value that fits within a probability distribution. The logit value is in the range of (-∞, +∞). The logit value can then be provided as input to an activation function (e.g., a softmax function) to generate a predicted likelihood of whether the input (e.g., an utterance) corresponds to a particular class in a set of classes. Here, the predicted likelihood lies within the probability distribution of the set of classes. The predicted likelihood can be in the range [0,1]. For example, corresponding to a particular class (e.g., order_pizza) Numerical output is generated by processing the input utterance (e.g., "I want to grab a pie") through one or more intermediate layers of a polynomial machine learning model. The output can be processed by a logit function of a specific class to produce a logit value of 9.4. An activation function can then be applied to the logit value to obtain a probability value in the range between 0 and 1 (e.g., 0.974), which indicates that the input utterance corresponds to the order_pizza class. This indicates that utterances that do not invoke any intents that the classifier is also trained to identify will correspond to out-of-scope or out-of-domain utterances. For example, the utterance "how is the weather today?" is an out-of-scope or out-of-domain utterance. This might be considered outside the scope of a classifier trained to predict whether an order is specified for an item.

[0020] Classification accuracy can be further improved by weighting one or more parameters of the logit function. For example, each intent class may be associated with a logit function which may be weighted by the centroid of the intent class. As used herein, the term “centroid” refers to the central location of a cluster used to classify an utterance, where the cluster identifies data corresponding to a particular end-user intent class. In some examples, the centroid is determined using data from the corresponding dataset of the utterance (e.g., a training dataset). Weighting the logit function by the centroid of the distribution allows the logit function to more accurately predict the classification of a given utterance, particularly when the utterance is within a domain or scope (e.g., an utterance trained for system recognition).

[0021] 3. Determining the intent of an out-of-scope utterance. Neural networks suffer from the problem of overconfidence. Overconfidence can occur because the confidence scores generated by a neural network trained on a class (e.g., a trained NLP algorithm) can be uncorrelated with the actual confidence scores. This problem typically occurs in deeper neural networks (i.e., neural network models with more layers). Deep neural network models are generally more accurate than shallower neural network models in predicting their outputs, but they can produce erroneous high-accuracy classification predictions if the actual input is not well represented by the training data used to train the neural network model. Therefore, while deep neural networks are desirable due to their high accuracy, the overconfidence problem associated with deep neural networks must be addressed to avoid performance issues with neural networks.

[0022] Traditional techniques involving the use of centroid-weighted logit functions do not effectively address the overconfidence problem. Generally speaking, for out-of-domain or out-of-scope utterances, the use of a centroid-weighted logit function can be said to treat the classifier as determining whether an utterance fits into a cluster of unresolved utterances. However, clusters of unresolved utterances can be inaccurate because out-of-domain or out-of-scope utterances are more likely to be dispersed than clustered due to their different natural language definitions and syntax. In other words, using a centroid-weighted logit function for out-of-scope utterances would mean applying a centroid based on a highly dispersed cluster. Applying such a centroid introduces inaccuracies into utterance classification, for example, by underrepresenting the probability that an intent is outside the chatbot's domain.

[0023] 4. Enhanced logits for classifying utterances as having unresolved intents. To overcome the shortcomings described above, this technology includes a system and method for accurately predicting whether out-of-scope utterances correspond to unresolved classes by using enhanced logits in a machine learning model. This can improve the accuracy of the utterance classifier, for example, by providing an improved utterance classifier that can more accurately identify out-of-domain or out-of-scope utterances that the classifier cannot reliably classify. This increased ability to identify out-of-domain / out-of-scope utterances can reduce the frequency with which the classifier attempts to classify utterances that it has not been trained to classify, and reduce the number of unpredictable or incorrect classification results that may result from attempting to classify utterances that are outside the classifier's capabilities, thus improving utterance classification as a whole. This technology may include a machine learning model (e.g., a neural network) that is trained to predict whether an utterance or message represents a resolvable class (e.g., the type of task that the skillbot is configured to perform, the intent associated with the skillbot) or an unresolved class. The machine learning model predicts whether an utterance corresponds to a class of resolvable intents. A logit function can be applied to generate a first logit value that predicts whether an utterance corresponds to a class of intents that cannot be resolved, and this can be configured to use an enhanced logit value that predicts whether an utterance corresponds to a class of intents that cannot be resolved. In some examples, the enhanced logit value replaces the logit function or is determined independently of it.

[0024] In some examples, the enhanced logit value for an unsolvable class may include one of the following: (i) a statistical value derived from a set of logit values ​​generated from the training dataset; (ii) a bounded value selected from a range of values ​​defined by the logarithm of a first odds corresponding to the probability for the unsolvable class, where the logarithm of the first odds is constrained to a range of values ​​by a bounding function and weighted by the centroid of the distribution associated with the unsolvable class; (iii) a weighted value generated by the logarithm of a second odds corresponding to the probability for the unsolvable class, where the logarithm of the second odds is scaled by a scaling coefficient, bounded by a bounding function, and weighted by the centroid of the distribution associated with the unsolvable class; (iv) a hyperparameter-optimized value generated based on hyperparameter tuning of the machine learning model; or (v) a learned value dynamically adjusted during training of the machine learning model.

[0025] B. Bots and Analysis Systems A bot (also known as a skill, chatbot, chatterbot, or talkbot) is a computer program that can engage in conversations with end users. Bots can generally respond to natural language messages (e.g., questions or comments) through messaging applications that use natural language messaging. Businesses may use one or more bot systems to communicate with end users through messaging applications. These messaging applications, sometimes called channels, can be the end user's preferred messaging application, one they have already installed and are familiar with. Therefore, end users do not need to download and install a new application to chat with the bot system. Examples of messaging applications include over-the-top (OTT) messaging channels (e.g., Facebook Messenger, Facebook WhatsApp, WeChat, Line, Kik, Telegram, Talk, Skype, Slack, or SMS), and virtual private assistants (e.g., Amazon Dot, Echo, or Show, Google® Home, Apple HomePod, etc.). ), mobile and web app extensions that extend native or hybrid / responsive mobile apps or web applications with chat functionality, or voice-based input (e.g., Siri, Cortana, Google Voice, or other voice input for dialogue) This may include devices or applications that have an interface to use.

[0026] In some examples, a bot system may be associated with a Unified Resource Identifier (URI). A URI can identify a bot system using a string. A URI can be used as a webhook for one or more messaging application systems. A URI can include, for example, a Unified Resource Location Specifier (URL) or a Unified Resource Name (URN). A bot system may be designed to receive messages (e.g., Hypertext Transfer Protocol (HTTP) post-call messages) from messaging application systems. HTTP post-call messages may be directed from messaging application systems to URIs. In some embodiments, messages may differ from HTTP post-call messages. For example, a bot system may receive messages from the Short Message Service (SMS). The discussion here may refer to communications that a bot system receives as messages, but it should be understood that messages can be HTTP post-call messages, SMS messages, or any other type of communication between two systems.

[0027] End users can interact with bot systems through conversational dialogue (sometimes called a conversational user interface (UI)), similar to how people interact with each other. In some cases, the dialogue might involve the end user saying "Hello" to the bot. The bot may respond with "Hi" and ask the end user how it can help. In some cases, the interaction may also be a transactional interaction with a banking bot, such as transferring money from one account to another; an informational interaction with an HR bot, such as checking a vacation balance; or an interaction with a retail bot, such as discussing returning a purchased item or seeking technical support.

[0028] In some embodiments, the bot system can intelligently handle end-user interactions without interaction with the bot system's administrator or developers. For example, an end-user may send one or more messages to the bot system to achieve a desired goal. The messages may contain some content, such as text, emojis, audio, images, video, or other means of conveying the message. In some embodiments, the bot system converts the content into a standardized format (e.g., a representational state transfer (REST) ​​call to an enterprise service with appropriate parameters) and translates it into natural language. The system can generate an answer. The bot system can also prompt the end user for additional input parameters or request other additional information. In some embodiments, the bot system may also initiate communication with the end user rather than passively responding to the end user utterance. Various techniques for identifying explicit invocations of bot systems and determining input to the bot system being invoked are described here. In some embodiments, explicit invocation analysis is performed by the master bot based on the detection of an invocation name in the utterance. In response to the detection of an invocation name, the utterance may be refined for input to the skill bot associated with the invocation name.

[0029] A conversation with a bot can follow a specific conversational flow that includes multiple states. The flow can define what happens next based on the input. In some embodiments, a bot system can be implemented using a state machine that includes user-defined states (e.g., end-user intents) and actions to be taken in or from a state. A conversation can take different paths based on end-user input, which can influence the decisions the bot makes about the flow. For example, in each state, based on end-user input or utterance, the bot can determine the end-user intent and decide on the appropriate next action. Here, and in the context of utterances, the term “intent” refers to the intent of the user who gave the utterance. For example, if a user intends to engage the bot in a conversation to order a pizza, the user’s intent might be expressed by the utterance “order a pizza.” A user’s intent can be directed towards a specific task that the user wants the chatbot to perform on their behalf. Thus, utterances can be expressed as questions, commands, requests, etc., that reflect the user’s intent. An intent can include goals that the end-user wishes to achieve.

[0030] In the context of chat configuration, the term "intent" here refers to configuration information used to map user utterances to specific tasks / actions or categories of tasks / actions that a chatbot can perform. To distinguish between utterance intents (i.e., user intents) and chatbot intents, the latter may be referred to here as "bot intents." A bot intent may include a set of one or more utterances associated with that intent. For example, an intent to order a pizza could have various permutations of utterances expressing the desire to order a pizza. These associated utterances can be used to train the chatbot's intent classifier, which then determines that the user input utterance is related to pizza. This allows determining whether an order intent matches. A bot intent can be associated with one or more dialog flows to initiate a conversation with a user in a certain state. For example, the first message for a pizza order intent might be the question, "What kind of pizza would you like?" In addition to the associated utterance, a bot intent may also include named entities related to that intent. For example, a pizza order intent might include variables or parameters used to perform the task of ordering a pizza, such as topping 1, topping 2, pizza type, pizza size, pizza quantity, etc. The values ​​of these entities are typically obtained through conversation with the user.

[0031] In one example, the utterance is analyzed to determine whether it contains a SkillBot invocation name. If no invocation name is found, the utterance is considered an implicit invocation, and the process proceeds to an intent classifier, such as a trained model. If a invocation name is found, the utterance is considered an explicit invocation, and the process proceeds to determine which part of the utterance is associated with the invocation name. If the trained model is invoked, the entire received utterance is provided as input to the intent classifier.

[0032] The intent classifier receiving the utterance may be the master bot's intent classifier (e.g., intent classifier 242 in Figure 2). The intent classifier may be a machine learning-based or rule-based classifier trained to determine whether the intent of the utterance matches a system intent (e.g., Exit, Help) or a specific skill bot. As described here, the intent analysis performed by the master bot may be limited to matching to a specific skill bot without determining which intent within that skill bot is the best match for the utterance. Thus, the intent classifier receiving the utterance may identify a specific skill bot to be invoked. Alternatively, if the utterance represents a specific system intent (e.g., the utterance contains the words "Exit" or "Help"), the intent classifier receiving the utterance may identify that specific system intent to trigger a conversation between the master bot and the user based on a dialogue flow configured for that specific system intent.

[0033] If a call name exists, one or more explicit call rules are applied to determine which part of the utterance is associated with the call name. This determination can be based on an analysis of the utterance's sentence structure, using POS tags, dependency information, and / or other extracted information received with the utterance. For example, a part associated with a call name could be a noun phrase containing the call name, or a prepositional object corresponding to the call name. Any part associated with a call name, as determined based on the processing, is removed. Other parts of the utterance that are not needed to convey the meaning of the utterance (e.g., prepositional words) may also be removed. Removing specific parts of the utterance generates input for the skillbot associated with the call name. If there are any parts remaining in the received utterance after removal, those remaining parts, for example, as a text string, form a new utterance for input to the skillbot. Otherwise, if the received utterance has been completely removed, the input may be an empty string. The skillbot associated with the call name is then invoked, and the generated input is provided to that skillbot.

[0034] Upon receiving the generated input, the invoked skillbot processes it by, for example, performing intent analysis using the trained skillbot's intent classifier to identify bot intents that match the user intents represented in the input. As a result of identifying matching bot intents, the skillbot then performs a specific action according to the dialogue flow associated with the matching bot intent. Alternatively, the SkillBot may initiate a conversation with the user. For example, if the input is an empty string, the conversation may begin in a default state defined for the dialog flow, such as a welcome message. Alternatively, if the input is not an empty string, the conversation may begin in some intermediate state, such as if the input contains a value about an entity or some other information that the SkillBot received as part of the input and therefore no longer needs to ask the user for. As another example, the SkillBot may determine that it cannot process the input (for example, because the confidence score of all bot intents configured for the SkillBot falls below a certain threshold). In this situation, the SkillBot may pass the input back to the master bot for processing (for example, intent analysis using the master bot's intent classifier), or the SkillBot may prompt the user for clarification.

[0035] 1. Overall Environment Figure 1 is a simplified block diagram of an environment 100 incorporating a chatbot system according to one embodiment. The environment 100 includes a Digital Assistant Builder Platform (DABP) 102, which enables users of the DABP 102 to create and deploy digital assistant or chatbot systems. The DABP 102 can be used to create one or more digital assistants (or DAs) or chatbot systems. For example, as shown in Figure 1, a user 104 representing a particular company can use the DABP 102 to create and deploy a digital assistant 106 for users of that particular company. For example, a bank can use the DABP 102 to create one or more digital assistants for use by its customers. Multiple companies can use the same DABP 102 platform to create digital assistants. As another example, the owner of a restaurant (e.g., a pizza shop) can use the DABP 102 to create and deploy a digital assistant that enables restaurant customers to order food (e.g., order pizza).

[0036] For the purposes of this disclosure, “digital assistant” is an entity that helps the user of the digital assistant accomplish various tasks through natural language conversation. A digital assistant may be implemented using only software (for example, a digital assistant is a digital entity implemented using a program, code, or instructions that can be executed by one or more processors), using hardware, or using a combination of hardware and software. A digital assistant may be embodied or implemented in various physical systems or devices, such as computers, mobile phones, watches, appliances, vehicles, etc. A digital assistant is also sometimes referred to as a chatbot system. Therefore, for the purposes of this disclosure, the terms digital assistant and chatbot system are interchangeable.

[0037] A digital assistant such as a digital assistant 106 built using DABP 102 can be used to perform various tasks through natural language-based conversations between the digital assistant and its user 108. As part of the conversation, the user may provide one or more user inputs 110 to the digital assistant 106 and receive responses 112 from the digital assistant 106. The conversation may include one or more of the inputs 110 and responses 112. Through these conversations, the user may request that one or more tasks be performed by the digital assistant 106, and in response, the digital assistant 106 is configured to perform the user-requested tasks and respond to the user with appropriate responses.

[0038] User input 110 is generally in natural language form and is called utterance. User utterance 110 can also be in text form, such as when the user types a sentence, a question, a piece of text, or even a single word and provides it to the digital assistant 106 as input. In one embodiment, user utterance 110 can be in the form of speech input or utterance, such as when the user says or speaks something provided as input to the digital assistant 106. The utterance is typically a language spoken by the user 108. For example, the utterance may be English or some other language. If the utterance is in speech form, the speech input is converted into a text-form utterance in that particular language, and the text utterance is then processed by the digital assistant 106. Various speech-to-text processing techniques may be used to convert the speech or auditory input into a text utterance, which is then processed by the digital assistant 106. In some embodiments, the speech-to-text conversion may be performed by the digital assistant 106 itself.

[0039] Utterances, which may be text or voice utterances, may include fragments, sentences, multiple sentences, one or more words, one or more questions, or combinations of the aforementioned types. The digital assistant 106 is configured to apply natural language understanding (NLU) techniques to utterances in order to understand the meaning of user input. As part of the NLU processing of an utterance, the digital assistant 106 is configured to perform processing to understand the meaning of the utterance, which involves identifying one or more intents and one or more entities corresponding to the utterance. Once the meaning of the utterance is understood, the digital assistant 106 can perform one or more actions or behaviors in response to the understood meaning or intent. For the purposes of this disclosure, we assume that utterances are either text utterances provided directly by the user 108 of the digital assistant 106, or the result of a conversion of input utterances into text format. However, this is not intended to be limiting or restrictive in any way.

[0040] For example, user 108's input may be a request to order a pizza by providing an utterance such as "I want to order a pizza." Upon receiving such an utterance, the digital assistant 106 is configured to understand the meaning of the utterance and take appropriate action. Appropriate action may include responding to the user with a question requesting user input, such as the type of pizza the user wants to order, the size of the pizza, and any toppings. The responses provided by the digital assistant 106 may also be in natural language form and may typically be in the same language as the input utterance. As part of generating these responses, the digital assistant 106 may perform natural language generation (NLG). In order for the user to order a pizza, the digital assistant may guide the user through a conversation with the user to provide all the necessary information to order the pizza, and then, at the end of the conversation, have the user order the pizza. The digital assistant 106 may terminate the conversation by outputting information to the user indicating that the pizza has been ordered.

[0041] At a conceptual level, the digital assistant 106 performs various processes in response to utterances received from the user. In some embodiments, this process involves a series of processing steps or a pipeline of processing steps, including, for example, understanding the meaning of the input utterance (sometimes called natural language understanding (NLU)), determining what action should be taken in response to the utterance, triggering the action to be taken where appropriate, generating a response to be output to the user in response to the user utterance, and outputting the response to the user. NLU processing may include parsing the received input utterance to understand its structure and meaning, and refining and reconstructing the utterance to develop a more understandable form (e.g., logical form) or structure for the utterance. Generating a response may include using NLG techniques.

[0042] NLU processing performed by digital assistants such as Digital Assistant 106 involves sentence analysis (e.g., tokenization, sorting, part-of-speech tag identification for sentences, etc.) This may include various NLP-related processes such as named entity recognition, generation of dependency trees to represent sentence structure, sentence splitting into clauses, analysis of individual clauses, resolution of anaphora, and chunking. In some embodiments, the NLU processing or a portion of it is performed by the digital assistant 106 itself. In some other embodiments, the digital assistant 106 can perform a portion of the NLU processing using other resources. For example, the syntax and structure of an input utterance may be identified by processing the sentence using parsing, part-of-speech tagging, and / or named entity recognition. In one implementation, for English, parsing, part-of-speech tagging, and named entity recognition, such as those provided by the Stanford Natural Language Processing (NLP) Group, are used to analyze sentence structure and syntax. These are provided as part of the Stanford CoreNLP toolkit.

[0043] The various examples provided in this disclosure demonstrate English utterances, but these are meant to be examples only. In some embodiments, the digital assistant 106 may also process utterances in languages ​​other than English. The digital assistant 106 may provide subsystems (e.g., components that implement NLU functionality) configured to perform processing for different languages. These subsystems may be implemented as pluggable units that can be invoked using service calls from the NLU core server. This makes the NLU processing flexible and extensible for each language, including allowing processing in different orders. Language packs may be provided for individual languages, and language packs may register a list of subsystems that can be serviced from the NLU core server.

[0044] Digital assistants such as the digital assistant 106 shown in Figure 1 can be made available or accessible to their user 108 through various different channels, including, but not limited to, a certain application, a social media platform, various messaging services and applications, and other applications or channels. A single digital assistant can have several channels configured for it, so that it can run on different services simultaneously and be accessed by different services simultaneously.

[0045] A digital assistant or chatbot system typically includes or is associated with one or more skills. In some embodiments, these skills are individual chatbots (referred to as skillbots) configured to interact with the user and fulfill specific types of tasks, such as tracking inventory, submitting time cards, creating expense reports, ordering food, verifying bank accounts, making reservations, and purchasing widgets. For example, in the embodiment shown in Figure 1, the digital assistant or chatbot system 106 includes skills 116-1, 116-2, etc. For the purposes of this disclosure, the term “skill” is used synonymously with the term “skillbot.”

[0046] Each skill associated with a digital assistant helps the user complete tasks through conversation with the user, and the conversation may include a combination of text or auditory input provided by the user and responses provided by the skill bot. These responses may take the form of text or auditory messages to the user, and / or use simple user interface elements (e.g., a selection list) that present the user with a choice to make.

[0047] There are various ways in which skills or skillbots can be associated with or added to a digital assistant. In one example, a skillbot may be developed by a company and then added to a digital assistant using DABP102. In another example, a skillbot may be developed and created using DABP102 and then added to a digital assistant created using DABP102. In yet another example, DABP102 may be... We provide an online digital store (called the "Skills Store") offering multiple skills aimed at a wide range of tasks. Skills offered through the Skills Store may also be published to various cloud services. To add skills to a digital assistant generated using DABP102, a DABP102 user can access the Skills Store via DABP102, select the desired skills, and indicate that the selected skills will be added to the digital assistant created using DABP102. Skills from the Skills Store can be added to the digital assistant as is or in a modified form (for example, a DABP102 user may select and clone a specific skill bot provided by the Skills Store, customize or modify the selected skill bot, and then add the modified skill bot to the digital assistant created using DABP102).

[0048] Various different architectures may be used to implement a digital assistant or chatbot system. For example, in one embodiment, the digital assistant created and deployed using DABP102 may be implemented using a master-bot / child (or sub)bot paradigm or architecture. According to this paradigm, the digital assistant is implemented as a master bot that interacts with one or more child bots, which are skill bots. For example, in the embodiment shown in Figure 1, the digital assistant 106 includes a master bot 114 and skill bots 116-1, 116-2, etc., which are child bots of the master bot 114. In one embodiment, the digital assistant 106 itself is considered to operate as the master bot.

[0049] A digital assistant implemented according to a master-sub-bot architecture allows users of the digital assistant to interact with multiple skills through an integrated user interface, i.e., through the master bot. When a user engages with the digital assistant, user input is received by the master bot. The master bot then performs processing to determine the meaning of the user input utterance. The master bot then determines whether the task requested by the user in the utterance can be handled by the master bot itself. If not, the master bot selects an appropriate skill bot to handle the user request and routes the conversation to the selected skill bot. This allows the user to converse with the digital assistant through a common, single interface while still providing the ability to use several skill bots configured to perform specific tasks. For example, in the case of a digital assistant developed for an enterprise, the digital assistant's master bot can interface with skill bots having specific functions, such as a CRM bot to perform functions related to customer relationship management (CRM), an ERP bot to perform functions related to enterprise resource planning (ERP), and an HCM bot to perform functions related to human capital management (HCM). Thus, end-users or consumers of digital assistants only need to know how to access the digital assistant through a common master bot interface, with multiple skill bots provided behind the scenes to handle user requests.

[0050] In one embodiment, in a master-bot / child-bot infrastructure, the master-bot is configured to recognize a list of available skill-bots. The master-bot may have access to various available skill-bots and, for each skill-bot, metadata that identifies the capabilities of each skill-bot, including tasks that each skill-bot can perform. Upon receiving a user request in the form of an utterance, the master-bot is configured to identify or predict from among the multiple available skill-bots which particular skill-bot is best suited to respond to or process the user request. The master-bot then routes the utterance (or a portion of the utterance) to that particular skill-bot for further processing. Thus, control is transferred from the master-bot to the skill-bot. The data flows to the bot. The master bot can support multiple input and output channels.

[0051] The embodiment shown in Figure 1 illustrates a digital assistant 106 comprising a master bot 114 and skill bots 116-1, 116-2, and 116-3, but this is not intended to be limiting. The digital assistant may include various other components (e.g., other systems and subsystems) that provide the functionality of the digital assistant. These systems and subsystems may be implemented using software only (e.g., code, instructions stored on a computer-readable medium and executable by one or more processors), hardware only, or a combination of software and hardware.

[0052] DABP102 provides infrastructure, services, and features that enable DABP102 users to create digital assistants that include one or more skill bots associated with the digital assistant. In some cases, skill bots can be created by cloning existing skill bots, for example, by cloning skill bots provided by the skill store. As mentioned above, DABP102 provides a skill store or skill catalog that offers multiple skill bots for performing various tasks. DABP102 users can clone skill bots from the skill store. They may modify or customize the cloned skill bots as needed. In some other cases, DABP102 users have created skill bots from scratch using the tools and services provided by DABP102. As mentioned above, the skill store or skill catalog provided by DABP102 may offer multiple skill bots for performing various tasks.

[0053] In one embodiment, creating or customizing a skill bot at a higher level involves the following steps: (1) Configure settings for the new skillbot (2) Set one or more intents for SkillBot (3) Set up one or more entities for one or more intents (4) Train SkillBot (5) Create a dialog flow for SkillBot (6) Add custom components to the skillbot as needed (7) Test and deploy the skillbot. The following is a brief explanation of each step.

[0054] (1) Configure settings for a new skillbot - various settings may be configured for the skillbot. For example, a skillbot designer can specify one or more invocation names for the skillbot being created. These invocation names can then be used by the digital assistant user to explicitly invoke the skillbot. For example, a user can explicitly invoke the corresponding skillbot by typing the invocation name into their utterance.

[0055] (2) Setting up one or more intents and associated exemplary utterances for the skillbot - The skillbot designer specifies one or more intents (also called bot intents) for the skillbot being created. The skillbot is then trained based on these specified intents. These intents represent categories or classes for which the skillbot is trained to infer about input utterances. Upon receiving an utterance, the trained skillbot infers the intent of the utterance. The inferred intent is selected from a predefined set of intents used to train the skillbot. The skillbot then takes the appropriate action in response to the utterance based on the inferred intent. In some cases, the intents for the skillbot represent tasks that the skillbot can perform for the user of the digital assistant. Each intent is given an intent identifier or intent name. For example, in the case of a skillbot trained for banking, the intents assigned to that skillbot might be "CheckBalance," "TransferMoney," "DepositCheck," etc. It may be included.

[0056] For each intent defined for a skillbot, the skillbot designer may also provide one or more exemplary utterances that represent that intent. These exemplary utterances are intended to represent utterances that a user may input to the skillbot for that intent. For example, for the balance inquiry intent, exemplary utterances may include: "What's my savings account balance?", "How much is in my checking account?", "How much money do I have in my account?", etc. This is also acceptable. Therefore, various permutations of typical user utterances may be specified as utterance examples for intents.

[0057] Intents and their associated exemplary utterances are used as training data to train a skillbot. Various different training techniques may be used. As a result of this training, a machine learning model is generated, which is configured to take utterances as input and output intents inferred about the utterances by the machine learning model. In some cases, the input utterances are provided to an intent analysis engine, which is configured to predict or infer intents for the input utterances using the trained model. The skillbot may then take one or more actions based on the inferred intents.

[0058] (3) Set up one or more entities for one or more intents - In some examples, additional context may be required to enable the skillbot to respond appropriately to user utterances. For example, there may be situations where user input utterances resolve to the same intent in the skillbot. For example, in the example above, both the utterances "What's my savings account balance?" and "How much is in my checking account?" resolve to the same balance While these utterances resolve to the same intent, they represent different requests that desire different things. To clarify such requests, one or more entities are added to the intent. Using the example of a banking skills bot, an entity called AccountType defines values ​​called "checking" and "saving," which are then added to the intent. It may be possible to allow Lubot to parse user requests and respond appropriately. In the example above, the utterances resolve to the same intent, but the values ​​associated with the AccountType entity are different. The two utterances are different. This allows the skillbot to perform different actions for two utterances, even though they resolve to the same intent. One or more entities may be specified for a particular intent set for the skillbot. Thus, entities are used to add context to the intent itself. Entities help to better describe the intent, enabling the skillbot to complete the user request.

[0059] In one embodiment, there are two types of entities, namely (a) embedded entities provided by DABP102, and (2) entities specified by the Skillbot designer. There are custom entities that can be added. Built-in entities are general-purpose entities that can be used with a wide variety of bots. Examples of built-in entities include, but are not limited to, entities related to time, date, address, number, email address, duration, cycle period, currency, phone number, URL, etc. Custom entities are used for more customized purposes. For example, for banking skills, the AccountType entity... The Entity may be defined by the SkillBot designer to enable various banking transactions by checking user input for keywords such as current, savings, and credit card.

[0060] (4) Training the SkillBot - The SkillBot is configured to receive user input in the form of utterances, parse or otherwise process the received input, and identify or select intents related to the received user input. As described above, the SkillBot must be trained for this purpose. In one embodiment, the SkillBot is trained on intents set for it and exemplary utterances associated with those intents (collectively, the training data), so that the SkillBot can resolve user input utterances to one of the SkillBot's set intents. In one embodiment, the SkillBot uses a machine learning model that is trained with the training data and enables the SkillBot to identify what the user is saying (or, in some cases, what the user is trying to say). DABP102 provides a variety of different training techniques that can be used by SkillBot designers to train SkillBots, including a variety of machine learning-based training techniques, rule-based training techniques, and / or combinations thereof. In one embodiment, a portion of the training data (e.g., 80%) is used to train the SkillBot model, and another portion (e.g., the remaining 20%) is used to test or validate the model. Once trained, the trained model (sometimes called a trained skillbot) can then be used to process and respond to user utterances. In some cases, a user utterance may be a question that requires only a single answer and does not require further conversation. To address such situations, a Q&A (question and answer) intent may be defined for the skillbot. This allows the skillbot to output a response to a user request without needing to update the dialogue definition. Q&A intents are generated in the same way as regular intents. The dialogue flow for Q&A intents may differ from the dialogue flow for regular intents.

[0061] (5) Create a dialogue flow for the skillbot - The dialogue flow specified for the skillbot describes how the skillbot responds when different intents to the skillbot are resolved in response to received user input. The dialogue flow defines the actions or behaviors that the skillbot takes, such as how the skillbot responds to user utterances, how the skillbot prompts the user for input, and how the skillbot returns data. The dialogue flow is like a flowchart that the skillbot follows. The skillbot designer specifies the dialogue flow using a language such as Markdown. In one embodiment, a version of YAML called OBotML can be used to specify the dialogue flow for the skillbot. The dialogue flow definition for the skillbot acts as a model of the conversation itself, allowing the skillbot designer to choreograph the interaction between the skillbot and the user it corresponds to.

[0062] In one embodiment, the SkillBot's dialog flow definition includes three sections: (a) Context section (b) Default transition section (c) State section.

[0063] Context Section - In the context section, skillbot designers can define variables used in the conversation flow. Other variables that may be designated in the context section include, but are not limited to, variables for error handling, variables for built-in or custom entities, and user variables that enable skillbot to recognize and persist user preferences.

[0064] Default Transition Section - Transitions for SkillBot can be defined in either a Dialogflow State section or a Default Transition section. Transitions defined in the Default Transition section act as fallbacks and are triggered when there are no applicable transitions defined within a state or when the conditions necessary to trigger a state transition are not met. The Default Transition section can be used to define routing that allows SkillBot to smoothly handle unexpected user actions.

[0065] The State Section – Dialog Flow and its associated behavior are defined as a set of transient states that govern the logic within the dialog flow. Each state node in the dialog flow definition designates a component that provides the functionality required for that point in the dialog. In this way, states are built around the components. Each state contains component-specific characteristics and defines transitions to other states that are triggered after the component has been executed.

[0066] Special case scenarios can be handled using the state section. For example, you might want to give a user the option to temporarily leave the first skill they are working on and do something in a second skill within the digital assistant. For instance, if a user is engaged in a conversation with a shopping skill (e.g., the user has made some selections for a purchase), they might want to jump to a banking skill (e.g., the user wants to make sure they have enough money for the purchase) and then return to the shopping skill to complete their order. To address this, an action in the first skill can be configured to initiate a conversation with a second, different skill within the same digital assistant, and then return to the original flow.

[0067] (6) Adding Custom Components to the SkillBot - As described above, the states specified in the dialog flow for the SkillBot designate the components that provide the necessary functionality corresponding to that state. The components enable the SkillBot to perform the functionality. In one embodiment, DABP102 provides a set of pre-configured components for performing a wide range of functions. The SkillBot designer can select one or more of these pre-configured components and associate them with states in the dialog flow for the SkillBot. The SkillBot designer can also create custom or new components using the tools provided by DABP102 and associate the custom components with one or more states in the dialog flow for the SkillBot.

[0068] (7) Test and deploy SkillBots - DABP102 provides several features that enable SkillBot designers to test SkillBots under development. The SkillBots can then be deployed and included in a digital assistant.

[0069] The above explanation describes how to create a skillbot, but similar techniques can also be used to create a digital assistant (or masterbot). At the masterbot or digital assistant level, you can set up built-in system intents for the digital assistant. These built-in system intents are used to identify common tasks that the digital assistant itself (i.e., the masterbot) can handle without calling the skillbot associated with the digital assistant. Examples of system intents defined for a masterbot include: (1) Exit: when the user wants to signal to the digital assistant that they wish to end the current conversation or context; (2) Help: when the user is asking for help or direction; (3) Unresolved Intent: for user input that does not well match the Exit intent or Help intent. The digital assistant also stores information about one or more skillbots associated with the digital assistant. This information allows the masterbot to select a specific skillbot to process an utterance.

[0070] At the master bot or digital assistant level, when a user inputs a phrase or utterance into the digital assistant, the digital assistant is configured to process how to route the utterance and the associated conversation. The digital assistant makes this determination using a routing model that can be rule-based, AI-based, or a combination of both. Using the routing model, the digital assistant determines whether the conversation corresponding to the user-input utterance should be routed to a specific skill for processing, processed by the digital assistant or master bot itself according to an embedded system intent, or processed as a different state in the current conversation flow.

[0071] In one embodiment, as part of this process, the digital assistant determines whether the user input utterance explicitly identifies a skillbot using its invocation name. If an invocation name is present in the user input, it is treated as an explicit invocation of the skillbot corresponding to the invocation name. In such a scenario, the digital assistant can route the user input to the explicitly invoked skillbot for further processing. If there is no specific or explicit invocation, in one embodiment, the digital assistant evaluates the received user input utterance and calculates confidence scores for the system intents and skillbots associated with the digital assistant. The scores calculated for a skillbot or system intent represent the likelihood that the user input represents a task or system intent configured for the skillbot to perform. System intents or skillbots whose associated calculated confidence scores exceed a threshold (e.g., a Confidence Threshold routing parameter) are selected as candidates for further evaluation. The digital assistant then selects a specific system intent or skillbot from the identified candidates for further processing of the user input utterance. In one embodiment, after one or more skill bots are identified as candidates, the intents associated with those candidate skills are evaluated (according to the intent model for each skill), and a confidence score is determined for each intent. Generally, intents with a confidence score exceeding a threshold (e.g., 70%) are treated as candidate intents. If a specific skill bot is selected, the user utterance is routed to that skill bot for further processing. If a system intent is selected, one or more actions are performed by the master bot itself according to the selected system intent.

[0072] 2. Components of the Master Bot System Figure 2 shows a simplified version of the MasterBot (MB) system 200 according to one embodiment. This is a lock diagram. The MB system 200 can be implemented using software only, hardware only, or a combination of hardware and software. The MB system 200 includes a preprocessing subsystem 210, a multiple intent subsystem (MIS) 220, an explicit call subsystem (EIS) 230, a skillbot call unit 240, and a data store 250. The MB system 200 shown in Figure 2 is merely an example of the configuration of components in a masterbot. Those skilled in the art will recognize many possible variations, substitutions, and modifications. For example, in some implementations, the MB system 200 may have more or fewer systems or components than those shown in Figure 2, may combine two or more subsystems, or may have subsystems with different configurations or arrangements.

[0073] The preprocessing subsystem 210 receives the utterance "A" 202 from the user and processes the utterance through the language detection unit 212 and the language parser 214. As described above, the utterance can be provided in various ways, including speech or text. The utterance 202 may be a fragment, a complete sentence, multiple sentences, etc. The utterance 202 may include punctuation. For example, if the utterance 202 is provided as speech, the preprocessing subsystem 210 may convert the speech to text using a speech-to-text converter (not shown) that inserts punctuation, such as commas, semicolons, periods, etc., into the resulting text.

[0074] The language detection unit 212 detects the language of the utterance 202 based on the text of the utterance 202. Since each language has its own grammar and semantics, the way in which the utterance 202 is processed depends on that language. Language differences are taken into consideration when analyzing the syntax and structure of the utterance.

[0075] The language parser 214 syntactically analyzes the utterance 202 and extracts part-of-speech (POS) tags for each individual linguistic unit (e.g., word) within the utterance 202. POS tags include, for example, nouns (NN), pronouns (PN), verbs (VB), etc. The language parser 214 may also tokenize the linguistic units of the utterance 202 (e.g., to convert each word into a separate token) and make the words headwords. Headwords are the primary forms of the set of words represented in the dictionary (e.g., "run" is a headword for run, runs, ran, running, etc.). Other types of preprocessing that the language parser 214 can perform include chunking compound expressions, for example, combining "credit" and "card" into a single expression "credit_card". 214 may also identify relationships between words within the utterance 202. For example, in some embodiments, the language parser 214 generates a dependency tree indicating which parts of the utterance (e.g., certain nouns) are direct objects, which parts are prepositions, etc. The results of the processing performed by the language parser 214 form extracted information 205, which is provided to the MIS 220 as input along with the utterance 202 itself.

[0076] As described above, utterance 202 may contain multiple sentences. For the purpose of detecting multiple intents and explicit calls, utterance 202 may be treated as a single unit, even if it contains multiple sentences. However, in some embodiments, preprocessing may be performed, for example by the preprocessing subsystem 210, to identify a single sentence among multiple sentences for multiple intent analysis and explicit call analysis. Generally, the results produced by MIS220 and EIS230 are substantially the same whether utterance 202 is processed at the level of individual sentences or as a single unit containing multiple sentences.

[0077] MIS220 determines whether utterance 202 represents multiple intents. While MIS220 can detect the presence of multiple intents in utterance 202, the processing performed by MIS220 does not involve determining whether the intents in utterance 202 match any intent configured for the bot. Instead, utterance 202 The process of determining whether the intent matches a bot intent can be performed by the intent classifier 242 of the MB system 200 or by the skill bot's intent classifier (for example, as shown in the embodiment of Figure 3). The process performed by MIS220 assumes that there is a bot (e.g., a specific skill bot or the master bot itself) that can process the utterance 202. Therefore, the process performed by MIS220 does not require knowledge of what bots are in the chatbot system (e.g., the identity of a skill bot registered with the master bot) or what intents are set for a particular bot.

[0078] To determine that utterance 202 contains multiple intents, MIS220 applies one or more rules from the set of rules 252 in datastore 250. The rules applied to utterance 202 are language-dependent of utterance 202 and may include sentence patterns indicating the presence of multiple intents. For example, a sentence pattern may include a conjunction connecting two parts of a sentence (e.g., a coordinating term), where both parts correspond to separate intents. If utterance 202 matches a sentence pattern, it can be inferred that utterance 202 represents multiple intents. Note that an utterance with multiple intents does not necessarily have different intents (e.g., intents directed to different bots, or different intents within the same bot). Instead, the utterance may have separate instances of the same intent, e.g., "order a pizza using payment account X, then order a pizza using payment account Y."

[0079] As part of determining that utterance 202 represents multiple intents, MIS220 also determines which parts of utterance 202 are associated with each intent. For each intent represented in an utterance containing multiple intents, MIS220 constructs a new utterance for alternative processing in place of the original utterance, e.g., utterance "B" 206 and utterance "C" 208, as shown in Figure 2. Thus, the original utterance 202 can be split into two or more separate utterances, each handled one at a time. Using the extracted information 205 and / or from an analysis of utterance 202 itself, MIS220 determines which of the two or more utterances should be processed first. For example, MIS220 may determine that utterance 202 contains a marker word indicating that a particular intent should be handled first. The newly formed utterance corresponding to this particular intent (e.g., one of utterances 206 or 208) will be sent first for further processing by EIS230. After the conversation triggered by the first utterance has ended (or been temporarily interrupted), the next highest priority utterance (e.g., utterance 206 or the other of utterance 208) may then be sent to the EIS230 for processing.

[0080] The EIS230 determines whether an received utterance (e.g., utterance 206 or utterance 208) contains a skillbot invocation name. In one embodiment, each skillbot in the chatbot system is assigned a unique invocation name that distinguishes it from other skillbots in the chatbot system. A list of invocation names can be maintained in the data store 250 as part of the skillbot information 254. When an utterance contains a word that matches an invocation name, the utterance is considered an explicit invocation. If the bot is not explicitly invoked, the utterance received by the EIS230 is considered an implicit invocation utterance 234 and is fed into the master bot's intent classifier (e.g., intent classifier 242) to determine which bot to use to process the utterance. In some examples, the intent classifier 242 determines that the master bot should process implicit invocation utterances. In other examples, the intent classifier 242 determines which skillbot to route the utterance to for processing.

[0081] The explicit invocation feature provided by EIS230 offers several advantages. It can reduce the amount of processing that the master bot must perform. For example, with explicit invocation, the master bot does not have to perform any intent classification analysis (e.g., using intent classifier 242), or it may have to perform reduced intent classification analysis to select a skill bot. Thus, explicit invocation analysis may enable the selection of a specific skill bot without relying on intent classification analysis.

[0082] Furthermore, there may be situations where functionality overlaps among multiple skill bots. This can occur, for example, when intents handled by two skill bots overlap or are very close to each other. In such situations, it can be difficult for the master bot to identify which of the multiple skill bots to select based solely on intent classification analysis. In such scenarios, explicit invocation resolves the ambiguity regarding which specific skill bot should be used.

[0083] In addition to determining that an utterance is an explicit invocation, the EIS230 is responsible for determining whether any part of the utterance should be used as input to an explicitly invoked skillbot. Specifically, the EIS230 can determine whether any part of the utterance is not associated with an invocation. The EIS230 can make this determination through analysis of the utterance and / or analysis of the extracted information 205. Instead of sending the entire utterance received by the EIS230, the EIS230 can send the portion of the utterance that is not associated with an invocation to the invoked skillbot. In some examples, the input to the invoked skillbot is formed simply by removing any portion of the utterance that is associated with an invocation. For example, "I want to order a pizza using Pizza Bot" It can be shortened to "I want to order a pizza" because "using PizzaBot" This is because it relates to the invocation of PizzaBot, but not to any processing performed by PizzaBot. In some cases, EIS230 may reformat the parts to be sent to the invoked bot, for example, to form a complete sentence. Thus, EIS230 determines not only whether there is an explicit invocation, but also what should be sent to the skill bot when there is an explicit invocation. In some cases, there may be no text to input to the invoked bot. For example, if the utterance was "Pizza Bot" In other words, EIS230 may determine that PizzaBot has been called, but there is no text to be processed by PizzaBot. In such a scenario, EIS230 may indicate to SkillBot Call Unit 240 that there is nothing to send.

[0084] The skillbot invocation unit 240 invokes skillbots in various ways. For example, the skillbot invocation unit 240 may invoke a bot in response to receiving instruction 235 that a particular skillbot has been selected as a result of an explicit invocation. Instruction 235 may be sent by the EIS 230 along with input to the explicitly invoked skillbot. In this scenario, the skillbot invocation unit 240 takes control of the conversation to the explicitly invoked skillbot. The explicitly invoked skillbot determines an appropriate response to the input from the EIS 230 by treating the input as an independent utterance. For example, the response may be to perform a specific action or to start a new conversation in a specific state, the initial state of the new conversation depending on the input sent from the EIS 230.

[0085] Another way in which the SkillBot Invocation Unit 240 can invoke a SkillBot is by implicit invocation using an Intent Classifier 242. The Intent Classifier 242 is trained using machine learning and / or rule-based training techniques to determine the likelihood that a given utterance represents a task configured for a particular SkillBot to perform. The Intent Classifier 242 has one classifier per SkillBot. The classes are trained in different classes. For example, each time a new skill bot is registered with a master bot, the intent classifier 242 can be trained using a list of exemplary utterances associated with that new skill bot to determine the likelihood that a particular utterance represents a task that the new skill bot can perform. The parameters generated as a result of this training (e.g., a set of values ​​for the parameters of a machine learning model) can be stored as part of the skill bot information 254.

[0086] In one embodiment, the intent classifier 242 is implemented using a machine learning model, as described in more detail here. Training the machine learning model may involve inputting at least a subset of utterances from exemplary utterances associated with various skill bots in order to generate inferences as the output of the machine learning model about which bot is the correct bot to process any particular training utterance. For each training utterance, instructions for the correct bot to use for that training utterance may be provided as ground truth information. The behavior of the machine learning model can then be adapted (e.g., through backpropagation) to minimize the difference between the generated inferences and the ground truth information.

[0087] In one embodiment, the intent classifier 242 determines a confidence score for each skill bot registered with the master bot, indicating the likelihood that the skill bot can process a given utterance (e.g., an implicit invocation utterance 234 received from the EIS 230). The intent classifier 242 may also determine a confidence score for each configured system-level intent (e.g., Help, Exit). If a particular confidence score satisfies one or more conditions, the skill bot call unit 240 will call the bot associated with that particular confidence score. For example, a certain threshold confidence score value may need to be met. Thus, the output 245 of the intent classifier 242 is either an identification of a system intent or an identification of a particular skill bot. In some embodiments, in addition to meeting a threshold confidence score value, the confidence score must exceed the next higher confidence score by a certain win margin. Imposing such conditions allows routing to a particular skill bot when the confidence scores of multiple skill bots each exceed a threshold confidence score value.

[0088] After identifying a bot based on the confidence score evaluation, the skill bot call unit 240 hands over processing to the identified bot. In the case of a system intent, the identified bot is the master bot. Otherwise, the identified bot is a skill bot. Furthermore, the skill bot call unit 240 will determine what to provide as input 247 to the identified bot. As described above, in the case of an explicit call, input 247 may be based on a portion of an utterance not associated with the call, or input 247 may be nothing (e.g., an empty string). In the case of an implicit call, input 247 may be the entire utterance.

[0089] The data store 250 comprises one or more computing devices that store data used by various subsystems of the master bot system 200. As described above, the data store 250 includes rules 252 and skillbot information 254. Rules 252 include rules for determining, for example, by the MIS 220, when an utterance represents multiple intents and how to divide an utterance that represents multiple intents. Rules 252 further include rules for determining, for example, by the EIS 230, which parts of an utterance that explicitly invokes a skillbot should be sent to the skillbot. Skillbot information 254 includes a list of invocation names of skillbots in the chatbot system, for example, the invocation names of all skillbots registered with a particular master bot. Skillbot information 254 also includes information for each skillbot in the chatbot system. The information used by the intent classifier 242 to determine the confidence score may include, for example, the parameters of the machine learning model.

[0090] 3. Components of the SkillBot System Figure 3 is a simplified block diagram of a skillbot system 300 according to one embodiment. The skillbot system 300 is a computing system that can be implemented with software only, hardware only, or a combination of hardware and software. In some embodiments, such as the embodiment shown in Figure 1, the skillbot system 300 can be used to implement one or more skillbots within a digital assistant.

[0091] The SkillBot system 300 includes an MIS 310, an intent classifier 320, and a conversation manager 330. The MIS 310 is similar to the MIS 220 in Figure 2 and provides similar functionality, including being operable to determine, using rules 352 in the data store 350, whether (1) an utterance represents multiple intents, and if so, how to split the utterance into separate utterances for each of the multiple intents. In one embodiment, the rules applied by the MIS 310 to detect multiple intents and split the utterance are the same as the rules applied by the MIS 220. The MIS 310 receives an utterance 302 and extracted information 304. The extracted information 304 is similar to the extracted information 205 in Figure 1 and can be generated using a language parser 214 or a language parser local to the SkillBot system 300.

[0092] The intent classifier 320 can be trained in a similar manner to the intent classifier 242 discussed above in relation to the embodiment of Figure 2, as will be described in more detail here. For example, in one embodiment, the intent classifier 320 is implemented using a machine learning model. The machine learning model of the intent classifier 320 is trained for a particular skill bot using at least a subset of exemplary utterances associated with that skill bot as training utterances. The ground truth for each training utterance would be the specific bot intent associated with that training utterance.

[0093] Utterance 302 can be received directly from the user or supplied via a master bot. When utterance 302 is supplied via a master bot, for example, as a result of processing through MIS220 and EIS230 in the embodiment shown in Figure 2, MIS310 can be bypassed to avoid repeating processing already performed by MIS220. However, when utterance 302 is received directly from the user, for example during a conversation that occurs after routing to a skill bot, MIS310 can process utterance 302 to determine whether utterance 302 represents multiple intents. If utterance 302 represents multiple intents, MIS310 applies one or more rules to split utterance 302 into separate utterances for each intent, for example, utterance "D" 306 and utterance "E" 308. If utterance 302 does not represent multiple intents, MIS310 forwards utterance 302 to intent classifier 320 for intent classification without splitting it.

[0094] The intent classifier 320 is configured to match an received utterance (e.g., utterance 306 or 308) with an intent associated with the skillbot system 300. As described above, a skillbot can be configured with one or more intents, each intent including at least one exemplary utterance associated with that intent and used to train the classifier. In the embodiment of Figure 2, the intent classifier 242 of the masterbot system 200 is trained to determine the confidence score of individual skillbots and the confidence score of system intents. Similarly, The intent classifier 320 can be trained to determine the confidence score of each intent associated with the skillbot system 300. The classification performed by the intent classifier 242 is at the bot level, while the classification performed by the intent classifier 320 is at the intent level and therefore of finer granularity. The intent classifier 320 has access to intent information 354. For each intent associated with the skillbot system 300, the intent information 354 represents and indicates the meaning of that intent and typically includes a list of utterances associated with tasks that can be performed by that intent. The intent information 354 may further include parameters generated as a result of training on this list of utterances.

[0095] The conversation manager 330 receives, as output from the intent classifier 320, an instruction 322 for a specific intent identified by the intent classifier 320 as the best match for the utterance input to the intent classifier 320. In some cases, the intent classifier 320 may be unable to determine any match. For example, the confidence score calculated by the intent classifier 320 may fall below a threshold confidence score value if the utterance is directed to a system intent or an intent of a different skillbot. When this occurs, the skillbot system 300 may leave the utterance to the master bot for processing, for example, to route it to a different skillbot. However, if the intent classifier 320 successfully identifies an intent within the skillbot, the conversation manager 330 initiates a conversation with the user.

[0096] A conversation initiated by the conversation manager 330 is a conversation specific to an intent identified by the intent classifier 320. For example, the conversation manager 330 may be implemented using a state machine configured to execute a certain dialogue flow for an identified intent. The state machine may include a default initiation state (for example, when an intent is invoked without any additional input) and one or more additional states, each associated with an action to be performed by the skillbot (e.g., execute a purchase transaction) and / or a dialogue to be presented to the user (e.g., a question, response). Thus, when the conversation manager 330 receives an instruction 322 that identifies an intent, it can determine an action / dialogue 335, and can determine additional actions or dialogues in response to subsequent utterances received during the conversation.

[0097] The data store 350 comprises one or more computing devices that store data used by various subsystems of the skillbot system 300. As shown in Figure 3, the data store 350 includes rules 352 and intent information 354. In one embodiment, the data store 350 can be integrated with the data store of the master bot or digital assistant, for example, the data store 250 in Figure 2.

[0098] 4. A scheme for classifying utterances using a trained intent classifier. Figure 4 is a block diagram showing a view of a chatbot system 400 configured to train and utilize a classifier (for example, the intent classifiers 242 or 320 described in relation to Figures 2 and 3) based on text data 405. As shown in Figure 4, the text classification performed by the chatbot system 400 in this example includes several stages: a machine learning model training stage 410; a skillbot invocation stage 415 to determine the likelihood that an utterance represents a task configured for a particular skillbot to perform; and an intent prediction stage 420 to classify the utterance as one or more intents. The machine learning model training stage 410 includes one or more machine learning models 425a to 425n (where "n" represents any natural number) used by the other stages (these may be individually called machine learning models 425, or collectively called machine learning models 425). A machine learning model (which may be exposed) is constructed and trained. For example, machine learning model 425 may include a model for determining the likelihood that an utterance represents a task configured for a particular skill bot to perform, another model for predicting intent from an utterance for a first type of skill bot, and another model for predicting intent from an utterance for a second type of skill bot. Further other types of machine learning models may be implemented in other examples provided herein.

[0099] Machine learning model 425 can be any other suitable ML model trained for natural language processing, such as a convolutional neural network ("CNN"), e.g., an inception neural network, a residual neural network ("Resnet"), or a recurrent neural network, e.g., a long short-term memory ("LSTM") model or a gated recurrent unit ("GRU") model, or other variations of a deep neural network ("DNN") (e.g., a multilevel n binary DNN classifier or a multiclass DNN classifier for single-intent classification). Machine learning model 425 can also be any other suitable ML model trained for natural language processing, such as a naive Bayes classifier, a linear classifier, a support vector machine, a random forest model, a bagging model, a boosting model, a shallow neural network, or a combination of one or more such techniques, e.g., a CNN-HMM or MCNN (multiscale convolutional neural network). The chatbot system 400 may employ the same or different types of machine learning models to determine the likelihood of a task configured for a particular skillbot to perform, to predict intents from utterances for a first type of skillbot, and to predict intents from utterances for a second type of skillbot. Further other types of machine learning models may be implemented in other examples provided herein.

[0100] To train various machine learning models 425, the training phase 410 consists of three main components: dataset preparation 430, feature engineering 435, and model training 440. Dataset preparation 430 includes the process of loading data assets 445, dividing the data assets 445 into training and validation sets 445a-n, and performing basic preprocessing so that the system can train and test the machine learning models 425. Data assets 445 may include at least a subset of utterances from exemplary utterances associated with various skillbots. As mentioned above, utterances can be provided in various ways, including speech or text. Utterances can be fragments, complete sentences, multiple sentences, etc. For example, if utterances are provided as speech, data preparation 430 may convert speech to text using a speech-to-text converter (not shown) that inserts punctuation, e.g., commas, semicolons, periods, etc., into the resulting text. In some examples, exemplary utterances are provided by a client or customer. In other examples, exemplary utterances are automatically generated from a library of previous utterances (for example, identifying utterances from a specific library for the skills the chatbot learns). The data asset 445 for the machine learning model 425 may include input text or speech (or input features of text or speech frames) and labels 450 corresponding to the input text or speech (or input features) as a matrix or table of values. For example, for each training utterance, instructions for the correct bot to use with that training utterance may be provided as ground truth information for the labels 450. The behavior of the machine learning model 425 can then be adapted (for example, through backpropagation) to minimize the difference between the generated inferences and the ground truth information. Alternatively, the machine learning model 425 may be trained for a particular skill bot using at least a subset of exemplary utterances associated with that particular skill bot as training utterances.The ground truth for label 450 for each training utterance would be the specific bot intent associated with that training utterance.

[0101] In various embodiments, feature engineering 435 involves converting the data asset 445 into feature vectors and / or creating new features using the data asset 445. Feature vectors may include count vectors as features, word-level, n-gram-level, or character-level word frequency-inverse document frequency (TF-IDF) vectors as features, word embeddings as features, text / NLP as features, topic models as features, or a combination thereof. A count vector is a matrix representation of the data asset 445 where each row represents an utterance, each column represents a word from the utterance, and each cell represents the frequency count of a particular word within the utterance. TF-IDF scores represent the relative importance of a word in an utterance. Word embeddings are a form of representing words and utterances using dense vector representations. The position of a word in vector space is learned from the text and is based on the words that surround that word when it is used. Text / NLP-based features may include the number of words in an utterance, the number of characters in an utterance, the average word density, the number of punctuation marks, the number of capital letters, the number of headwords, the frequency distribution of part-of-speech tags (e.g., nouns and verbs), or any combination thereof. Topic modeling is a technique for identifying groups of words (called topics) that contain the best information within a collection of utterances.

[0102] In various embodiments, model training 440 includes training a classifier using feature vectors and / or new features created in feature engineering 435. In some examples, the training process includes iterative operations to find a set of parameters for the machine learning model 425 that minimizes or maximizes a cost function (e.g., minimizing a loss function or error function) for the machine learning model 425. Each iteration may involve finding a set of parameters for the machine learning model 425 such that the value of the cost function using the set of parameters for the machine learning model 425 is less than or greater than the value of the cost function using a different set of parameters in the previous iteration. The cost function can be constructed to measure the difference between the output predicted using the machine learning model 425 and the labels 450 contained in the data asset 445. Once the set of parameters is identified, the machine learning model 425 is trained and can be used for prediction as designed.

[0103] In various embodiments, the training phase 410 further includes a logit decision 455. The logit decision 455 is configured to apply a logit function to the probability that an utterance is associated with a resolvable class. To determine the probability that an utterance is associated with an unresolvable class, the logit decision 455 may use an enhanced logit value instead of a logit function. The enhanced logit value may be processed (e.g., by an activation function) to determine the classification of an utterance, including the classification that the utterance has an unresolved intent. The use of enhanced logit values ​​by the logit decision 455 can improve the performance of the model and chatbot, particularly in predicting whether an utterance is out of scope or out of domain. Although the logit decision 455 is shown in Figure 4 as a subprocess of the training phase, in some embodiments, the logit decision 455 may form part of the model training 440, such that the enhanced logit value corresponding to an unresolvable class is a learned value that can be dynamically adjusted during training of the intent classifier. Additionally and / or alternatively, enhanced logit values ​​can be tuned by optimizing the hyperparameters of the intent classifier. Details of the implementation of logit decision 455 are further described below herein.

[0104] In addition to the data assets 445, labels 450, feature vectors and / or new features, other techniques and information can also be employed to improve the training process of the machine learning model 425. For example, feature vectors and / or new features can be used to improve the training process of the machine learning model 425. They may be combined with each other to help improve the accuracy of the classifiers or models. In addition, or alternatively, hyperparameters may be tuned or optimized, and for example, several parameters such as tree length, leaves, and network parameters may be fine-tuned to obtain a best-fit model. However, the training mechanisms described herein mainly focus on training machine learning models 425. These training mechanisms can also be used to fine-tune existing machine learning models 425 that have been trained on other data assets. For example, in some cases, machine learning model 425 may have been pre-trained using utterances specific to a different skillbot. In such cases, machine learning model 425 can be retrained using data asset 445, as discussed herein.

[0105] The machine learning model training stage 410 outputs a trained machine learning model 425, which includes a task machine learning model 460 and an intent machine learning model 465. The task machine learning model 460 may be used in the skillbot invocation stage 415 to determine the likelihood that an utterance represents a task configured for a particular skillbot to perform (470), and the intent machine learning model 465 may be used in the intent prediction stage 420 to classify an utterance as one or more intents (475). In some examples, the skillbot invocation stage 415 and the intent prediction stage 420 may proceed independently, using separate models in some examples. For example, the trained intent machine learning model 465 may be used in the intent prediction stage 420 to predict intents for a skillbot without first identifying the skillbot in the skillbot invocation stage 415. Similarly, the task machine learning model 460 may be used in the skillbot invocation stage 415 to predict the task or skillbot that should be used for the utterance, without identifying the intent of the utterance in the intent prediction stage 420.

[0106] Alternatively, the skillbot invocation stage 415 and the intent prediction stage 420 may be performed sequentially, with one stage using the output of the other stage as input, or one stage being invoked in a specific manner for a particular skillbot based on the output of the other stage. For example, for a given text data 405, the skillbot invocation unit can invoke a skillbot through an implicit invocation using the skillbot invocation stage 415 and the task machine learning model 460. The task machine learning model 460 is trained using machine learning and / or rule-based training techniques to determine the likelihood that a given utterance represents a task configured for a particular skillbot 470 to perform. Then, for the identified or invoked skillbot and the given text data 405, the intent prediction stage 420 and the intent machine learning model 465 can be used to match the received utterance (e.g., an utterance in a given data asset 445) to an intent 475 associated with the skillbot. As described herein, a skillbot can be composed of one or more intents, each intent including at least one exemplary utterance associated with that intent and used to train a classifier. In some embodiments, the skillbot invocation stage 415 and task machine learning model 460 used in a master bot system are trained to determine confidence scores for individual skillbots and for system intents. Similarly, the intent prediction stage 420 and intent machine learning model 465 may be trained to determine confidence scores for each intent associated with the skillbot system. The classification performed by the skillbot invocation stage 415 and task machine learning model 460 is at the bot level, while the classification performed by the intent prediction stage 420 and intent machine learning model 465 is at the intent level and therefore of finer granularity.

[0107] C. Logit function Figure 5 shows schematic diagrams of exemplary neural networks 500 according to several embodiments. The neural network 500 can be a machine learning model trained by a training system and implemented by a chatbot system, where the machine learning model is trained to predict whether an utterance corresponds to a particular intent class. In some examples, the neural network 500 includes an input layer 502, a hidden layer 504, and an output layer 506. In some examples, the neural network 500 includes multiple hidden layers, where the hidden layer 504 corresponds to the final hidden layer of the neural network.

[0108] The input layer 502 receives input data or a representation of the input data (e.g., an n-dimensional array of values ​​representing an utterance) and can apply one or more learned parameters to the input data to generate a set of outputs (e.g., a set of numbers). The set of outputs can be processed by the hidden layer 504.

[0109] The hidden layer 504 may include one or more learned parameters that transform the set of outputs into different feature spaces where input data points from different classes are better separated.

[0110] The output layer 506 may include a classification layer that maps the output from the hidden layer to logit values, each corresponding to one specific class (e.g., a resolvable class, an unresolvable class). In some examples, the output layer 506 includes an activation function to constrain the logit values ​​to a set of probability values ​​that sum to 1. Thus, the logit values ​​generated by the classification layer (logit function) for each class can then be processed by an activation function in the output layer (also referred to herein as the “activation layer”) to predict the classification for an utterance.

[0111] As part of predicting a single intent from a set of output values, a machine learning model may employ a logit function (short for “logistic regression function”) in the network layers of a neural network 500. The output layer 506 may include a logit function for converting intermediate outputs (e.g., probability values ​​predicting whether an utterance corresponds to a particular class) into logit values. In some cases, the logit function is the logarithm of odds, taking inputs corresponding to probability values ​​between 0 and 1 for a particular class, and outputting logit values ​​within an unbounded range between negative infinity and positive infinity. The logit function may be used to normalize each intermediate output in the set of intermediate outputs, so that the resulting set of logit values ​​can be represented as a unimodal probability distribution symmetrical across the predicted output classes. Mathematically speaking, the logit function is,

[0112]

number

[0113] The logit function is defined as follows, where p is the probability of an input corresponding to a particular class. The probability p can be set within the range of (0,1). The output logit can correspond to logit values ​​within the range of (-∞,+∞). In this way, a machine learning model may employ a logit function such that the subsequent activation function can predict the most likely output (e.g., intent) from the classifier's other less likely outputs.

[0114] As part of training a machine learning model (for example, the intent classifier 320 in Figure 3), the model may learn the centroids for each class, where the centroids are part of the logit function. In particular, the centroids are used to weight the logit function, for example, It can be used as part of filtering classifier output to separate possible intents from utterances. In mathematical terms, the equation relating the logit function is logit i = f(x)*W iThis may also be the case, and in the formula, x is the input to the model. f(x) is a set of transformations (e.g., a highway network function), and W i This acts as a weighting coefficient for the model output for intent "i", at the centroid of that intent. Therefore, for a set of identified intents that an intent classifier has trained to recognize, the centroid plays a role in classifying an utterance as belonging to a particular intent rather than others, for example, by using the centroid as a position to measure the distance between adjacent clusters.

[0115] Next, the logit values ​​generated for each predicted output class can be processed by an activation function to map the odds represented by the logit values ​​to a probability distribution across all predicted output classes. As an example of an activation function, the softmax function can be used. Use this to get the network output (for example, the logit value corresponding to the order_pizza class) This applies to all predicted output classes (e.g., order_pizza, unresolved intents). It may also be normalized to a probability distribution.

[0116] Before applying softmax, the logit value may be negative or greater than 1. And the sum may not be 1. When softmax is applied, each output is in the interval ( This means the input will be in the range 0,1), and the output will sum to 1. Furthermore, a larger input logit value will correspond to a larger probability. In functional terms, the softmax function is: It is represented as follows:

[0117]

number

[0118] i = 1:K, and z is the set of K real numbers (z1: z K ) In some cases, the softmax function can be weighted by a base factor b that produces a probability distribution more concentrated around the location of the largest input value. In such cases, the softmax function is

[0119]

number

[0120] And in the formula, β is a real number. In the context of intent classification, z i is f(x)*w i Set to That's good too.

[0121] For a set of utterances, a resolved intent may fit into one of the identified intents that an intent classifier can train on. An unresolved intent may, by definition, describe an utterance that the classifier cannot confidently place into any of the intent categories. As an illustrative example, a skillbot trained to handle pizza orders may be trained to classify utterances into one of three intents: "order_pizza," "cancel_pizza," and "unresolvedIntent." An utterance intended to order a new pizza or modify an existing order might be classified, for example, into "order_pizza." It may be classified as such. In contrast, an utterance intended to cancel an existing order may be classified as "cancel_pizza". Finally, an utterance intended to do something else may be classified under "unresolvedIntent". For example, "What will the weather be like tomorrow?" is not related to ordering a pizza. It is irrelevant and should be classified as "unresolvedIntent". Therefore, in this example, "unresolvedIntent" is negative for "order_pizza" and "cancel_pizza". It is defined by its meaning. Utterances that do not fit the latter fit the former.

[0122] Therefore, unresolved intents may be defined to cover utterances that do not fit into the identified set of intents. In this way, for unresolved classes, the classifier output may be distributed rather than clustered. While the examples described herein focus on intent prediction, it should be understood that other classification tasks can be handled similarly. For example, a chatbot system may include a classifier to handle "unresolved" utterances at other levels, such as skillbot calls.

[0123] D. Techniques for classifying utterances as having unresolved intents using enhanced logits. Applying the concept of centroid as a weighting coefficient in a logit function can be effective in some cases. However, in the case of the "unresolved intent" class, the effectiveness of using centroid is greatly reduced. For defined intents such as "order_pizza", The discussion can cluster around the center of gravity. ("unresolved_intent") If a value is defined as a negative classification for all utterances outside the set of classifications, such utterances may indicate a relative lack of clustering. Therefore, learning the centroid can introduce errors into logistic regression through inaccurate weighting coefficients.

[0124] For example, in the case of the pizza skill bot mentioned above, utterances with an intent to order or an intent to cancel may cluster around their respective centroids, while utterances ranging from weather to finance and asking about specific people may fall into the "unresolved_intent" category. Similarly, to direct a query to a specific skill bot, the classifier may output the probability that the utterance invokes a specific skill bot or that the utterance does not invoke any of the skill bots the system has been trained to classify. In each of these cases, the centroid for the unresolved class may not represent the population characteristics that arise from the definition of “unresolved” classification as including any utterance that does not map to any learned intent, skill, domain, or scope.

[0125] To overcome this problem and others, various embodiments address techniques for using enhanced logit values ​​for unresolved intents to improve the classification of utterances as unresolved intents. For example, a machine learning model (e.g., the intent classifier 320 in Figure 3) can implement enhanced logit values ​​for unresolved intent classes. For example, enhanced logit values ​​for unresolved classes may be obtained by using a modified logit function that includes trainable scaler parameters instead of calculating the logit value with the centroid as a weighting coefficient. Additionally or alternatively, enhanced logit values ​​for unresolved classes can be optimized through hyperparameter tuning of the machine learning model. In some examples, if the training data is insufficient to indicate a resolvable class, the output is classified by the intent classifier as an unresolved class. In practice, the intent classifier may determine whether an utterance invokes a resolvable class or not, rather than determining whether an utterance invokes an unresolved class.

[0126] Enhanced logit values ​​can be obtained using an automated process that can be integrated into the chatbot system, as illustrated with respect to Figures 1, 2, and 3, whenever utterance classification (e.g., intent, scope, skill, etc.) can be undertaken. Advantageously, models and chatbots perform better when classifying out-of-scope utterances using enhanced logit values ​​because, at least in part, they are enhanced. The resulting logit values ​​may not reflect true clustering in the model output because they do not rely on the centroid of unresolved classes. Furthermore, because extensions are applied automatically, customers or clients are less likely to experience misdirected inquiries from the chatbot system.

[0127] As further described below, the enhanced logit value for classifying utterances into unsolvable classes can be based on one of: (i) a statistical value determined based on a set of logit values generated from a training dataset; (ii) a value selected from a range of values defined by the logarithm of a first odds corresponding to the probability for the unsolvable class, the logarithm of the first odds being a bounded value constrained to a range of values by a bounding function and weighted by the centroid of the distribution associated with the unsolvable class; (iii) a weighted value generated by the logarithm of a second odds corresponding to the probability for the unsolvable class, the logarithm of the second odds being scaled by a scaling factor, bounded by a bounding function, and weighted by the centroid of the distribution associated with the unsolvable class; (iv) a hyperparameter optimization value generated based on hyperparameter tuning of a machine learning model; or (v) a learning value dynamically adjusted during training of a machine learning model. Additionally or alternatively, a rule-based approach can be implemented as part of determining the enhanced logit value. The rule-based approach may include a logical operator for identifying a particular enhanced logit value from a set of enhanced logit values for the unsolvable class. For example, the enhanced logit value can be retrieved from a database or lookup table based on a query value generated by applying a set of rules to the utterance.

[0128] 1. Statistical value In some embodiments, the enhanced logit value is a statistical value that replaces the logit function when determining whether an utterance corresponds to an out-of-scope class or an out-of-domain class. The logit value can include a value selected from a range of values. Referring to the functional equation logit i =f(x)*W i logit i for the unsolvable class is the function f(x)*W i to a given number "A" (i.e., logit i ​= It can be set by replacing it with A). Such a configuration is fixed value

[0129]

number

[0130] It provides a probability function that depends on [the specified value]. In some examples, the statistic is set to 0.3, 0.5, 0.7, 0.9, 1.0, or any higher number. The statistic is determined so that the corresponding probability value indicates that the utterance is classified as unresolved (e.g., an unresolved skill or an unresolved intent).

[0131] In some embodiments, the statistics are based on the probability that out-of-domain or out-of-scope utterances in the training set of utterances map to classes that cannot be resolved. Determining appropriate statistics may be part of training stage 410 in Figure 4, and the number may partially depend on the training dataset applied in training stage 410 (e.g., data asset 445).

[0132] Figure 6 shows a flowchart illustrating an exemplary process 600 for obtaining statistics representing enhanced logit values ​​to predict whether an utterance corresponds to an unresolved class, according to several embodiments. The process shown in Figure 6 is performed by software (for example, a processor, a core) on one or more processing units (e.g., processors, cores) of each system. For example, it may be implemented in code, instructions, or programs, hardware, or a combination thereof. The software may be stored on a non-temporary storage medium (e.g., on a memory device). The methods presented in Figure 6 and described below are intended to be illustrative and non-limiting. Figure 6 shows various processing steps occurring in a particular sequence or order, but this is not intended to be limiting. In some embodiments, those steps may be performed in some different order, or some steps may be performed in parallel. In some embodiments, such as the embodiments shown in Figures 1 to 4, the processing shown in Figure 6 may be performed by a training subsystem to map probability values ​​corresponding to unresolved classes to enhanced logit values ​​of one or more machine learning models (e.g., intent classifiers 242 or 320). The training subsystem may be part of a data processing system (e.g., the chatbot system 400 described with respect to Figure 4) or a component of another system configured to train and deploy machine learning models.

[0133] In 605, the training subsystem receives a training dataset. The training dataset may include a set of utterances or messages. Each utterance in the set is associated with a training label, which identifies the predicted intent class of the utterance. The training dataset may also include utterances or messages that are out of scope from a particular task that the corresponding skill or chatbot is configured to perform. For example, if a skillbot is configured to process food orders for a pizza restaurant, an out-of-scope utterance might be a message asking about the weather. In some examples, each out-of-scope utterance or message in the training dataset is associated with a training label, which identifies the utterance or message as belonging to a class that cannot be resolved.

[0134] In 610, the training subsystem uses the training dataset to train a machine learning model to predict whether an utterance or message represents a task configured for the skillbot to perform, or to match an utterance or message to an intent associated with the skillbot. One or more parameters of the machine learning model may be learned to minimize the loss between the predicted output produced by the machine learning model and the expected output indicated by the training label of the corresponding utterance. Training of the machine learning model can be performed until the loss reaches a minimum error threshold.

[0135] In 615, the training subsystem identifies a set of training logit values. Each training logit value in the set corresponds to an out-of-scope utterance in the training dataset. Each training logit value can be generated based on inputting each out-of-scope utterance into a machine learning model. To obtain the training logit values, one or more helper functions (e.g., the get_layer function) are used to perform machine learning. It is possible to import the output generated from one or more layers of the model.

[0136] In 620, the training subsystem calculates a statistic representing the set of training logit values. For example, the statistic could be the median of the set of training logit values. The median may be used in response to a judgment that the statistical distribution of the set of training logit values ​​corresponds to a skewed distribution. In some examples, the mean and median are calculated for the set of training logit values. If the difference between the mean and the median is statistically significant, the median is selected as the statistic representing the set of training logit values.

[0137] In 625, the training subsystem handles classes that utterances cannot resolve. Set a statistical value as an enhanced logit value to predict whether or not. The enhanced logit value can replace the existing logit function of a machine learning model that would have been used to predict whether an utterance corresponds to an unresolved class. In some examples, the existing logit function is for resolvable classes (e.g., order_pizza, cancel_pizza) It continues to be used to predict the utterance corresponding to one of the following.

[0138] In step 630, the training subsystem deploys the trained machine learning model to the chatbot system (for example, as part of a skillbot), and the trained machine learning model includes statistics set as enhanced logit values ​​to predict whether an utterance corresponds to a class that cannot be resolved. The process then terminates in step 600.

[0139] By using statistics and removing the centroid's influence from unresolved classes, machine learning model performance can be significantly improved in accurately predicting whether an utterance invokes a resolvable or unresolved class. In an exemplary example, the pizza skill described above contains the following three intents: "order_pizza", "cancel_pizza", and "unresolvedIntent". The utterance "What will the weather be like tomorrow?" contains three intents. Regarding the center of gravity weighting coefficient w i The logit value calculated based on (for example, logit i = F(x)*w i ) has (0.3,0.01,0.01). In this case, the logit vector The three values ​​are obtained using the centroid-weight relationship and the softmax function is used to determine the probability (0.4 It is converted to ,0.3,0.3). In this example, "order_pizza" is converted to 40% confidence. This is the most likely result with a core. However, it should be clear that a query about weather does not represent an intent to order a pizza. The classification in this example is inappropriate, partly due to the relatively low logit value of the unresolved class, which is equal to 0.01 (the correct classification would be out-of-domain unresolvedIntent).

[0140] Continuing from this example, the enhanced logit value could be a statistic equal to 0.4. The statistic can be applied while maintaining the centroid-weighted logit value for the resolvable class. Thus, the logit value can be redefined as (0.3, 0.01, 0.4). Applying the softmax function again, the logit value becomes (0.35, 0.26, 0. 39) This is converted to a probability. In this case, the result of the softmax function is a 39% confidence score. This reflects the precise classification of "unresolvedIntent" that possesses this characteristic.

[0141] 2. Bounded values Finding enhanced logit values ​​for unsolvable classes can be complicated because the standard logit function is unbounded. As mentioned above, the logit function outputs logit values ​​in the range between negative infinity and positive infinity. Therefore, the unbounded range can introduce complexity to finding enhanced logit values. For example, logit values ​​in the range extending to negative and positive infinity may result in the logit values ​​being distributed across a very wide range of values, which can make optimizing enhanced logit values ​​difficult.

[0142] Therefore, an enhanced logit value can be a bounded value within the range specified by the modified logit function. The modified logit function can be modified to restrict the logit value to a range including a lower and upper bound. In some examples, the modified logit function includes a bounded function. A bounded function can include a trigonometric function that produces an output within the range between negative 1 and 1, such as the sine or cosine function. For example, the logit function is logit i = cosine(f(x)*W i ) can be defined as such, with a minimum value of -1 and a maximum value of 1. The enhanced logit value can then be selected from a bounded range between -1 and 1 for classes that cannot be resolved. Continuing from the example, the order_pizza class The logit function for the `s` and `cancel_pizza` classes is `logit`. i = cosine(f(x)*W i ) can be used, and a bounded value of 0.4 may be chosen for the unresolvedIntent class.

[0143] Figure 7 shows a flowchart illustrating an exemplary process 700 for modifying a logit function to obtain an enhanced logit value within a specified range to predict whether an utterance corresponds to an unresolved class, according to several embodiments. The process shown in Figure 7 may be implemented by software (e.g., code, instructions, programs), hardware, or a combination thereof, executed by one or more processing units (e.g., processors, cores) of each system. The software may be stored on a non-temporary storage medium (e.g., on a memory device). The methods presented in Figure 7 and described below are intended to be exemplary and non-limiting. Figure 7 shows various processing steps that occur in a particular sequence or order, but this is not intended to limit them. In some embodiments, the steps are performed by a training subsystem to map probability values ​​corresponding to unresolved classes to enhanced logit values ​​of one or more machine learning models (e.g., intent classifiers 242 or 320). The training subsystem may be part of a data processing system (e.g., the chatbot system 400 described with respect to Figure 4) or a component of another system configured to train and deploy machine learning models.

[0144] In 705, the training subsystem initializes the machine learning model. The machine learning model may be a convolutional neural network ("CNN"), e.g., an inception neural network, a residual neural network ("Resnet"), or a recurrent neural network, e.g., a long short-term memory ("LSTM") model or a gated recurrent unit ("GRU") model, or other variants of a deep neural network ("DNN") (e.g., a multilevel n binary DNN classifier or a multiclass DNN classifier for single-intent classification). The machine learning model 425 may also be a naive Bayes classifier, a linear classifier, a support vector machine, a bagging model such as a random forest model, a boosting model, a shallow neural network, or a combination of one or more such techniques, e.g., a CNN-HMM or MCNN (multiscale convolutional neural network), or any other suitable ML model trained for natural language processing.

[0145] Initializing a machine learning model can include defining the number of layers, the type of each layer (e.g., fully connected, convolutional neural network), and the type of activation function for each layer (e.g., sigmoid, ReLU, softmax).

[0146] In step 710, the training subsystem extracts the logit function from the final layer of the machine learning model. For example, the training system can select the final layer from a set of layers within a fully connected neural network. The logit function of the final layer can then be accessed for modification.

[0147] In 715, the training subsystem modifies the logit function by adding a bounded function. The bounded function can include trigonometric functions that produce an output within the range of negative 1 and 1, such as the sine or cosine function. In some examples, the bounded function is given by the following equation:

[0148]

number

[0149] One of the following is selected, where f(x) is the logit function. As an addition or alternative, the logit function is the centroid W i It can include, where i corresponds to each predicted class. .

[0150] In 720, the training subsystem trains a machine learning model to generate enhanced logit values ​​based on a range specified by the modified logit function. Thus, the training subsystem can train a machine learning model to generate enhanced logit values ​​by identifying statistics from the training dataset that predict whether an utterance corresponds to an unresolved class (e.g., an unresolved skill or an unresolved intent).

[0151] In step 725, the training subsystem deploys the trained machine learning model to the chatbot system (for example, as part of SkillBot), and the trained machine learning model includes a modified logit function. Process 700 then terminates.

[0152] 3. Weighted values Enhanced logit values ​​for predicting out-of-scope utterances may be weighted values ​​produced by a modified logit function, weighted by a scaling factor. In some examples, the scaling factor (also referred to herein as the "scaler") expressed as a function for class "i" is logit i =scaler * cosine(f(x)*W) i ) is the solution. The scaling factor value (e.g., 2) used for unresolved classes (e.g., unresolvedIntent) is the same as the scaling factor value (e.g., 2) used for resolvable classes (e.g., order_pizza, cancel_pizza). The scaling factor used can be different from the value of (e.g., 1). Including the scaling factor in the logit function can widen the gap between the probability of a predicted intent and the probabilities of other intents, thus allowing predicted intents of the unresolved class to be better distinguished from other intents of the resolvable class.

[0153] Figure 8 shows a flowchart 800 illustrating an exemplary process for adding a scaling factor to a logit function to obtain an enhanced logit value to predict whether an utterance corresponds to an unresolved class, according to several embodiments. The process shown in Figure 8 may be implemented by software (e.g., code, instructions, programs), hardware, or a combination thereof, executed by one or more processing units (e.g., processors, cores) of each system. The software may be stored on a non-temporary storage medium (e.g., on a memory device). The methods presented in Figure 8 and described below are intended to be exemplary and non-limiting. Figure 8 shows various processing steps that occur in a particular sequence or order, but this is not intended to limit them. In some embodiments, the steps are performed by a training subsystem to map probability values ​​corresponding to unresolved classes to enhanced logit values ​​of one or more machine learning models (e.g., intent classifiers 242 or 320). The training subsystem may be part of a data processing system (e.g., the chatbot system 400 described with respect to Figure 4) or a component of another system configured to train and deploy machine learning models.

[0154] In 805, the training subsystem initializes the machine learning model. The machine learning model is a convolutional neural network ("CNN"), such as an inception neural network, a residual neural network ("ResNet"), or a recurrent neural network, such as a long short-term memory ("LSTM") model or a gated recurrent neural network. Numerical Unit ("GRU") models, other variations of Deep Neural Networks ("DNNs") (for example, a multilevel n-binary DNN classifier or a multiclass DNN classifier for single-intent classification), machine learning models 425 can also be any other suitable ML model trained for natural language processing, such as a naive Bayes classifier, a linear classifier, a support vector machine, a random forest model, a bagging model, a boosting model, a shallow neural network, or a combination of one or more such techniques, for example, a CNN-HMM or MCNN (Multiscale Convolutional Neural Network).

[0155] Initializing a machine learning model can include defining the number of layers, the type of each layer (e.g., fully connected, convolutional neural network), and the type of activation function for each layer (e.g., sigmoid, ReLU, softmax).

[0156] In step 810, the training subsystem extracts the logit function from the final layer of the machine learning model. For example, the training system can select the final layer from a set of layers within a fully connected neural network. The logit function of the final layer can then be accessed for modification.

[0157] In 815, the training subsystem modifies the logit function by adding a scaling factor. In some examples, different initial values ​​for the scaling factor are assigned based on the predicted class of the corresponding utterance. For example, a scaling factor value (e.g., 2) used for unresolved classes (e.g., unresolvedIntent) is used for resolvable classes (e.g., order_pizza, cancel_pizza) The value of the scaling coefficient (e.g., 1) can be different. The first scaling coefficient of a logit function that generates the probability that an utterance corresponds to an unresolvable class is assigned a value greater than the second scaling coefficient of another logit function that generates the probability that an utterance corresponds to one of the resolvable classes. As an addition or alternative, the logit function may be further modified by adding a bounded function (e.g., a cosine function) to the logit function to which the scaling coefficient is added.

[0158] In 820, the training subsystem trains a machine learning model to generate enhanced logit values ​​based on scaling factors assigned to unresolved classes. In some examples, the scaling factors for unresolved classes are learned parameters of the machine learning model. Thus, the training subsystem can train the machine learning model to adjust the scaling factors corresponding to unresolved classes. In addition, or alternatively, the scaling factors may be hardcoded numbers that remain static throughout the training and deployment of the machine learning model.

[0159] In step 825, the training subsystem deploys the trained machine learning model to the chatbot system (for example, as part of SkillBot), where the trained machine learning model includes a modified logit function. Process 800 then terminates.

[0160] 4. Hyperparameter Tuning Enhanced logit values ​​for predicting whether an utterance corresponds to an unresolved class may be hyperparameter-optimized values ​​determined based on hyperparameter tuning of the machine learning model. In some embodiments, hyperparameter tuning enables meta-training of enhanced logit values ​​as part of logit decisions. Tunable hyperparameters of the machine learning model include learning rate, epochs, etc. This can include the number of elements, momentum, regularization constant, the number of layers in the machine learning model, and the number of weights in the machine learning model. Hyperparameter tuning can involve determining the optimal combination of the above hyperparameters, which allows the intent classifier to obtain enhanced logit values ​​for classes it cannot resolve.

[0161] Figure 9 shows a flowchart 900 illustrating an exemplary process for using hyperparameter tuning to obtain enhanced logit values ​​to predict whether an utterance corresponds to an unresolved class, in several embodiments. The process shown in Figure 9 may be implemented by software (e.g., code, instructions, programs), hardware, or a combination thereof, executed by one or more processing units (e.g., processors, cores) of each system. The software may be stored on a non-temporary storage medium (e.g., on a memory device). The methods presented in Figure 9 and described below are intended to be illustrative and non-limiting. Figure 9 shows various processing steps that occur in a particular sequence or order, but this is not intended to be limiting. In some embodiments, those steps may be executed in some different order, or some steps may be executed in parallel. In some embodiments, such as the embodiments shown in Figures 1 to 4, the process shown in Figure 9 may be performed by a training subsystem to map probability values ​​corresponding to unresolved classes to enhanced logit values ​​of one or more machine learning models (e.g., intent classifiers 242 or 320). The training subsystem may be an intent classifier of a data processing system (for example, the chatbot system 400 described in relation to Figure 4) or a component of another system configured to train and deploy a machine learning model.

[0162] In 905, the training subsystem receives a training dataset. The training dataset may include a set of utterances or messages. Each utterance in the set is associated with a training label, which identifies the predicted intent class of the utterance. The training dataset may also include utterances or messages that are out of scope from a particular task that the corresponding skill or chatbot is configured to perform. For example, if a skillbot is configured to process food orders for a pizza restaurant, an out-of-scope utterance might be a message asking about the weather. In some examples, each out-of-scope utterance or message in the training dataset is associated with a training label, which identifies the utterance or message as belonging to a class that cannot be resolved.

[0163] In 910, the training subsystem uses the training dataset to train a machine learning model to predict whether an utterance or message represents a task configured for the skillbot to perform, or to match an utterance or message to an intent associated with the skillbot. One or more parameters of the machine learning model may be learned to minimize the loss between the predicted output produced by the machine learning model and the expected output indicated by the training label of the corresponding utterance. Training of the machine learning model can be performed until the loss reaches a minimum error threshold.

[0164] In 915, the training subsystem identifies a set of training logit values. Each training logit value in the set corresponds to an out-of-scope utterance in the training dataset. Each training logit value can be generated based on inputting each out-of-scope utterance into a machine learning model. To obtain the training logit values, one or more helper functions (e.g., the get_layer function) are used to perform machine learning. It is possible to import the output generated from one or more layers of the model.

[0165] In 920, the training subsystem calculates a statistic representing the set of training logit values. For example, the statistic could be the median of the set of training logit values. The median may be used in response to a judgment that the statistical distribution of the set of training logit values ​​corresponds to a skewed distribution. In some examples, the mean and median are calculated for the set of training logit values. If the difference between the mean and the median is statistically significant, the median is selected as the statistic representing the set of training logit values.

[0166] In 925, the training subsystem tunes one or more hyperparameters of the machine learning model to optimize its statistics. Training of the machine learning model can be iterated with the tuned hyperparameters, where the combination of hyperparameters that provided the best performance for predicting unresolved classes can be selected. In some embodiments, the best performance corresponds to a model configuration of the machine learning model that generates the minimum loss between the predicted output produced by the machine learning model and the expected output indicated by the training label of the corresponding out-of-scope utterance for out-of-scope utterances in the training dataset.

[0167] Various hyperparameter optimization techniques can be used to optimize statistical values. In some examples, random search techniques are used for hyperparameter tuning. Random search techniques may involve generating a grid of possible values ​​for the hyperparameters. Each iteration of the search can select a random combination of hyperparameters from this grid, and the performance of each iteration can be recorded. In addition, or as an alternative, grid search techniques can be used for hyperparameter tuning. Grid search techniques may involve generating a grid of possible values ​​for the hyperparameters. Each iteration of the search can select combinations of hyperparameters in a specific order. The performance of each iteration can be recorded. The combination of hyperparameters that provided the best performance can be selected.

[0168] Other techniques for hyperparameter tuning can be considered. These techniques include, but are not limited to, Bayesian optimization algorithms, tree-based Parzen estimators (TPEs), hyperbanding, population-based training (PBT), and Bayesian optimization and hyperbanding (BOHB).

[0169] In 930, the training subsystem sets optimized statistics as enhanced logit values ​​for unresolved classes. These enhanced logit values ​​can replace the existing logit function of the machine learning model that would have been used to predict that an utterance corresponds to an unresolved class. In some examples, the existing logit function corresponds to one of the resolvable classes (e.g., order_pizza, cancel_pizza). It continues to be used to predict the corresponding utterance. In 935, the training subsystem deploys the trained machine learning model to the chatbot system (e.g., as part of a skillbot), and the trained machine learning model includes optimized statistics that predict whether the utterance or message corresponds to a class that cannot be resolved. Process 900 then terminates.

[0170] 5. Learned values Enhanced logit values ​​for predicting whether an utterance corresponds to an unresolved class can be learned values ​​that can be dynamically adjusted during training of the machine learning model. Model parameters can be learned during the training phase of the machine learning model. In practice, additional training data (e.g., out-of-scope utterances) is used for training. This allows for the optimization of enhanced logit values.

[0171] Various techniques can be employed to generate and refine the learned values. Figure 10 shows a flowchart 1000 illustrating an exemplary process for using the learned values ​​as enhanced logit values ​​to predict whether an utterance corresponds to an unresolved class, according to several embodiments. The process shown in Figure 10 may be implemented by software (e.g., code, instructions, programs), hardware, or a combination thereof, executed by one or more processing units (e.g., processors, cores) of each system. The software may be stored on a non-temporary storage medium (e.g., on a memory device). The methods presented in Figure 10 and described below are intended to be illustrative and non-limiting. Figure 10 shows various processing steps that occur in a particular sequence or order, but this is not intended to be limiting. In some embodiments, the steps are performed by a training subsystem to map probability values ​​corresponding to unresolved classes to enhanced logit values ​​of one or more machine learning models (e.g., intent classifiers 242 or 320). The training subsystem may be part of a data processing system (e.g., the chatbot system 400 described with respect to Figure 4) or a component of another system configured to train and deploy machine learning models.

[0172] In 1005, the training subsystem receives a training dataset. The training dataset may include a set of utterances or messages. Each utterance in the set is associated with a training label, which identifies the predicted intent class of the utterance. The training dataset may also include utterances or messages that are out of scope from a particular task that the corresponding skill or chatbot is configured to perform. For example, if a skillbot is configured to process food orders for a pizza restaurant, an out-of-scope utterance might be a message asking about the weather. In some examples, each out-of-scope utterance or message in the training dataset is associated with a training label, which identifies the utterance or message as belonging to a class that cannot be resolved.

[0173] In step 1010, the training subsystem performs batch equilibration of the training dataset to generate an augmented training dataset that includes an augmented copy of out-of-scope utterances. This step may be optional based on the amount of out-of-scope utterances available in the training dataset. For example, the training subsystem may generate duplicate or augmented copies of the training data corresponding to classes that cannot be resolved (e.g., out-of-scope utterances). Data augmentation techniques may include inverse transformation of utterances in the training dataset, synonym substitution of one or more tokens in an utterance, random insertion of tokens into an utterance, swapping of two tokens in an utterance, and random deletion of one or more tokens in an utterance.

[0174] In 1015, the training subsystem initializes the machine learning model. The machine learning model may be a convolutional neural network ("CNN"), e.g., an inception neural network, a residual neural network ("Resnet"), or a recurrent neural network, e.g., a long short-term memory ("LSTM") model or a gated recurrent unit ("GRU") model, or other variants of a deep neural network ("DNN") (e.g., a multilevel n binary DNN classifier or a multiclass DNN classifier for single-intent classification). The machine learning model 425 may also be a naive Bayes classifier, a linear classifier, a support vector machine, a random forest model, a bagging model, a boosting model, a shallow neural network, or a combination of one or more such techniques, e.g., a CNN-HMM or MCNN (multiscale This could be any other suitable ML model trained for natural language processing, such as a convolutional neural network.

[0175] Initializing a machine learning model can include defining the number of layers, the type of each layer (e.g., fully connected, convolutional neural network), and the type of activation function for each layer (e.g., sigmoid, ReLU, softmax).

[0176] In 1020, the training subsystem trains a machine learning model using a training dataset or an augmented training dataset to predict whether an utterance or message represents a task configured for the skillbot to perform, or to match an utterance or message to an intent associated with the skillbot. One or more parameters of the machine learning model may be learned to minimize the loss between the predicted output produced by the machine learning model and the expected output indicated by the training label of the corresponding utterance. Training of the machine learning model can be performed until the loss reaches a minimum error threshold. In particular, the machine learning model may be trained using an augmented training dataset so that it can generate probabilities of whether a given utterance corresponds to an intent that cannot be resolved.

[0177] In some cases, to reduce overfitting of a machine learning model with the training dataset or an augmented training dataset, the training subsystem can use regularization to adjust one or more parameter weights. For example, model parameter regularization may include Gaussian / L2 regularization, where a small percentage of the weights of the machine learning model are removed in each training iteration. Such an approach can improve the accuracy of the enhanced logit value with a relatively small increase in the loss value, thereby mitigating the overfitting problem.

[0178] In some cases, the performance of a trained machine learning model is evaluated by cross-validation. For example, a trained machine learning model can be evaluated using k-fold cross-validation, which involves (i) shuffling the training data (e.g., utterances) within an augmented training dataset; (ii) dividing the training dataset into k subsets; (iii) training each machine learning model using each of the (k-1) subsets; (iv) evaluating the trained machine learning model using the provided k1 subsets; and (v) different provided k i This involves repeating steps (iii) and (iv) using a subset.

[0179] In step 1025, the training subsystem deploys the trained machine learning model to the chatbot system (e.g., as part of a skillbot) to generate enhanced logit values ​​that predict whether an utterance or message corresponds to an unresolved class. Since the machine learning model was trained with the enhanced training dataset, the enhanced logit values ​​can be used to accurately determine the probability values ​​that an out-of-scope utterance will predict to be associated with an unresolved class. Process 1000 then terminates.

[0180] 6. Example of calculating an enhanced logit value As mentioned above, enhanced logit values ​​for unresolved classes can be dynamically adjusted using machine learning techniques based on coefficients including, but not limited to, data size or intent number. For example, a scaler might adjust the class (e.g., order_pizza (This can be changed for each unresolvedIntent) and limits the erroneous optimality in the scaler value. Weight decay is then added. In another example, the logit may be dynamically changed within a given boundary. For unresolved intent classes, the logit value can be learned as part of machine learning model training, but may be constrained to a certain range (e.g., 0.0 => 0.2).

[0181] In an exemplary example of a machine learning model that uses logit values ​​for unresolved intent classes, a fixed logit value is applied to the unresolved class, and the trained model is tested against two standard out-of-domain test sets. The data shown in Table 1 below demonstrates the improvement in out-of-domain classification when logit values ​​are applied without using a centroid-weighted approach. For example, by applying a given logit value, the accuracy of the trained classifier in identifying out-of-domain utterances improved by 9% (comparing isMatch values ​​in row 5 and row 1), and the recall over epoch improved by 8% (comparing isMatch values ​​in row 5 and row 1). (Compare the e2e_ood_recall values ​​for 1).

[0182] [Table 1]

[0183] Furthermore, the experimental results did not reflect a corresponding or simultaneous decrease in performance with respect to the accurate classification of intradomain utterances, as shown in Table 2 below. For example, the intradomain test was 0, in contrast to a cosine-bounded softmax function weighted by the centroids of other intent classifications. Applying a predetermined logit value of 0.5 did not result in a significant decrease in accuracy between row 3 and row 5.

[0184] [Table 2]

[0185] E. Process for training a machine learning model to achieve enhanced logit values ​​for out-of-scope speech classification. Figure 11 is a flowchart of a process 1100 for training a machine learning model to achieve enhanced logit values ​​for unresolved classes, according to several embodiments. The process shown in Figure 11 may be implemented by software (e.g., code, instructions, programs), hardware, or a combination thereof, executed by one or more processing units (e.g., processors, cores) of each system. The software may be stored on a non-temporary storage medium (e.g., on a memory device). The methods presented in Figure 11 and described below are intended to be illustrative and non-limiting. Figure 11 shows various processing steps occurring in a particular sequence or order, but this is not intended to be limiting. In some embodiments, these steps may be executed in some different order, or some steps may be executed in parallel. In some embodiments, such as the embodiments shown in Figures 1 to 4, the process shown in Figure 11 may be executed by a training subsystem to map probability values ​​corresponding to unresolved classes to enhanced logit values ​​of one or more machine learning models (e.g., intent classifiers 242 or 320 or machine learning model 425). The training subsystem can be part of a data processing system (for example, the chatbot system 400 described in relation to Figure 4) or a component of another system configured to train and deploy machine learning models.

[0186] In 1105, the training subsystem receives a training dataset. The training dataset may include a set of utterances or messages. Each utterance in the set is associated with a training label, which identifies the predicted intent class of the utterance. In some cases, an utterance may include text data converted from speech input (e.g., speech utterances), where the speech input may be converted into a text-form utterance in a particular language, and then that text utterance may be processed. The training dataset may also include utterances or messages that are out of scope from a particular task configured to be performed by the corresponding skill or chatbot. For example, if a skillbot is configured to process food orders for a pizza restaurant, an out-of-scope utterance might be a message inquiring about the weather. In some examples, each out-of-scope utterance or message in the training dataset is associated with a training label, which identifies the class that the utterance or message cannot be resolved to. To identify.

[0187] In 1110, the training subsystem initializes the machine learning model. The machine learning model may include a series of network layers, the final network layer of which includes a logit function that converts a first probability for solvable classes into a first real number representing a first logit value, and a second probability for unsolvable classes into a second real number representing a second logit value. Thus, the logit function can be the logarithm of odds that converts the probability values ​​for each class into real numbers that fit within a probability distribution (e.g., a logit-normal distribution). In some examples, the logit function is weighted by the centroid of the distribution associated with each class.

[0188] Machine learning models can be convolutional neural networks ("CNNs"), such as inception neural networks, residual neural networks ("ResNets"), or recurrent neural networks, such as long short-term memory ("LSTM") models or gated recurrent unit ("GRU") models, or other variations of deep neural networks ("DNNs") (for example, a multilevel n-binary DNN classifier or a multiclass DNN classifier for single-intent classification). Machine learning models can also be naive Bayes classifiers, linear classifiers, support vector machines, bagging models such as random forest models, boosting models, shallow neural networks, or any other suitable ML model trained for natural language processing, such as CNN-HMM or MCNN (multiscale convolutional neural network).

[0189] Initializing a machine learning model can include defining the number of layers, the type of each layer (e.g., fully connected, convolutional neural network), and the type of activation function for each layer (e.g., sigmoid, ReLU, softmax).

[0190] In 1115, the training subsystem uses the training dataset to train a machine learning model to predict whether an utterance or message represents a task configured for SkillBot to perform, or to match an utterance or message to an intent associated with SkillBot. In some examples, the machine learning model is trained to determine a first probability for resolvable classes and a second probability for unresolvable classes. The first probability for resolvable classes may be generated by a first output channel of the final network layer of the machine learning model. The second probability for unresolvable classes may be generated by a second output channel of the final network layer of the machine learning model. In some examples, the machine learning model is trained to determine each of two or more resolvable classes (e.g., order_pizza, cancel_pizza) Find the probability of this happening.

[0191] One or more parameters of a machine learning model can be trained to minimize the loss between the predicted output generated by the machine learning model and the expected output indicated by the training label of the corresponding utterance. Training of the machine learning model can be performed until the loss reaches a minimum error threshold. In particular, the machine learning model can be trained using an augmented training dataset so that it can generate probabilities of whether a given utterance corresponds to an unresolved intent.

[0192] In step 1120, the training subsystem replaces the logit function associated with unsolvable classes with enhanced logit values. For resolvable classes, the logit function can continue to be used. Thus, the trained machine learning model applies the logit function to the probabilities of resolvable classes to produce the corresponding logit values. It is possible.

[0193] For unsolvable classes, the logit function is replaced with an enhanced logit value. In some examples, the enhanced logit value is a different real number obtained independently of the logit function used for solvable classes. The enhanced logit value may include one of the following: (i) a statistical value obtained based on a set of logit values ​​generated from the training dataset; (ii) a bounded value selected from a range of values ​​defined by the logarithm of a first odds corresponding to the probability for unsolvable classes, where the logarithm of the first odds is constrained to a range of values ​​by a bounding function and weighted by the centroid of the distribution associated with the unsolvable classes; (iii) a weighted value generated by the logarithm of a second odds corresponding to the probability for unsolvable classes, where the logarithm of the second odds is constrained to a range of values ​​by a bounding function, scaled by a scaling coefficient and weighted by the centroid of the distribution associated with the unsolvable classes; (iv) a hyperparameter-optimized value generated based on hyperparameter tuning of the machine learning model; or (v) a learned value adjusted during training of the machine learning model.

[0194] In step 1125, a trained machine learning model with enhanced logit values ​​may be deployed within the chatbot system (for example, as part of a skillbot) to predict whether an utterance or message represents a task that the skillbot is configured to perform, or whether an utterance or message matches an intent associated with the skillbot, or whether an utterance or message corresponds to a class that cannot be resolved. The process then terminates in step 1100.

[0195] F. Process for classifying out-of-scope utterances using enhanced logit values Figure 12 is a flowchart of process 1200 for using enhanced logit values ​​to classify utterances as unresolved classes, according to several embodiments. The process shown in Figure 12 may be implemented by software (e.g., code, instructions, programs), hardware, or a combination thereof, executed by one or more processing units (e.g., processors, cores) of each system. The software may be stored on a non-temporary storage medium (e.g., on a memory device). The methods presented in Figure 12 and described below are intended to be illustrative and non-limiting. Figure 12 shows various processing steps occurring in a particular sequence or order, but this is not intended to be limiting. In some embodiments, those steps may be executed in some different order, or some steps may be executed in parallel. In some embodiments, such as the embodiments shown in Figures 1 to 4, the process shown in Figure 12 may be performed by a chatbot or skillbot system that can implement a machine learning model and enhanced logit values ​​to predict whether an utterance corresponds to an unresolved class. In this particular exemplary process, the chatbot system performs the classification.

[0196] In 1205, the chatbot system receives utterances generated by a user interacting with the chatbot system. Utterances may refer to a set of words (e.g., one or more sentences) exchanged during a conversation with the chatbot. In some cases, utterances may include text data converted from speech input (e.g., speech utterances), where the speech input may be converted into text utterances in a particular language, and then the text utterances may be processed.

[0197] In 1210, the chatbot system inputs utterances into a machine learning model. The machine learning model may include a series of network layers, with the final network being the final layer of the series of layers. The sheath includes a logit function that converts the first probability for a solvable class into a first real number representing the first logit value, and the second probability for an unsolvable class into a second real number representing the second logit value.

[0198] The machine learning model can perform actions 1215, 1220, and 1225 to predict whether an utterance corresponds to an unresolved class. In 1215, the machine learning model calculates a first probability for a resolvable class and a second probability for an unresolved class. The first probability for a resolvable class may be generated by the first output channel of the final network layer of the machine learning model. The second probability for an unresolved class may be generated by the second output channel of the final network layer of the machine learning model. In some examples, the machine learning model calculates probabilities for each of two or more resolvable classes (e.g., order_pizza, cancel_pizza).

[0199] In 1220, the machine learning model uses a logit function to map the first probability for a resolvable class to a first logit value. The logit function can be the logarithm of the odds corresponding to the first probability for a resolvable class. The logarithm of the first odds can be weighted by the centroid of the distribution associated with the resolvable class.

[0200] In 1225, the machine learning model maps a second probability for unsolvable classes to a second logit value, which is determined independently of the logit function used for solvable classes. In some examples, the second logit value may include (i) a statistical value determined based on a set of logit values ​​generated from the training dataset, (ii) a bounded value selected from a range of values ​​defined by the logarithm of the first odds corresponding to the probability for unsolvable classes, where the logarithm of the first odds is constrained to a range of values ​​by a bounding function and weighted by the centroid of the distribution associated with the unsolvable classes, (iii) a weighted value generated by the logarithm of the second odds corresponding to the probability for unsolvable classes, where the logarithm of the second odds is constrained to a range of values ​​by a bounding function, scaled by a scaling coefficient and weighted by the centroid of the distribution associated with the unsolvable classes, (iv) a hyperparameter-optimized value generated based on hyperparameter tuning of the machine learning model, or (v) a learned value adjusted during training of the machine learning model. The first and second logit values ​​are returned to the chatbot system for classification.

[0201] In 1230, the chatbot system classifies utterances as either resolvable or unresolvable based on the first and second logit values. In some examples, the activation function of the machine learning model (e.g., the softmax function) is (i) (ii) Applying the first logit value, the probability that the utterance belongs to a class that can be resolved within the multinomial distribution is calculated. (ii) Applying the second logit value, the probability that the utterance belongs to a class that cannot be resolved within the multinomial distribution is calculated. Based on the calculated probabilities, the chatbot system can classify the utterance. After that, process 1200 ends.

[0202] G. Exemplary System Figure 13 shows a simplified diagram of a distributed system 1300. In the illustrated example, the distributed system 1300 includes one or more client computing devices 1302, 1304, 1306, and 1308 connected to a server 1312 via one or more communication networks 1310. The client computing devices 1302, 1304, 1306, and 1308 may be configured to run one or more applications.

[0203] In various examples, Server 1312 may be one or more embodiments of the present disclosure. It can be adapted to run one or more services or software applications that enable this. In one example, server 1312 may also provide other services or software applications that may include non-virtual and virtual environments. In some examples, these services may be web-based or cloud-based services, as under the Software as a Service (SaaS) model, for users of client computing devices 1302, 1304, 1306 and / or 1308. These may be provided as a service. Users operating client computing devices 1302, 1304, 1306, and / or 1308 may access the services provided by these components by interacting with server 1312 using one or more client applications.

[0204] In the configuration shown in Figure 13, server 1312 may include one or more components 1318, 1320, and 1322 that implement the functions performed by server 1312. These components may include one or more processors, hardware components, or software components that can be executed by a combination thereof. It should be recognized that a wide variety of system configurations are possible, which may differ from the distributed system 1300. Therefore, the example shown in Figure 13 is an example of a distributed system for implementing the example system, and is not intended to be limiting.

[0205] A user may use client computing devices 1302, 1304, 1306, and / or 1308 to run one or more applications, models, or chatbots, which may generate one or more events or models, which may then be implemented or processed in accordance with the teachings of this disclosure. A client device may provide an interface that allows a user of the client device to interact with the client device. A client device may also output information to the user through this interface. Although Figure 13 shows only four client computing devices, any number of client computing devices may be supported.

[0206] Client devices may include various types of computing systems, such as portable handheld devices, general-purpose computers like personal computers and laptops, workstation computers, wearable devices, game systems, thin clients, various messaging devices, sensors or other sensing devices. These computing devices may have various types and versions of software applications and operating systems (e.g., Microsoft Windows®, Apple Macintosh®, UNIX® or UNIX-like operating systems, Linux® or Linux-like operating systems), various mobile operating systems (e.g., Microsoft Windows Mobile®, iOS®, Windows Phone®, Android®) This may include Google Chrome OS, including BlackBerry®, Palm OS®, and BlackBerry®. Portable handheld devices include cellular phones, smartphones, etc. Examples include iPhone (registered trademark), tablets (for example, iPad (registered trademark)), and personal digital assistants (PDAs). This may include, for example, wearable devices may include Google Glass® head-mounted displays and other devices. Game systems may include various handheld game devices and internet-connected game devices (e.g., Microsoft Xbox® game consoles with or without Kinect® gesture input devices, Sony PlayStation® systems, and various game systems provided by Nintendo®). Client devices may be capable of running a wide variety of applications, such as various internet-related applications and communication applications (e.g., email applications, short message service (SMS) applications). Various communication protocols may be used.

[0207] Network 1310 may be any type of network known to those skilled in the art, capable of supporting data communication using any of the various available protocols, including but not limited to TCP / IP (Transmission Control Protocol / Internet Protocol), SNA (System Network Architecture), IPX (Internet Packet Switching), AppleTalk®, etc. This is merely an example. Network 1310 includes local area networks (LANs), Ethernet®-based networks, Token Ring, wide area networks (WANs), the Internet, virtual networks, virtual private networks (VPNs), intranets, extranets, public switched telephone networks (PSTNs), infrared networks, wireless networks (for example, wireless networks operating under any of the IEEE 1002.11 protocol suites, Bluetooth®, and / or This may include any other radio protocol, and / or any combination of these and / or other networks.

[0208] Server 1312 may consist of one or more general-purpose computers, dedicated server computers (including, for example, PC (personal computer) servers, UNIX® servers, midrange servers, mainframe computers, rack-mount servers, etc.), server farms, server clusters, or other appropriate configurations and / or combinations. Server 1312 may include one or more virtual machines running a virtual operating system, or other computing architectures with virtualization. This could be, for example, one or more flexible pools of logical storage that can be virtualized to maintain virtual storage for the server. In various examples, Server 1312 may be adapted to run one or more services or software applications that provide the functions described above.

[0209] The computing system within server 1312 may run one or more operating systems, including any of the above operating systems, and commercially available server operating systems. Server 1312 may also run any of a variety of other server applications and / or middle-tier applications, including HTTP (Hypertext Transfer Protocol) servers, FTP (File Transfer Protocol) servers, CGI (Common Gateway Interface) servers, JAVA® servers, and database servers. Illustrative database servers include, but are not limited to, those commercially available from Oracle®, Microsoft®, Sybase®, IBM® (International Business Machines), and others. It will not be done.

[0210] In some implementations, server 1312 may include one or more applications for analyzing and organizing data feeds and / or event updates received from users of client computing devices 1302, 1304, 1306, and 1308. For example, data feeds and / or event updates may include, but are not limited to, Twitter® feeds, Facebook® updates, or real-time updates received from one or more third-party sources and continuous data streams, including real-time events related to sensor data applications, financial stock market boards, network performance measurement tools (e.g., network monitoring and traffic management applications), clickstream analysis tools, and automotive traffic monitoring. Server 1312 displays data feeds and / or real-time events on one or more display devices of client computing devices 1302, 1304, 1306, and 1308. This may also include one or more applications for displaying information via it.

[0211] The distributed system 1300 may also include one or more data repositories 1314, 1316. In a particular example, these data repositories can be used to store data and other information. For example, one or more of the data repositories 1314, 1316 can be used to store information such as information related to chatbot performance or generated models for use by the chatbot used by the server 1312 when performing various functions according to various embodiments. The data repositories 1314, 1316 may reside in various locations. For example, the data repository used by the server 1312 may be located locally with the server 1312, or it may be located remotely from the server 1312 and communicates with the server 1312 via a network-based connection or a dedicated connection. The data repositories 1314, 1316 may be of different types. In a particular example, the data repository used by the server 1312 may be a database, such as a relational database, like a database provided by Oracle Corporation® and other manufacturers. One or more of these databases may be adapted to allow data to be stored, updated, and retrieved from the database in response to commands in SQL format.

[0212] In a particular example, one or more of the data repositories 1314 and 1316 may be used by the application to store application data. The data repositories used by the application may be of various types, such as a key-value store repository, an object store repository, or a general-purpose storage repository supported by the file system.

[0213] In a particular example, the functions described herein may be provided as services through a cloud environment. Figure 14 is a simplified block diagram of a cloud-based system environment that may provide various services as cloud services in a particular example. In the example shown in Figure 14, the cloud infrastructure system 1402 may provide one or more cloud services that a user may request using one or more client computing devices 1404, 1406, and 1408. The cloud infrastructure system 1402 may include one or more computers and / or servers, which may include those described above with respect to server 1312. The computers within the cloud infrastructure system 1402 may be organized as general-purpose computers, dedicated server computers, server farms, server clusters, or any other appropriate arrangement and / or combination.

[0214] Network 1410 can facilitate data communication and exchange between clients 1404, 1406, and 1408 and the cloud infrastructure system 1402. Network 1410 may include one or more networks. The networks may be of the same type or different types. Network 1410 may support one or more communication protocols, including wired and / or wireless protocols, to facilitate communication.

[0215] The example shown in Figure 14 is merely one example of a cloud infrastructure system and is not intended to be limiting. It should be understood that in several other examples, the cloud infrastructure system 1402 may have more or fewer components than shown in Figure 14, may combine two or more components, or may have components in different configurations or arrangements. For example, while Figure 14 shows three client computing devices, alternative examples may include any number of client computing devices. S may be supported.

[0216] The term "cloud service" is generally used to refer to services made available to users on demand via communication networks such as the Internet, through a service provider's system (e.g., cloud infrastructure system 1402). Typically, in a public cloud environment, the servers and systems that make up the cloud service provider's system are different from the customer's own on-premises servers and systems. The cloud service provider's system is managed by the cloud service provider. Therefore, customers can use the cloud services provided by the cloud service provider without having to purchase separate licenses, support, or hardware and software resources for the service. For example, the cloud service provider's system can host applications, and users can order and use applications on demand via the Internet without having to purchase infrastructure resources to run the applications. Cloud services are designed to provide easy and scalable access to applications, resources, and services. Several providers offer cloud services. For example, some cloud services, such as middleware services, database services, and Java® cloud services, are offered by Oracle Corporation® in Redwood Shores, California.

[0217] In a particular example, the cloud infrastructure system 1402 may provide one or more cloud services using various models, including a hybrid service model, such as a Software as a Service (SaaS) model, a Platform as a Service (PaaS) model, and an Infrastructure as a Service (IaaS) model. The cloud infrastructure system 1402 may include a suite of applications, middleware, databases, and other resources that enable the provisioning of various cloud services.

[0218] The SaaS model enables the delivery of applications or software as a service to customers over a communication network such as the internet, without requiring customers to purchase the underlying hardware or software for the application. For example, the SaaS model can be used to allow customers to access on-demand applications hosted on a cloud infrastructure system 1402. Examples of SaaS services offered by Oracle Corporation® include, but are not limited to, various services for human resource / capital management, customer relationship management (CRM), enterprise resource planning (ERP), supply chain management (SCM), enterprise performance management (EPM), analytics services, and social applications.

[0219] The IaaS model is generally used to provide flexible computing and storage capabilities by offering infrastructure resources (such as servers, storage, hardware, and networking resources) to customers as cloud services. Various IaaS services are offered by Oracle Corporation®.

[0220] The PaaS model is generally used to provide a platform and environmental resources as a service, enabling customers to develop, run, and manage applications and services without having to procure, build, or manage those resources themselves. Examples of PaaS services offered by Oracle Corporation (registered trademark) include, but are not limited to, Oracle Java Cloud Service (JCS), Oracle Database Cloud Service (DBCS), data management cloud services, and various application development solution services. stomach.

[0221] Cloud services are generally delivered in an on-demand, self-service, subscription-based, flexibly scalable, reliable, highly available, and secure manner. For example, a customer may order one or more services provided by the cloud infrastructure system 1402 through a subscription order. The cloud infrastructure system 1402 then provides the services requested in the customer's subscription order by performing processing. For example, a user may use utterances to request the cloud infrastructure system to take a specific action (e.g., an intent) as described above, and / or to provide services for a chatbot system as described herein. The cloud infrastructure system 1402 may be configured to provide one cloud service or more cloud services.

[0222] The cloud infrastructure system 1402 can provide cloud services through various deployment models. In a public cloud model, the cloud infrastructure system 1402 may be owned by a third-party cloud service provider, and the cloud services are provided to a general public customer. This customer may be an individual or a company. In another example, under a private cloud model, the cloud infrastructure system 1402 may function within an organization (for example, within a corporate organization), and the services are provided to customers within this organization. For example, this customer may be various departments within the company, such as the human resources department or the payroll department, or an individual within the company. In yet another example, under a community cloud model, the cloud infrastructure system 1402 and the services provided may be shared among various organizations within the relevant community. Other models, such as hybrid models of the above models, may also be used.

[0223] The client computing devices 1404, 1406, and 1408 may be of different types (for example, the client computing devices 1302, 1304, 1306, and 1308 shown in Figure 13) and may be capable of operating one or more client applications. By using the client devices, users can interact with the cloud infrastructure system 1402, such as requesting services provided by the cloud infrastructure system 1402. For example, a user may use a client device to request information or actions from a chatbot, as described in this disclosure.

[0224] In some examples, the processes that the cloud infrastructure system 1402 executes to provide services may include model training and deployment. This analysis may include training and deploying one or more models by using, analyzing, and processing a dataset. This analysis may be performed by one or more processors, which may, in some cases, process data in parallel and execute simulations using the data. For example, big data analysis may be performed by the cloud infrastructure system 1402 to generate and train one or more models for a chatbot system. The data used in this analysis may include structured data (e.g., data stored in a database or data structured according to a structured model) and / or unstructured data (e.g., a data blob (binary large object)).

[0225] As shown in the example of FIG. 14, the cloud infrastructure system 1402 may include infrastructure resources 1430 utilized to facilitate the provision of various cloud services provided by the cloud infrastructure system 1402. The infrastructure resources 1430 may include, for example, processing resources, storage or memory resources, networking resources, and the like. In a particular example, a storage virtual machine available to process storage requested from an application may be part of the cloud infrastructure system 1402. In other examples, the storage virtual machine may be part of a different system.

[0226] ​In certain examples, to facilitate the efficient provisioning of these resources for the various cloud services provided by the cloud infrastructure system 1402 to different customers, the resources may be grouped into sets of resources or resource modules (also referred to as "pods"). Each resource module or pod may include a pre-integrated and optimized combination of one or more types of resources. In certain examples, different pods may be pre-provisioned for different types of cloud services. For example, a first set of pods may be provisioned for database services, and a second set of pods, which may include a different combination of resources than the pods in the first set of pods, may be provisioned for Java services or the like. For some services, the resources allocated to provision these services may be shared among the services.

[0227] The cloud infrastructure system 1402 itself may internally use a service 1432 that is shared by different components of the cloud infrastructure system 1402 and that facilitates the provisioning of services by the cloud infrastructure system 1402. These internal shared services may include, but are not limited to, security identity services, integration services, enterprise repository services, enterprise manager services, virus scan whitelist services, high availability, backup recovery services, services that enable cloud support, email services, notification services, file transfer services, and the like.

[0228] The cloud infrastructure system 1402 may include multiple subsystems. These subsystems may be implemented in software, hardware, or a combination thereof. As shown in Figure 14, the subsystems may include a user interface subsystem 1412 that enables users or customers of the cloud infrastructure system 1402 to interact with the cloud infrastructure system 1402. The user interface subsystem 1412 may include various different interfaces, such as a web interface 1414, an online store interface 1416 where cloud services offered by the cloud infrastructure system 1402 are advertised and available for purchase by consumers, and other interfaces 1418. For example, a customer may use a client device to request (service request 1434) one or more services offered by the cloud infrastructure system 1402 using one or more of interfaces 1414, 1416, and 1418. For example, a customer may access the online store, browse the cloud services offered by the cloud infrastructure system 1402, and place a subscription order for one or more services offered by the cloud infrastructure system 1402 that the customer wishes to subscribe to. This service request may include information identifying the customer and one or more services that the customer wishes to request. For example, a customer may submit a request order for services provided by the cloud infrastructure system 1402. As part of the order, the customer may provide information identifying the chatbot system on which the service will be provided, and optionally one or more credentials for the chatbot system.

[0229] In a specific example, such as the example shown in Figure 14, the cloud infrastructure system 1402 may include an order management subsystem (OMS) 1420 configured to process new orders. As part of this process, the OMS 1420 may be configured to prepare the order for provisioning by creating a customer account if one does not already exist, receiving billing and / or account information from the customer to be used to charge the customer in order to provide the requested services to the customer, verifying the customer information, reserving the order for the customer after verification, and coordinating various workflows.

[0230] If properly validated, OMS1420 may invoke Order Provisioning Subsystem (OPS)1424, configured to provision resources for this order, including processing, memory, and networking resources. Provisioning may include allocating resources for the order and configuring those resources to facilitate the services requested by the customer order. The way resources are provisioned for an order and the types of resources provisioned may depend on the type of cloud service ordered by the customer. For example, following a certain workflow, OPS1424 may be configured to determine the specific cloud service being requested and identify the number of pods that would have been pre-configured for that particular cloud service. The number of pods allocated for an order may depend on the size / volume / level / scope of the requested service. For example, the number of pods to allocate may be determined based on the number of users the service should support, the duration for which the service is requested, etc. The allocated pods may then be customized to suit the specific customer making the request in order to provide the requested service.

[0231] In certain cases, the setup phase process may be performed by the cloud infrastructure system 1402 as part of the provisioning process, as described above. The cloud infrastructure system 1402 can generate an application ID and select a storage virtual machine for the application from among the storage virtual machines provided by the cloud infrastructure system 1402 itself, or from storage virtual machines provided by other systems other than the cloud infrastructure system 1402.

[0232] The cloud infrastructure system 1402 may send a response or notification 1444 to the requesting customer to indicate when the requested service will be available. In some examples, the customer may be sent information (e.g., a link) that enables the customer to begin using and utilizing the benefits of the requested service. In certain examples, the response to the customer requesting the service may include a chatbot system ID generated by the cloud infrastructure system 1402, and information identifying the chatbot system selected by the cloud infrastructure system 1402 for the chatbot system corresponding to the chatbot system ID.

[0233] The cloud infrastructure system 1402 may provide services to multiple customers. For each customer, the cloud infrastructure system 1402 manages information related to one or more subscription orders received from the customer, maintains customer data related to the orders, and provides the requested services to the customer. The cloud infrastructure system 1402 may also collect usage statistics about the customer's use of the subscribed services. For example, statistics may be collected on the amount of storage used, the amount of data transferred, the number of users, and the amount of system uptime and system downtime. This usage information may be used to charge customers. Billing may be done, for example, on a monthly basis.

[0234] The cloud infrastructure system 1402 may provide services to multiple customers in parallel. The cloud infrastructure system 1402 may store information about these customers, which may include copyright information. In a particular example, the cloud infrastructure system 1402 includes an Identity Management Subsystem (IMS) 1428 configured to manage customer information and separate the managed information so that information about one customer cannot be accessed from information about another customer. The IMS 1428 may be configured to provide various security-related services, such as identity services including information access management, authentication and authorization services, and services for managing customer identities and roles and related capabilities.

[0235] Figure 15 shows an example of computer system 1500. In some examples, computer system 1500 could be used to implement any digital assistant or chatbot system in a distributed environment, as well as the various servers and computer systems described above. As shown in Figure 15, computer system 1500 includes various subsystems, including a processing subsystem 1504 that communicates with several other subsystems via a bus subsystem 1502. These other subsystems may include a processing acceleration unit 1506, an I / O subsystem 1508, a storage subsystem 1518, and a communication subsystem 1524. The storage subsystem 1518 may include non-temporary computer-readable storage media, including a storage medium 1522 and system memory 1510.

[0236] The bus subsystem 1502 provides a mechanism for various components and subsystems of the computer system 1500 to communicate with each other as intended. Although the bus subsystem 1502 is schematically shown as a single bus, alternative examples of the bus subsystem may utilize multiple buses. The bus subsystem 1502 may be one of several types of bus structures, including a memory bus or memory controller, peripheral bus, local bus, etc., using one of various bus architectures. For example, such architectures may include the Industry Standard Architecture (ISA) bus, Micro Channel Architecture (MCA) bus, Enhanced ISA (EISA) bus, Video Electronics Standards Association (VESA) local bus, and IEEE P1386.1 standard. This could include peripheral component interconnect (PCI) buses, which can be implemented as mezzanine buses manufactured in this manner.

[0237] The processing subsystem 1504 controls the operation of the computer system 1500 and may include one or more processors, application-specific integrated circuits (ASICs), or field-programmable gate arrays (FPGAs). The processors may include single-core or multi-core processors. The processing resources of the computer system 1500 can be organized into one or more processing units 1532, 1534, etc. A processing unit may include one or more processors, one or more cores from the same or different processors, a combination of cores and processors, or other combinations of cores and processors. In some examples, the processing subsystem 1504 may include one or more dedicated coprocessors, such as graphics processors or digital signal processors (DSPs). In some examples, some or all of the processing units in the processing subsystem 1504 may use customized circuitry, such as application-specific integrated circuits (ASICs) or field-programmable gate arrays (FPGAs).

[0238] In some examples, processing units within the processing subsystem 1504 may execute instructions stored in system memory 1510 or computer-readable storage medium 1522. In various examples, processing units may execute various program or code instructions and maintain multiple programs or processes running concurrently. At any given time, some or all of the program code to be executed may reside in system memory 1510 and / or computer-readable storage medium 1522, which may contain one or more storage devices. Through appropriate programming, the processing subsystem 1504 may provide the various functions described above. In an example where the computer system 1500 is running one or more virtual machines, one or more processing units may be assigned to each virtual machine.

[0239] In a particular example, a processing acceleration unit 1506 may be optionally provided to perform customized processing to accelerate the overall processing performed by the computer system 1500, or to offload a portion of the processing performed by the processing subsystem 1504.

[0240] The I / O subsystem 1508 may include devices and mechanisms for inputting information into and / or outputting information from or through the computer system 1500. Generally, the use of the term “input device” is intended to include all conceivable types of devices and mechanisms for inputting information into the computer system 1500. User interface input devices may include, for example, pointing devices such as keyboards, mice or trackballs, touchpads or touchscreens integrated into displays, scroll wheels, click wheels, dials, buttons, switches, keypads, voice input devices with voice command recognition systems, microphones, and other types of input devices. User interface input devices may also include motion sensing and / or gesture recognition devices, such as Microsoft Kinect® motion sensors, Microsoft Xbox® 360 game controllers, and devices that provide interfaces for receiving input using gestures and voice commands, enabling users to control and interact with the input device. The user interface input device may also include an eye gesture recognition device, such as the Google Glass® blink detector, which detects eye movements from the user (e.g., blinking while taking a picture and / or making a menu selection) and translates the eye gestures into input to an input device (e.g., Google Glass®). The user interface input device may also include a voice recognition sensing device that enables the user to interact with a voice recognition system (e.g., Siri® Navigator) via voice commands.

[0241] Other examples of user interface input devices may include, but are not limited to, three-dimensional (3D) mice, joysticks or pointing sticks, gamepads and graphic tablets, as well as auditory / visual devices such as speakers, digital cameras, digital camcorders, portable media players, webcams, image scanners, fingerprint scanners, barcode readers, 3D scanners, 3D printers, laser rangefinders, and eye-tracking devices. User interface input devices may also include medical imaging input devices such as computed tomography, magnetic resonance imaging, positional emission tomography, and medical ultrasound devices. User interface input devices may also include audio input devices such as MIDI keyboards and digital musical instruments.

[0242] Generally, the use of the term "output device" is from computer systems 1500 to the user. It is intended to include all conceivable types of devices and mechanisms for outputting information to other computers. User interface output devices may include non-visual displays such as display subsystems, indicator lights, or audio output devices. Display subsystems may include flat panel devices such as those using cathode ray tubes (CRTs), liquid crystal displays (LCDs), or plasma displays, planning devices, touchscreens, etc. For example, user interface output devices may include, but are not limited to, a variety of display devices that visually convey text, graphics, and audio / video information, such as monitors, printers, speakers, headphones, car navigation systems, plotters, audio output devices, and modems.

[0243] The storage subsystem 1518 provides a repository or datastore for storing information and data used by the computer system 1500. The storage subsystem 1518 provides a tangible, non-temporary, computer-readable storage medium for storing basic programming and data configurations that provide some example functionality. Software (e.g., programs, code modules, instructions) that, when executed by the processing subsystem 1504, provides the above-described functionality may be stored in the storage subsystem 1518. The software may be executed by one or more processing units of the processing subsystem 1504. The storage subsystem 1518 may also provide authentication in accordance with the teachings of this disclosure.

[0244] The storage subsystem 1518 may include one or more non-temporary memory devices, including volatile and non-volatile memory devices. As shown in Figure 15, the storage subsystem 1518 includes system memory 1510 and computer-readable storage medium 1522. The system memory 1510 may include several memories, including volatile primary random access memory (RAM) for storing instructions and data during program execution, and non-volatile read-only memory (ROM) or flash memory for storing fixed instructions. In some implementations, a basic input / output system (BIOS), which includes basic routines that help transfer information between elements within the computer system 1500 during startup, etc., may typically be stored in ROM. Typically, RAM contains data and / or program modules currently being operated and executed by the processing subsystem 1504. In some implementations, system memory 1510 may include several different types of memory, such as static random access memory (SRAM) and dynamic random access memory (DRAM).

[0245] As an example, without limitation, as shown in FIG. 15, system memory 1510 may load an application program 1512, program data 1514, and an operating system 1516 that may include various applications such as a web browser, middleware applications, a relational database management system (RDBMS), etc. As an example, the operating system 1516 may be Microsoft Windows (registered trademark), Apple Macintosh (registered trademark), and / or Linux operating systems, various commercially available UNIX (registered trademark) or UNIX-based operating systems (including but not limited to various GNU / Linux operating systems, Google Chrome (registered trademark) OS, etc.), and / or, iOS (registered trademark ), Windows Phone, Android (registered trademark) OS, BlackBerry (registered trademark) OS, various versions of mobile operating systems such as Palm (registered trademark) OS operating system, etc.

[0246] Computer-readable storage medium 1522 provides programming for several example functions and data structures can be stored. The computer-readable storage medium 1522 can provide storage for computer-readable instructions, data structures, program modules, and other data for the computer system 1500. Software (programs, code modules, instructions) that provides the above functions when executed by the processing subsystem 1504 may be stored in the storage subsystem 1518. As an example, the computer-readable storage medium 1522 may include non-volatile memory such as a hard disk drive, magnetic disk drive, optical disk drive such as a CD-ROM, DVD, Blu-ray® disc, or other optical media. The computer-readable storage medium 1522 may include, but is not limited to, a Zip® drive, flash memory card, Universal Serial Bus (USB) flash drive, Secure Digital (SD) card, DVD disc, digital videotape, etc. Computer-readable storage media 1522 may also include solid-state drives (SSDs) based on non-volatile memory such as flash memory-based SSDs, enterprise flash drives, and solid-state ROMs; SSDs based on volatile memory such as solid-state RAM, dynamic RAM, and static RAM; DRAM-based SSDs; magnetoresistive RAM (MRAM) SSDs; and hybrid SSDs that use a combination of DRAM and flash memory-based SSDs.

[0247] In certain examples, the storage subsystem 1518 may also include a computer-readable storage medium reader 1520 that can be further connected to the computer-readable storage medium 1522. The reader 1520 may be configured to receive and read data from memory devices such as disks, flash drives, etc.

[0248] In certain examples, computer system 1500 may support virtualization technologies, including but not limited to virtualization of processing and memory resources. For example, computer system 1500 may provide support for running one or more virtual machines. In certain examples, computer system 1500 may run programs such as hypervisors that facilitate the configuration and management of virtual machines. Each virtual machine may be allocated memory, computing resources (e.g., processors, cores), I / O, and networking resources. Each virtual machine typically runs independently of other virtual machines. A virtual machine typically runs its own operating system, which may be the same as or different from the operating system run by other virtual machines run by computer system 1500. Thus, potentially multiple operating systems may run simultaneously by computer system 1500.

[0249] The communication subsystem 1524 provides interfaces to other computer systems and networks. It functions as an interface for sending and receiving data between other systems and the computer system 1500. For example, the communication subsystem 1524 may enable the computer system 1500 to establish communication channels to one or more client devices via the internet in order to send and receive information with one or more client devices. For instance, if the computer system 1500 is used to implement the bot system 120 shown in Figure 1, the communication subsystem may be used to communicate with a chatbot system selected for the application.

[0250] The communication subsystem 1524 may support both wired and / or wireless communication protocols. In one example, the communication subsystem 1524 may support wireless voice and wireless communication (for example, using cellular telephone technology, 3G, 4G or EDGE (High Speed ​​Data Rate for Global Evolution) or other advanced data network technologies, WiFi (IEEE 802.XX family standards, or other mobile communication technologies, or a combination thereof)). It may also include radio frequency (RF) transceiver components for accessing a data network, a global positioning system (GPS) receiver component, and / or other components. In some examples, the communications subsystem 1524 may provide a wired network connection (e.g., Ethernet®) in addition to or instead of a wireless interface.

[0251] The communication subsystem 1524 can receive and transmit data in various formats. In some examples, in addition to other formats, the communication subsystem 1524 may receive input communications in the form of structured data feeds and / or unstructured data feeds 1526, event streams 1528, event updates 1530, etc. For example, the communication subsystem 1524 may receive social media networks and / or Twitter (registration Trademark feed, Facebook® registered trademark updates, Rich Site Summary (RSS) feed It may be configured to receive (or send) data feeds 1526 in real time from users of other communication services, such as web feeds and / or real-time updates from one or more third-party sources.

[0252] In certain examples, the communication subsystem 1524 may be configured to receive data in the form of a continuous data stream, which may include an event stream 1528 and / or event update 1530 of real-time events that are inherently continuous or infinite and do not have a clear end. Examples of applications that generate continuous data include, for example, sensor data applications, financial stock market boards, network performance measurement tools (e.g., network monitoring and traffic management applications), clickstream analysis tools, and automotive traffic monitoring.

[0253] The communication subsystem 1524 may be configured to transmit data from the computer system 1500 to other computer systems or networks. This data may be transmitted in various different formats, such as structured and / or unstructured data feeds 1526, event streams 1528, and event updates 1530, to one or more databases that can communicate with one or more streaming data source computers connected to the computer system 1500.

[0254] Computer system 1500 may be one of many types, including handheld portable devices (e.g., iPhone® cellular phone, iPad® computing tablet, PDA), wearable devices (e.g., Google Glass® head-mounted display), personal computers, workstations, mainframes, kiosks, server racks, or other data processing systems. Because the nature of computers and networks is constantly changing, the description of computer system 1500 shown in Figure 15 is intended only as a specific example. Many other configurations are possible, having more or fewer components than the system shown in Figure 15. Based on the disclosures and teachings herein, it should be recognized that there are other embodiments and / or methods for realizing various examples.

[0255] While specific examples have been described, various variations, modifications, alternative configurations, and equivalents are possible. The examples are not limited to operation within a specific data processing environment, but can freely operate within multiple data processing environments. Furthermore, although the examples were illustrated using a specific set of transactions and steps, it should be apparent to those skilled in the art that this is not intended as limitation. While some flowcharts illustrate operations as sequential processes, many of these operations may be executed in parallel or concurrently. In addition, the operation... The order may be rearranged. The process may have additional steps not shown in the diagram. The various features and aspects of the above example may be used individually or together.

[0256] Furthermore, while specific examples have been described using particular combinations of hardware and software, it should be understood that other combinations of hardware and software are also possible. The specific examples may be implemented using hardware alone, software alone, or a combination thereof. The various processes described herein may be implemented on the same processor or on different processors in any combination.

[0257] Where it is stated that a device, system, component, or module is configured to perform a particular operation or function, such configuration can be achieved, for example, by designing electronic circuits to perform the operation; by programming programmable electronic circuits (such as a microprocessor) to perform the operation; by executing computer instructions or code, or any combination thereof, stored in a non-temporary memory medium, or by executing a processor or core. Processes can communicate using a variety of techniques, including but not limited to conventional techniques for inter-process communication, different pairs of processes may use different techniques, and the same pair of processes may use different techniques at different times.

[0258] This disclosure provides specific details to ensure that the examples are fully understood. However, the examples can be implemented without these specific details. For example, well-known circuits, processes, algorithms, structures, and techniques are shown without unnecessary details in order to avoid ambiguity in the examples. This specification provides only exemplary examples and is not intended to limit the scope, applicability, or configuration of other examples. Rather, the above description of the examples provides a description that enables the implementation of various examples for those skilled in the art. Various modifications are possible within the scope of the function and configuration of the elements.

[0259] Therefore, the specification and drawings should be considered illustrative rather than restrictive. However, it will be clear that additions, reductions, deletions, and other modifications and changes may be made to them without departing from the broader spirit and scope set forth in the claims. Thus, specific examples have been described, but these are not intended to be limiting. Various variations and equivalents are within the scope of the attached claims.

[0260] While the above specification illustrates aspects of the disclosure with reference to specific examples, those skilled in the art will recognize that the disclosure is not limited thereto. The various features and aspects of the above disclosure may be used individually or together. Furthermore, the examples can be used in a variety of environments and applications beyond those described herein, without departing from the broader spirit and scope of the specification. Therefore, the specification and drawings should be considered illustrative rather than restrictive.

[0261] In the above description, the method is presented in a specific order for illustrative purposes. It should be understood that in alternative examples, the method may be executed in a different order than described. It should also be understood that the above method may be executed by hardware components, or it may be embodied in a sequence of machine-executable instructions, which, when used, may cause a machine such as a general-purpose or dedicated processor or logic circuit programmed with such instructions to execute the method. These machine-executable instructions may be found on CD-ROMs or other types of optical discs, floppy disks, ROMs, RAMs, EPROMs, EEPROMs, magnetic or optical cards, flash memory, etc. The data can be stored on one or more machine-readable media, or on other types of machine-readable media suitable for storing electronic instructions. Alternatively, these methods may be implemented by a combination of hardware and software.

[0262] Where a component is described as being configured to perform a particular operation, such configuration may be achieved, for example, by designing an electronic circuit or other hardware to perform a particular operation, by programming a programmable electronic circuit (e.g., a microprocessor or other suitable electronic circuit) to perform a particular operation, or by any combination thereof.

[0263] While examples illustrating this application are described in detail here, it should be understood that the concepts of the present invention can be embodied and adopted in various other aspects, and that the claims are intended to be interpreted to include such variations, unless limited by the prior art.

Claims

1. A method performed by a computer, Receiving utterances generated by users interacting with the chatbot system, The chatbot system inputs the utterance into a machine learning model that includes a network layer containing a logit function. The machine learning model calculates a first probability for the resolvable class of the utterance and a second probability for the unresolvable class of the utterance, The machine learning model maps the first probability to the first logit value using the logit function, The machine learning model includes mapping the second probability to a second logit value, the second logit value being an enhanced logit value that predicts whether the utterance corresponds to the unresolved class, and the method further includes A computer-based method comprising classifying the chatbot system into the solvable class or the unsolvable class based on the first logit value and the second logit value.

2. The computer-based method according to claim 1, wherein the second logit value comprises (i) a statistical value obtained based on a set of logit values ​​generated from a training dataset, (ii) a bounded value selected from a range of values ​​defined by the logarithm of the first odds corresponding to the second probability, wherein the logarithm of the first odds is constrained to a range of values ​​by a bounding function and weighted by the centroid of the distribution associated with the unsolvable class, (iii) a weighted value generated by the logarithm of the second odds corresponding to the second probability, wherein the logarithm of the second odds is constrained to a range of values ​​by the bounding function, scaled by a scaling coefficient and weighted by the centroid of the distribution associated with the unsolvable class, (iv) a hyperparameter-optimized value generated based on hyperparameter tuning of the machine learning model, or (v) a learned value adjusted during training of the machine learning model.

3. The method performed by a computer according to claim 1 or 2, further comprising the chatbot system responding to the user based on the classification of the utterance as either a resolvable class or an unresolvable class.

4. The method executed by a computer according to any one of claims 1 to 3, wherein the machine learning model includes a series of network layers, and the network layer is the final network layer of the series of network layers.

5. The method performed by a computer according to any one of claims 1 to 4, wherein the logit function is configured to convert the first probability into a first real number representing the first logit value, and the second probability into a second real number representing the second logit value.

6. The method performed by a computer according to any one of claims 1 to 5, wherein the logit function is the logarithm of odds weighted by the centroid of the distribution associated with the solvable class.

7. A computer-based method according to any one of claims 1 to 6, wherein resolved classes are skills or intents within the domain and scope, and unresolved classes are skills or intents outside the domain or scope.

8. It is a system, One or more data processors, The system comprises a non-temporary computer-readable storage medium containing instructions, wherein, when the instructions are executed on one or more data processors, the one or more data processors cause the data processors to perform an operation, and the operation is Receiving utterances generated by users interacting with the chatbot system, The chatbot system inputs the utterance into a machine learning model that includes a network layer containing a logit function. The machine learning model calculates a first probability for the resolvable class of the utterance and a second probability for the unresolvable class of the utterance, The machine learning model maps the first probability to the first logit value using the logit function, The machine learning model includes mapping the second probability to a second logit value, the second logit value being an enhanced logit value that predicts whether the utterance corresponds to the unresolved class, and the operation further includes: A chatbot system comprising classifying the utterance into the solvable class or the unsolvable class based on the first logit value and the second logit value.

9. The system according to claim 8, wherein the second logit value includes (i) a statistical value obtained based on a set of logit values ​​generated from a training dataset, (ii) a bounded value selected from a range of values ​​defined by the logarithm of the first odds corresponding to the second probability, wherein the logarithm of the first odds is constrained to a range of values ​​by a bounding function and weighted by the centroid of the distribution associated with the unsolvable class, (iii) a weighted value generated by the logarithm of the second odds corresponding to the second probability, wherein the logarithm of the second odds is constrained to a range of values ​​by the bounding function, scaled by a scaling coefficient and weighted by the centroid of the distribution associated with the unsolvable class, (iv) a hyperparameter-optimized value generated based on hyperparameter tuning of the machine learning model, or (v) a learned value adjusted during training of the machine learning model.

10. The system according to claim 8 or 9, further comprising the chatbot system responding to the user based on the classification of the utterance as either a resolvable class or an unresolvable class.

11. The system according to any one of claims 8 to 10, wherein the machine learning model includes a series of network layers, and the network layer is the final network layer of the series of network layers.

12. The system according to any one of claims 8 to 11, wherein the logit function is configured to convert the first probability into a first real number representing the first logit value, and the second probability into a second real number representing the second logit value.

13. The system according to any one of claims 8 to 12, wherein the logit function is the logarithm of odds weighted by the centroid of the distribution associated with the solvable class.

14. The system according to any one of claims 8 to 13, wherein resolved classes are skills or intents within the domain and scope, and unresolved classes are skills or intents outside the domain or scope.

15. A program for causing a computer to perform the method described in any one of claims 1 to 7.

Images