Techniques for providing explanations about text classification

Abstract

The chatbot system is configured to execute code to cause the chatbot system to determine a classification result for the utterance and one or more anchors, each anchor of the one or more anchors corresponding to one or more anchor words of the utterance. For each anchor of the one or more anchors, one or more synthetic utterances are generated, and one or more classification results for the one or more synthetic utterances are determined. A report including a representation of a particular anchor of the one or more anchors is generated by the chatbot system, the particular anchor corresponding to the highest confidence value among the one or more anchors. The one or more synthetic utterances may be used to generate a new training dataset for training a machine learning model. The training dataset may be refined according to a threshold confidence value to filter out datasets for training.

Description

Technical Field

[0001] Reference to Related Applications This application claims the benefit and priority under 35 U.S.C. § 119(e) of U.S. Provisional Application No. 63 / 068,608, filed on August 21, 2020, and U.S. Provisional Application No. 63 / 174,336, filed on April 13, 2021. The entire contents of the above applications are hereby incorporated by reference in their entirety for all purposes.

[0002] Field The present disclosure generally relates to chatbot systems, and more particularly, to techniques for providing an explanation for text classification.

Background Art

[0003] Background Many users around the world are on instant messaging or chat platforms to get instant responses. Organizations often engage in live conversations with customers (or end users) using these instant messaging or chat platforms. However, hiring service personnel to engage in live communication with customers or end users can be very costly for organizations. Chatbots (also referred to herein as "bots" or "skills") have thus begun to be developed to engage in conversations with end users, particularly over the Internet. End users or customers can communicate with the bot via a messaging application that the end user has already installed and uses. Generally, intelligent bots driven by artificial intelligence (AI) can communicate more intelligently and contextually in a live conversation, and thus can enable a more natural conversation between the bot and the end user for an improved conversation experience. Instead of the end user learning a fixed set of keywords or commands that the bot knows how to respond to, intelligent bots can understand the end user's intent based on the end user's utterance in natural language and respond accordingly.

[0004] However, building chatbots is challenging because these automated solutions require specific knowledge in a particular field and the application of specific technologies, which may only be within the capabilities of expert developers. As part of building such a chatbot, developers may first understand the needs of the company and end users. Then, developers may analyze and make decisions related to, for example, selecting the dataset to be used for analysis, preparing the input dataset for analysis (e.g., data cleansing, pre-analysis data extraction, formatting and / or transformation, performing data feature engineering, etc.), identifying appropriate machine learning (ML) techniques or models to perform the analysis, and improving the techniques or models to improve results / outcomes based on feedback. The task of identifying appropriate models may include developing multiple models, possibly in parallel, and identifying a particular model for use after iteratively testing and experimenting with these models. Furthermore, supervised learning-based solutions typically include a training phase, a subsequent application (i.e., inference) phase, and an iterative loop between the training and application phases. Developers may be responsible for carefully executing and monitoring these phases to achieve an optimal solution.

[0005] Despite their widespread adoption, machine learning models, including neural network models, generally operate as black boxes, performing inferences (outputs) by operating mathematical or logical operations on input data. The output of a machine learning model may include, for example, predictions or classifications, and / or corresponding confidence levels. However, the output of a machine learning model, or the output for a particular inference, may not provide insights into the machine learning model's behavior, such as why a particular input would produce a particular output, in order to determine whether the model and / or a particular prediction is reliable, and how to improve the training data and the model. In addition, the training sets developed for machine learning models are often manually generated and then fed into the machine learning model. The time available for training or retraining a machine learning model can be constrained by the ability to generate these training sets. There is no comprehensive method for integrating inference explanation systems into a retraining system to improve a machine learning model. [Overview of the Initiative] [Means for solving the problem]

[0006] overview The technologies disclosed herein generally relate to chatbots. More specifically, but not limited to, the technologies disclosed herein relate to, for example, technologies for providing explanations for text classification in chatbots. A chatbot can classify user utterances into different classes, such as predefined user intents. The chatbot's classifier may include a trained ML model that generates outputs (e.g., intents) based on input (e.g., user utterances). Classification is generally a black box lacking interpretability and explainability.

[0007] The technologies disclosed herein provide descriptions of classifications, which may enable chatbot owners to determine whether a classifier and / or a particular classification is reliable, and how to improve training utterances and the classifier based on the descriptions. Various embodiments described herein include methods, systems, and non-temporary computer-readable storage media for storing programs, code, or instructions executable by one or more processors.

[0008] According to various embodiments, a computer-implemented method is provided, the method comprising: a chatbot system receiving a request to describe the inference result of an intent classifier of the chatbot system regarding an utterance; obtaining the inference result of an intent classifier regarding an utterance; and selecting one or more anchors based on the utterance, each of the one or more anchors comprising one or more anchor words from the utterance; the method further comprising, for each of the one or more anchors, changing one or more words of the utterance that are not the anchor words of the anchor in order to generate variations of the utterance; obtaining the inference result of an intent classifier regarding variations of the utterance; determining the respective confidence levels associated with each of the one or more anchors based on the inference result of an utterance and the inference result of variations of the utterance; and generating a report that includes the anchor having the highest confidence level among the one or more anchors.

[0009] In some embodiments, determining the confidence level includes determining whether there is agreement or disagreement between the inference result for an utterance and the inference result for a variation of the utterance.

[0010] In some embodiments, selecting one or more anchors includes selecting one or more anchors in one or more rounds using beam search techniques.

[0011] In some embodiments, the report includes a JavaScript® object notation (JSON) file.

[0012] In some embodiments, the report includes an anchor with a confidence level greater than the threshold.

[0013] In some embodiments, the computer-implemented method further includes storing variations of utterances and inference results about those variations in a database or cloud-based storage structure.

[0014] According to various embodiments, a computer-implemented method is provided, which includes a chatbot system receiving multiple training results, each containing multiple utterances and multiple corresponding confidence values; comparing the multiple training results with a threshold; based on the comparison, acquiring one or more utterances having confidence values ​​below the threshold; determining one or more training categories corresponding to the one or more acquired utterances; and training a machine learning model using the one or more utterances based on the one or more training categories.

[0015] In some embodiments, the computer-implemented method further includes determining each corresponding confidence value of a plurality of corresponding confidence values ​​based on the agreement or disagreement between a particular utterance among a plurality of utterances and the corresponding predicted training category for that utterance.

[0016] In some embodiments, the computer-implemented method further includes generating one or more synthesized utterances based on multiple utterances from multiple training results.

[0017] In some embodiments, the computer-implemented method further includes generating one or more synthesized utterances based on one or more acquired utterances.

[0018] In some embodiments, determining one or more training categories involves receiving an input indicating a specific utterance from one or more utterances and a specific corresponding training category from one or more training categories.

[0019] In some embodiments, the computer-implemented method further includes receiving an input indicating a threshold. Input may be received from the user of an interactive interface by inputting the threshold into an interactive interface element.

[0020] In some embodiments, one or more training categories are one or more correct intents, and each correct intent of the one or more correct intents corresponds to one of one or more utterances for retraining a machine learning model.

[0021] In some embodiments, comparing multiple training results with a threshold involves calculating at least one difference value representing the numerical difference between at least one confidence value and the threshold, and obtaining one or more utterances involves selecting utterances that have a difference value indicating that the confidence value is less than or equal to the threshold.

[0022] In some embodiments, the computer-implemented method further includes grouping one or more utterances into training batches and inputting the training batches into a machine learning model.

[0023] In some embodiments, the machine learning model is constructed as an intent classifier that includes multiple model parameters to be learned by using an objective function, and training the machine learning model involves minimizing or maximizing the objective function, which measures the difference between the predicted intent and the correct intent.

[0024] According to various embodiments, a system is provided that includes one or more data processors and a non-transitory computer-readable storage medium including instructions that, when executed on the one or more data processors, cause the one or more data processors to execute some or all of the one or more methods disclosed herein.

[0025] According to various embodiments, a computer program product is provided that is tangibly embodied in a non-transitory machine-readable storage medium and includes instructions configured to cause one or more data processors to execute some or all of the one or more methods disclosed herein.

[0026] In some embodiments, a plurality of utterances used to train a machine learning model are compared to a confidence threshold. The number of utterances having a confidence value below the confidence threshold is determined. The number of utterances is presented to a comprehensive user interface to select a correspondence between the expected intents of specific utterances for retraining the machine learning model. The number of utterances is modified by an utterance generation engine to form one or more modified utterances for use in retraining the machine learning model. The one or more modified utterances may also be presented within the comprehensive user interface to select the same or different expected intents for retraining. The utterances may be displayed as part of a comprehensive retraining interface for interpreting the intent classification explanation and generating a retraining data set for improving the machine learning model.

[0027] The technology and infrastructure disclosed herein may enable a bot designer to understand why an utterance is classified into a particular class by indicating the relative importance of words within the utterance, which may enable the bot designer to improve the bot's classifier for future utterance classification.

[0028] The techniques described above and below can be realized in several ways and in several situations. Some exemplary realizations and situations are provided with reference to the following drawings, as will be described in more detail below. However, the following realizations and situations are only a small number of many.

Brief Description of the Drawings

[0029] [Figure 1] It is a simplified block diagram of a distributed environment incorporating an exemplary embodiment. [Figure 2] FIG. 2 shows a distributed system that realizes a bot system for communicating with an end user using one or more messaging applications according to various embodiments. [Figure 3] FIG. 14 shows an integrated system including a bot system and a bot analysis system for monitoring, analyzing, visualizing, and improving the performance of the bot system according to various embodiments. [Figure 4] It is a simplified flowchart showing an example of a process for developing a bot according to various embodiments. [Figure 5] It is a simplified flowchart showing an example of a process for explaining inference results according to various embodiments. [Figure 6] FIG. 23 shows an example of a system for providing an explanation of text classification according to various embodiments. [Figure 7] FIG. 26 shows an example of a system for providing an explanation of text classification according to various embodiments. [Figure 8A] FIG. 29 shows an example of simulated CPU usage during explanation calculation according to various embodiments. [Figure 8B] FIG. 32 shows an example of simulated memory usage during explanation calculation according to various embodiments. [Figure 9A] FIG. 35 shows an example of simulated CPU usage during explanation calculation according to various embodiments. [Figure 9B] FIG. 38 shows an example of simulated memory usage during explanation calculation according to various embodiments. [Figure 10A] An example of simulated CPU usage during explanatory calculations according to one embodiment is shown. [Figure 10B] Examples of simulated memory usage during explanatory calculations in various embodiments are shown. [Figure 11] This is a simplified flowchart illustrating an example of the process for retraining a model using artificial speech, based on various embodiments. [Figure 12] This document presents exemplary interfaces for retraining a model using artificial speech, based on various embodiments. [Figure 13] This document presents exemplary interfaces for retraining a model using artificial speech, based on various embodiments. [Figure 14] This document presents illustrative interfaces for explaining inference results using various embodiments. [Figure 15] This document presents illustrative interfaces for explaining inference results using various embodiments. [Figure 16] This document presents illustrative interfaces for explaining inference results using various embodiments. [Figure 17] This document presents illustrative interfaces for explaining inference results using various embodiments. [Figure 18] Examples of explanatory outputs for utterance classification using various embodiments are shown. [Figure 19] A simplified block diagram of an example of a distributed system for realizing various embodiments is shown. [Figure 20] This is a simplified block diagram of an example of a cloud-based system environment for realizing various implementations. [Figure 21] An example of a computer system for realizing various embodiments is shown. [Modes for carrying out the invention]

[0030] Detailed explanation Preface This disclosure relates in general to chatbots. More specifically, and not limited to, the techniques disclosed herein relate to providing explanations for text classification in chatbots. A chatbot may classify a user's utterances into different classes, such as a user's predetermined predefined intent. Classifiers are generally black boxes lacking interpretability and explainability. Users may know the classification results but may not understand why a particular input is classified into a corresponding class. The techniques disclosed herein may provide explanations for classifications so that training utterances and classifiers can be improved based on explanations and users can gain confidence in the classification results. This specification describes various embodiments, including methods, systems, and non-temporary computer-readable storage media for storing programs, code, or instructions executable by one or more processors.

[0031] Companies may want to create chatbot systems for various purposes. A chatbot system may include one or more user intent classification engines for identifying end-user intents based on user utterances, and one or more dialogue engines for intelligently and contextually constructing messages to respond to user utterances according to the determined end-user intent. However, building a chatbot system that includes both a user intent classification engine capable of determining end-user intents based on user utterances and a dialogue engine for intelligently and contextually generating responses is a challenging task, partly due to the nuances and ambiguities of natural language, as well as the dimensions of the input space (e.g., possible user utterances) and the size of the output space (number of intents).

[0032] A chatbot's classifier may include a trained machine learning (ML) model that infers the end-user's intent from the end-user's utterance. An ML model generally operates as a black box, producing an output (e.g., an intent) based on an input (e.g., a user utterance) by performing specific mathematical or logical operations on the input. ML models generally lack interpretability and explainability, meaning that a chatbot owner, lacking specific knowledge of how the chatbot is built, may not understand why the chatbot produces a particular output based on certain input assumptions. More specifically, an ML model's output may include, for example, a classification (or another prediction) and / or a corresponding confidence level for that prediction (i.e., how closely the chatbot judges the predicted output intent to match the utterance input), but may not provide insights into the ML model's behavior, such as why a particular input would produce a particular output, in order to determine whether the ML model and / or a particular prediction are reliable, and how to improve the training data and the ML model.

[0033] Furthermore, there is no way to automatically and comprehensively generate and use training data to improve the judged behavior problems of an ML model. For example, given a report, a client might understand how the ML model may have made some wrong predictions, but they will still be stuck with the wrong model. The client might try to improve these problems by finding a new training dataset or attempting to create their own training dataset in order to retrain the ML model. The client has no way of knowing whether any training dataset they find or create will improve the specific problems presented in the report.

[0034] Therefore, different approaches are needed to address these problems. In various embodiments, a computer-implemented method is provided, which includes receiving a request to describe the inference result of an intent classifier of a chatbot system about an utterance; obtaining inference results about variations of the utterance from the intent classifier; and selecting one or more anchors, each of which anchors may include one or more anchor words in the utterance. The method further includes, for each anchor of the one or more anchors, varying one or more words in the utterance that are not anchor words of the anchor in order to generate variations of the utterance; obtaining inference results about variations of the utterance from the intent classifier; determining the confidence level associated with each anchor of the one or more anchors based on the utterance and the inference results about variations of the utterance; and generating a report that includes the anchor with the highest confidence level among the one or more anchors. In some embodiments, selecting one or more anchors may include selecting one or more anchors in one or more rounds using beam search techniques. The variations of the utterance, the inference results about variations of the utterance, and the report may reside in a database.

[0035] The technologies disclosed herein provide explanations for classifications, which may enable chatbot owners to determine whether a classifier and / or a particular classification is reliable (to take action based on the classification or prediction), and how to improve training utterances and the classifier based on the explanations. For example, an ML model may be retrained using new utterances used when generating a report. The new utterances are similar to the original utterances from which the report was requested. Therefore, given new utterances as input and a known intent to be predicted, the ML model may be retrained using the new utterances to improve future predictions. Since the new utterances have already been generated and are similar to the original utterances from which the report was requested, the ML model may be retrained using the new utterances in a more efficient and functional manner.

[0036] As used herein, “chatbot,” “bot,” “skill,” or “skillbot” may refer to a computer program designed to simulate conversations with human end users, particularly over the internet. Individual skills may be designed to interact with end users and perform specific types of tasks, such as ordering food, making reservations, changing contact information, technical support, and customer service. Each skill may help end users complete tasks through a combination of visual, voice, or text messages and UI elements such as buttons, tables, and lists.

[0037] As used herein, the term “intent” can refer to a category of actions or tasks that an end user expects a skill to perform on their behalf. The term “entity” can refer to a variable that identifies information from user input that enables a skill to perform a task. The term “component” can refer to various functions that a skill may use to respond to an end user, such as outputting text, returning information from a backend, and executing custom logic. The term “dialogue flow” can refer to a definition of a skill-user interaction, which may describe how a skill responds and behaves according to user input. The term “channel” can refer to a platform-specific configuration that enables a skill to access a messaging platform or client messaging app. A single skill may have several channels configured for it so that the skill can run concurrently on different services or platforms that an end user may prefer to use.

[0038] As used herein, “utterance” or “message” may refer to one or more sentences exchanged during a conversation, and a conversation may refer to a communication session which may include one or more utterances or messages. A conversation may include one or more stages or states. A conversation flow may be an abstraction of multiple conversations which include the same stages or states and the same transitions from stage (or state) to stage (or state). Each conversation may be a specific instance of a corresponding conversation flow. A state (or stage) of a conversation (or conversation flow) may be associated with a state machine state maintained by the bot system for conversing with other bot systems or people. In some cases, a state may correspond to an end-user’s intent or goal. As used herein, an end-user may refer to an end-user of the bot system, such as a person or another entity that converses with the bot system through a messaging application or platform. For example, an end-user may be a customer or client of the company that owns the bot system. As used herein, a user of the bot system may refer to the owner, operator, administrator, or developer of the bot system.

[0039] In the following description, certain details are given for illustrative purposes and to ensure a full understanding of the examples in this disclosure. However, it will be apparent that various examples may be implemented without these specific details. The following description is for illustrative purposes only and is not intended to limit the scope, applicability, or configuration of this disclosure. Rather, the following description of the examples will provide a practical explanation for implementing the examples for those skilled in the art. It should be understood that various modifications can be made to the function and configuration of the elements without departing from the spirit and scope of this disclosure as set forth in the claims. Figures and descriptions are not intended to be limiting. Circuits, systems, networks, processes, and other components may be shown as components in block diagram form so as not to obscure the examples with unnecessary details. In other examples, well-known circuits, processes, algorithms, structures, and techniques may be shown without unnecessary details to avoid obscuring the examples. The teachings disclosed herein may also be applicable to various types of applications, such as mobile applications, non-mobile applications, desktop applications, web applications, and enterprise applications. Furthermore, the teachings in this disclosure are not limited to a specific operating environment (e.g., operating system, device, platform, etc.) but may instead be applicable to multiple different operating environments.

[0040] Furthermore, it should be noted that individual examples may be described as processes shown as flowcharts, flow diagrams, data flow diagrams, structure diagrams, or block diagrams. While flowcharts can describe operations as sequential processes, many operations may be performed in parallel or concurrently. In addition, the order of operations may be changed. A process terminates when its operations are completed, but it may have additional steps not shown in the diagram. A process may correspond to a method, function, procedure, subroutine, subprogram, etc. If a process corresponds to a function, its termination may correspond to returning that function to the calling function or main function.

[0041] The terms “example” or “exemplary” are used here to mean “provided as an example, case, or illustration.” Any embodiment or design described herein as “exemplary” or “example” should not necessarily be construed as being preferable or advantageous to other embodiments or designs.

[0042] The terms “machine-readable storage medium” or “computer-readable storage medium” include, but are not limited to, portable or stationary storage devices, optical storage devices, and various other media that can store, contain, or carry instructions and / or data. Machine-readable storage medium or computer-readable storage medium may also include non-temporary media on which data may be stored, and which do not contain carrier waves and / or transient electronic signals propagating over wireless or wired connections. Examples of non-temporary media may include, but are not limited to, magnetic disks or tapes, optical storage media such as compact discs (CDs) or digital multi-purpose discs (DVDs), flash memory, memory, or memory devices. Computer program products may include code and / or machine-executable instructions that can represent procedures, functions, subprograms, programs, routines, subroutines, modules, software packages, classes, or any combination of instructions, data structures, or program statements. Code segments may be coupled to other code segments or hardware circuits by passing and / or receiving information, data, arguments, parameters, or memory content. Information, arguments, parameters, data, etc., may be passed, transferred, or transmitted via any preferred means, including memory sharing, message passing, token passing, network transmission, etc.

[0043] Furthermore, the examples may be implemented by hardware, software, firmware, middleware, microcode, hardware description languages, or any combination thereof. When implemented by software, firmware, middleware, or microcode, the program code or code segment (e.g., a computer program product) that performs the required task may be stored in a machine-readable medium. The required task may be performed by a processor. Some illustrated systems may be provided in various configurations. In some examples, the system may be configured as a distributed system in which one or more components of the system are distributed across one or more networks of a cloud computing system. Where a component is described as being "configured" to perform a particular operation, such configuration may be achieved, for example, by designing electronics or other hardware to perform the operation, by programming and controlling electronics (e.g., a microprocessor or other suitable electronics) to perform the operation, or by any combination thereof.

[0044] Bot and analytics systems A bot (also known as a skill, chatbot, chatterbot, or talkbot) is a computer program that can engage in conversations with end users. Bots can generally respond to natural language messages (e.g., questions or comments) through messaging applications that use natural language messaging. Businesses may use one or more bot systems to communicate with end users through messaging applications. These messaging applications, sometimes called channels, can be the end user's preferred messaging application, one that the end user has already installed and is familiar with. Therefore, end users do not need to download and install a new application to chat with the bot system. Messaging applications may include, for example, over-the-top (OTT) messaging channels (e.g., Facebook Messenger, Facebook WhatsApp, WeChat, Line, Kik, Telegram, Talk, Skype, Slack, or SMS), virtual private assistants (e.g., Amazon Dot, Echo, or Show, Google® Home, Apple HomePod, etc.), mobile and web app extensions that extend native or hybrid / responsive mobile or web applications with chat capabilities, or voice-based input (e.g., devices or apps with interfaces that use Siri, Cortana, Google Voice, or other voice input for interaction).

[0045] In some examples, a bot system may be associated with a Unified Resource Identifier (URI). A URI can identify a bot system using a string. A URI can be used as a webhook for one or more messaging application systems. A URI can include, for example, a Unified Resource Location Specifier (URL) or a Unified Resource Name (URN). A bot system may be designed to receive messages (e.g., Hypertext Transfer Protocol (HTTP) post-call messages) from messaging application systems. HTTP post-call messages may be directed from messaging application systems to URIs. In some embodiments, messages may differ from HTTP post-call messages. For example, a bot system may receive messages from the Short Message Service (SMS). The discussion here may refer to communications that a bot system receives as messages, but it should be understood that messages can be HTTP post-call messages, SMS messages, or any other type of communication between two systems.

[0046] End users can interact with bot systems through conversational interactions (sometimes called conversational user interfaces (UIs)), similar to how people interact with each other. In some cases, the interaction may involve the end user saying "Hello" to the bot, the bot responding "Hi," and the bot asking the end user how it can help. In some cases, the interaction may also be a transactional interaction with a banking bot, such as transferring money from one account to another; an informational interaction with an HR bot, such as checking a vacation balance; or an interaction with a retail bot, such as discussing returning a purchased item or seeking technical support.

[0047] In some embodiments, a bot system can intelligently handle end-user interactions without interaction with the bot system's administrator or developer. For example, an end-user may send one or more messages to the bot system to achieve a desired goal. The messages may contain some content, such as text, emojis, audio, images, video, or other means of conveying the message. In some embodiments, the bot system can convert the content into a standardized format (e.g., a representational state transfer (REST) ​​call to an enterprise service with appropriate parameters) and generate a natural language response. The bot system may also prompt the end-user for additional input parameters or request other additional information. In some embodiments, the bot system may also initiate communication with the end-user rather than passively responding to the end-user utterance. Here, various techniques for identifying explicit invocations of a bot system and determining input to the bot system being invoked are described. In some embodiments, explicit invocation analysis is performed by the master bot based on the detection of an invocation name in the utterance. In response to the detection of an invocation name, the utterance may be refined for input to a skill bot associated with the invocation name.

[0048] A conversation with a bot can follow a specific conversational flow that includes multiple states. The flow can define what happens next based on the input. In some embodiments, a bot system can be implemented using a state machine that includes user-defined states (e.g., end-user intents) and actions to be taken in or from a state. A conversation can take different paths based on end-user input, which can influence the decisions the bot makes about the flow. For example, in each state, based on end-user input or utterance, the bot can determine the end-user intent and decide on the appropriate next action. Here, and in the context of utterances, the term “intent” refers to the intent of the user who gave the utterance. For example, if a user intends to engage the bot in a conversation to order a pizza, the user’s intent might be expressed by the utterance “order a pizza.” A user’s intent can be directed towards a specific task that the user wants the chatbot to perform on their behalf. Thus, utterances can be expressed as questions, commands, requests, etc., that reflect the user’s intent. An intent can include goals that the end-user wishes to achieve.

[0049] In the context of chat configuration, the term “intent” is used here to refer to configuration information for mapping user utterances to specific tasks / actions or categories of tasks / actions that a chatbot can perform. To distinguish between utterance intents (i.e., user intents) and chatbot intents, the latter may be referred to here as “bot intents.” A bot intent may include a set of one or more utterances associated with that intent. For example, an intent to order a pizza could have various permutations of utterances expressing the desire to order a pizza. These associated utterances can be used to train the chatbot’s intent classifier, which can then determine whether an input utterance from the user matches the pizza order intent. A bot intent may be associated with one or more dialogue flows for initiating a conversation with the user in a certain state. For example, the first message for a pizza order intent might be the question, “What kind of pizza would you like?” In addition to associated utterances, a bot intent may further include named entities associated with that intent. For example, a pizza order intent might include variables or parameters used to perform the task of ordering a pizza, such as topping 1, topping 2, pizza type, pizza size, and pizza quantity. The entity's values ​​are typically obtained through conversation with the user.

[0050] Figure 1 is a simplified block diagram of a distributed environment 100 incorporating an exemplary embodiment. The distributed environment 100 includes a Digital Assistant Builder Platform (DABP) 102 that enables enterprises to create and deploy digital assistants for their end users. The DABP 102 may be used to create one or more digital assistants (DAs). The DABP 102 may be used by multiple enterprises to create digital assistants for end users of multiple enterprises. For example, as shown in Figure 1, a user 104 representing a particular enterprise may use the DABP 102 to create and deploy a digital assistant 106 for end users of that particular enterprise. For example, a restaurant (e.g., a pizza shop) may use the DABP 102 to create and deploy a digital assistant that enables the restaurant's customers to order food (e.g., order a pizza).

[0051] For the purposes of this disclosure, “Digital Assistant” refers to an entity that helps the end user of the Digital Assistant accomplish various tasks through natural language conversation. A Digital Assistant may be implemented using software alone (for example, a Digital Assistant is a digital entity implemented using programs, code, or instructions executable by one or more processors), using hardware, or using a combination of hardware and software. A Digital Assistant may be embodied or implemented in various physical systems or devices, such as computers, mobile phones, watches, appliances, or vehicles. A Digital Assistant is also sometimes referred to as a chatbot system.

[0052] For example, as shown in Figure 1, an end user 108 may use a digital assistant 106 to perform various tasks through natural language-based conversations with the digital assistant 106. As part of the conversation, the user can provide one or more user inputs 110 and receive responses 112 from the digital assistant 106. Through these conversations, the user can request that one or more tasks be performed by the digital assistant, and in response, the digital assistant may perform the user-requested tasks and respond to the user with appropriate responses.

[0053] User input 110 may be natural language and may be called utterance. User utterance can be in text format (e.g., when a user types something as input to the digital assistant 106) or in auditory input or speech format (e.g., when a user says something as input to the digital assistant 106). Utterance is typically the language spoken by the end user 108. If user input is in speech format, user input may be converted to a text utterance in that particular language, which is then processed by the digital assistant 106. Various speech-to-text processing techniques may be used to convert speech or auditory input to a text utterance, which is then processed by the digital assistant 106.

[0054] Text utterances, either entered by the user or generated by converting voice input into text format, can be text fragments, sentences, or multiple sentences. The digital assistant 106 is configured to understand the meaning of user input by applying natural language understanding (NLU) technology to the text utterances. As part of the NLU processing of the utterances, the digital assistant 106 is configured to perform processing to understand the meaning of the utterances, which involves identifying one or more intents and one or more entities corresponding to the utterances. Once the meaning of the utterances is understood, the digital assistant 106 can perform one or more actions or behaviors in response to the understood meaning or intents.

[0055] For example, user input may request that a pizza be ordered, such as "I want to order a pizza." The digital assistant 106 is configured to understand the meaning of the utterance and take appropriate actions, which may involve responding to the end user with questions requesting user input regarding the type of pizza the end user wishes to order, the size of the pizza, any toppings on the pizza, etc. The response provided by the digital assistant 106 may also be in the form of natural language, which may include natural language generation (NLG) processing performed by the digital assistant 106. Once the digital assistant 106 has obtained the necessary information from the user, the digital assistant 106 will order the pizza. The digital assistant 106 can terminate the conversation with the end user by outputting information indicating that the pizza has been ordered.

[0056] In certain embodiments, the utterance received as input by the digital assistant 106 passes through a sequence or pipeline of processing steps. These steps may include, for example, parsing the utterance, understanding the meaning of the utterance, scrutinizing and modifying the utterance to develop a more understandable structure, determining an action to be taken in response to the utterance, having the action performed, generating a response to be output to the end user in response to the user utterance, and outputting the response to the end user.

[0057] NLU processing performed by a digital assistant such as digital assistant 106 may include various NLP-related processes such as sentence analysis (e.g., tokenization, sorting, part-of-speech tagging for sentences, named entity recognition in sentences, generation of dependency trees to represent sentence structure, sentence splitting into clauses, analysis of individual clauses, resolution of anaphora, and chunking). In some embodiments, the NLU processing or a portion thereof may be performed by digital assistant 106 itself. In some other embodiments, digital assistant 106 may perform a portion of the NLU processing using other resources. For example, the syntax and structure of a sentence may be identified by processing the sentence using parsing, part-of-speech tagging, and / or named entity recognition. In one implementation, for English, parsing, part-of-speech tagging, and / or named entity recognition provided by the Stanford Natural Language Processing (NLP) Group are used to analyze sentence structure and syntax. These are provided as part of the Stanford CoreNLP toolkit.

[0058] The various examples provided in this disclosure demonstrate English utterances, but these are meant to be examples only. In certain embodiments, the digital assistant 106 may also process utterances in languages ​​other than English. In certain embodiments, the digital assistant 106 provides subsystems configured to process different languages ​​(e.g., components that implement NLU functionality). These subsystems may be implemented as pluggable units that can be invoked using service calls from the NLU core server. This makes the NLU processing flexible and extensible for each language, including allowing processing in different orders. Language packs may be provided for each respective language, and language packs may register a list of subsystems that may be provided by the NLU core server, and may also utilize generic subsystems provided as needed.

[0059] Digital assistants such as Digital Assistant 106 can be made available to their end users through a variety of different channels, including, but not limited to, specific applications (also called apps), social media platforms, various messaging services and applications, and other applications or channels. A single digital assistant can have several channels configured for it, so that it can run on different services simultaneously and be accessed by different services simultaneously.

[0060] A digital assistant may consist of or be associated with one or more skills. In some embodiments, these skills are individual chatbots designed to interact with the end user and fulfill specific types of tasks, such as tracking inventory, submitting time cards, creating expense reports, ordering food, verifying bank accounts, making reservations, and purchasing widgets. For example, in the embodiment shown in Figure 1, the digital assistant 106 includes skillbots 116-1, 116-2, 116-3… As stated above, the term “skill” is used synonymously with the term “skillbot” in this disclosure.

[0061] Each skillbot associated with a digital assistant helps the end user complete tasks through conversation with the digital assistant, and the conversation may include a combination of text or auditory input provided by the end user and responses provided by the skillbot. These responses may take the form of text or auditory messages to the end user, and / or use simple user interface elements (e.g., a selection list) that are presented to the end user for making a choice.

[0062] There are various ways in which skills or skillbots can be added to a digital assistant. In one example, a skillbot may be developed by a company and then added to a digital assistant using DABP102. In another example, a skillbot may be developed and created using DABP102 and then added to a digital assistant created using DABP102. In yet another example, DABP102 provides an online digital store (called a "skillstore") that offers multiple skills aimed at a wide range of tasks. Skills offered through the skillstore can be published to various cloud services. DABP102 users can access the skillstore via DABP102, select the desired skills, and add the selected skills to a digital assistant created using DABP102. Skills from the skillstore can be added to a digital assistant either as is or in a modified form. For example, a DABP102 user can select and clone a specific skill offered by the skillstore, customize or modify the selected skillbot, and then add the modified skillbot to a digital assistant created using DABP102.

[0063] In one embodiment, the digital assistant created and deployed using DABP102 is implemented using a master-bot / child (or sub)-bot paradigm or architecture. According to this paradigm, the digital assistant is implemented as a master bot that interacts with one or more child bots, which are skill bots. For example, in the embodiment shown in Figure 1, the digital assistant 106 includes a master bot 114 and skill bots 116-1, 116-2, 116-3, etc., which are child bots of the master bot 114. In one embodiment, the digital assistant 106 itself functions as the master bot.

[0064] A digital assistant implemented according to a master-child bot architecture allows the end user of the digital assistant to interact with multiple skills through an integrated user interface. When an end user engages with the digital assistant, user input is received by the master bot, which processes the user input to identify the user request. Based on the processing, the master bot determines whether the user request can be processed by the master bot itself. If it is determined that the user request does not need to be processed by the master bot itself, the master bot selects an appropriate skill bot to process the user request and routes the conversation to the selected skill bot. This allows the end user to interact with and use several skill bots configured to perform specific tasks through a common, single interface. For example, in the case of a digital assistant developed for a company, the master bot of the digital assistant may interface with skill bots with specific functions, such as a Customer Relationship Management (CRM) bot to perform functions related to customer relationship management, an Enterprise Resource Planning (ERP) bot to perform functions related to enterprise resource planning, and a Human Capital Management (HCM) bot to perform functions related to human capital management. In this way, the end user or consumer of the digital assistant only needs to know how to access the digital assistant.

[0065] In a master-bot / child-bot infrastructure, the master bot is configured to be aware of a list of skill bots. The master bot can access metadata that identifies various available skill bots, and for each skill bot, it can access the capabilities of each skill bot, including the tasks that each skill bot can perform. Upon receiving a user request in the form of an utterance, the master bot is configured to identify or predict which of the multiple available skill bots is best suited to processing or handling the user request. The master bot then routes the utterance (or a portion of the utterance) to that specific skill bot for further processing. Thus, control flows from the master bot to the skill bots. The master bot can support multiple input and output channels.

[0066] The embodiment shown in Figure 1 illustrates a digital assistant 106 comprising a master bot 114 and skill bots 116-1, 116-2, and 116-3, but this is not intended to be limiting. The digital assistant may include various other components (e.g., other systems and subsystems) that provide the functionality of the digital assistant. These systems and subsystems may be implemented via software only (e.g., code, instructions stored on a computer-readable medium and executable by one or more processors), hardware only, or a combination of software and hardware.

[0067] DABP102 provides infrastructure, services, and features that enable the creation of digital assistants that include one or more skillbots associated with the digital assistant. For example, a skillbot can be created by cloning an existing skillbot and then modifying it, or it can be created from scratch using the tools and services provided by DABP102. In one embodiment, DABP102 provides a skill store or skill catalog that offers multiple skillbots for performing various tasks. Users of DABP102 can clone skillbots from the skill store and create new skillbots.

[0068] DABP102 also allows users (e.g., skillbot designers) to create skillbots from scratch. In certain embodiments, at a higher level, creating a skillbot involves the following actions: (1) Configure settings for the new skillbot (2) Set up one or more intents for SkillBot (3) Set up an entity for one or more intents (4) Train SkillBot (5) Create a dialog flow for SkillBot (6) Add custom components to the skillbot, and (7) Test and deploy the skillbot.

[0069] (1) Setting up the new skillbot - The skillbot designer can specify one or more invocation names for the skillbot being created. These invocation names can be used in utterances to explicitly identify and invoke the skillbot in the digital assistant. The skillbot designer can also specify exemplary utterances for the skillbot. These exemplary utterances represent utterances directed at the skillbot. When user input is received, the digital assistant's intent analysis engine compares the user input to these exemplary utterances to determine whether a particular skillbot is invoked.

[0070] (2) Setting one or more intents for a skillbot - A skillbot designer can set one or more intents (also called bot intents) for the skillbot being created. These intents identify tasks that the skillbot can perform for the end user of the digital assistant. Each intent is given a name. For example, for a skillbot configured to help a user perform various banking transactions, the skillbot designer might specify intents for the skillbot such as "Balance Inquiry," "Money Transfer," and "Check Deposit." For each intent, the skillbot designer represents and indicates the meaning of the intent and typically specifies a set of exemplary utterances associated with the task performed by that intent. For example, for the "Balance Inquiry" intent, exemplary utterances could include "What's my savings account balance?", "How much is in my checking account?", and "How much money do I have in my account?". Permutations of typical user requests and statements may also be specified as exemplary utterances for an intent.

[0071] (3) Setting up entities for one or more intents of the skillbot - In some cases, additional context may be needed to enable the skillbot to respond appropriately to user requests. For example, there may be situations where two or more user input utterances resolve to the same intent in the skillbot. For example, in the example above, the utterances "What's my savings account balance?" and "How much is in my checking account?" both resolve to the same "Balance Inquiry" intent, but these utterances are different requests that want different answers. To clarify such requests, one or more entities are added to the intent. Using the banking skill example, an entity called "Account Type" that defines values ​​called "checking" and "saving" may enable the skillbot to parse user requests and respond appropriately. One or more entities may be specified for a particular intent set up for the skillbot. Thus, entities are used to add context to the intent itself. Entities help to better describe intents and enable skills to complete user requests. In one embodiment, there are two types of entities: (a) built-in entities provided by DABP102, and (2) custom entities that may be specified by the skill bot designer. Built-in entities are general-purpose entities that can be used with a wide variety of bots. Examples of built-in entities include, but are not limited to, entities related to time, date, address, number, email address, duration, cycle period, currency, phone number, URL, etc. Custom entities are used for more customized applications. For example, for a banking skill, the "account type" entity may be defined by the skill bot designer to enable various banking transactions by checking user input for keywords such as checking, savings, and credit card.

[0072] (4) Training the SkillBot - The SkillBot is configured to receive user input, parse or otherwise process the received user input, and identify or select intents related to the received user input. To achieve this, the SkillBot may be trained. In one embodiment, the SkillBot is trained on intents set for the SkillBot and exemplary utterances associated with those intents (collectively referred to as training data), so that the SkillBot can resolve user input to one of the SkillBot's set intents. In a particular embodiment, the SkillBot is represented by a model that is trained with the training data and enables the SkillBot to identify what the end user is saying (or, in some cases, trying to say). DABP102 provides a variety of different training techniques that can be used by SkillBot designers to train SkillBots, including a variety of machine learning-based training techniques, rule-based training techniques, and / or combinations thereof. In one embodiment, a portion of the training data (e.g., 80%) is used to train the SkillBot model, and another portion (e.g., the remaining 20%) is used to test or validate the model. Once trained, skillbots can be used to address and respond to user utterances. In some cases, a user utterance may be a question that requires only a single answer and does not require further conversation. To address such situations, a Q&A (Question & Answer) intent can be set for the skillbot. This allows the skillbot to output a response to a user request without needing to update the dialog definition. Q&A intents are generated in the same way as regular intents. However, the dialog flow for Q&A intents differs from that of regular intents.

[0073] (5) Create a dialogue flow for the skillbot - The dialogue flow specified for the skillbot describes how the skillbot responds when different intents to the skillbot are resolved in response to user input received. The dialogue flow defines the actions or behaviors that the skillbot takes (e.g., how the skillbot responds to user utterances, how the skillbot prompts the user for input, how the skillbot returns data, etc.). The dialogue flow is like a flowchart that the skillbot follows. The skillbot designer specifies the dialogue flow using a language such as Markdown. In one embodiment, a version of YAML called OBotML can be used to specify the dialogue flow for the skillbot. The dialogue flow definition for the skillbot acts as a model of the conversation itself, allowing the skillbot designer to choreograph the interaction between the skillbot and the end user it corresponds to.

[0074] In one embodiment, the dialog flow definition includes the following three sections: (a) Context section; (b) Default transition section; and (c) The status section.

[0075] Context Section - In the context section, skillbot designers can define variables used in the conversation flow. Other variables that may be designated in the context section include, but are not limited to, variables for error handling, variables for built-in or custom entities, and user variables that enable skillbot to recognize and persist user preferences.

[0076] Default Transition Section - Skillbot transitions can be defined in either a dialog flow state section or a default transition section. Transitions defined in the default transition section act as fallbacks and are triggered when there are no applicable transitions defined within a state or when the conditions necessary to trigger a state transition are not met. The default transition section can be used to define routing that allows Skillbot to smoothly handle unexpected user actions.

[0077] The State Section – Dialog Flow and its associated behavior are defined as a set of transient states that govern the logic within the dialog flow. Each state node in the dialog flow definition designates a component that provides the functionality required for that point in the dialog. In this way, states are built around the components. Each state contains component-specific characteristics and defines transitions to other states that are triggered after the component has been executed.

[0078] Special case scenarios can be handled using the state section. For example, you might want to give an end user the option to temporarily leave the first skill they are working on and do something in a second skill within the digital assistant. In one example, if an end user is engaged in a conversation with a shopping skill (e.g., the user has made some selections for a purchase), they might want to jump to a banking skill (e.g., the end user wants to make sure they have enough money for the purchase) and then return to the shopping skill to complete their order. To address this, an action in the first skill can be configured to initiate a conversation with a second, different skill within the same digital assistant, and then return to the original flow.

[0079] (6) Adding Custom Components to the SkillBot - As described above, the states specified in the dialog flow for the SkillBot designate the components that provide the functionality required for that state. The components enable the SkillBot to perform the functionality. In one embodiment, DABP102 provides a set of pre-configured components for performing a wide range of functions. The SkillBot designer can select one or more of these pre-configured components and associate them with states in the dialog flow for the SkillBot. The SkillBot designer can also create custom or new components using the tools provided by DABP102 and associate the custom components with one or more states in the dialog flow for the SkillBot.

[0080] (7) Test and deploy SkillBots - DABP102 provides several features that enable SkillBot designers to test SkillBots under development. The SkillBots can then be deployed and included in a digital assistant.

[0081] The above description explains how to create a skillbot, but similar techniques can also be used to create a digital assistant (or masterbot). At the masterbot or digital assistant level, you can set up built-in system intents for the digital assistant. These built-in system intents are used to identify common tasks that the digital assistant itself (i.e., the masterbot) can handle without calling the skillbot associated with the digital assistant. Examples of system intents defined for a masterbot include: (1) Exit: when the end user wants to signal to the digital assistant that they wish to end the current conversation or context; (2) Help: when the end user requests assistance or orientation; and (3) Unresolved Intent: for user input that does not well match the Exit intent or Help intent. The digital assistant also remembers information about one or more skillbots associated with the digital assistant.

[0082] At the master bot or digital assistant level, when an end user inputs a phrase or utterance into the digital assistant, the digital assistant is configured to process how to route the conversation. The digital assistant makes this determination using a routing model that can be rule-based, artificial intelligence-based, or a combination of both. Using the routing model, the digital assistant determines whether the conversation corresponding to the user input should be routed to a specific skill for processing, processed by the digital assistant or master bot itself based on built-in system intents, or processed as a different state in the current conversation flow.

[0083] In certain embodiments, as part of this process, the digital assistant determines whether the user input identifies a skillbot using its invocation name. The invocation name present in the user input may be treated as an explicit invocation of the skillbot corresponding to that invocation name. In such scenarios, the digital assistant can route the user input to the explicitly invoked skillbot for further processing. If there is no specific invocation, in some embodiments, the digital assistant evaluates the received user input and calculates confidence scores for the system intent and the skillbot associated with the digital assistant. The scores calculated for a skillbot or system intent represent the likelihood that the user input represents a task or system intent that the skillbot is configured to perform. System intents or skillbots whose associated calculated confidence scores exceed a threshold (e.g., a confidence threshold routing parameter) are selected as candidates for further evaluation. The digital assistant then selects a specific system intent or skillbot from the identified candidates for further processing of the user input. In certain embodiments, after one or more skill bots are identified as candidates, the intents associated with those candidate skills are evaluated (according to the intent model for each skill), and a confidence score is applied to each intent. Generally, intents with confidence scores exceeding a threshold are treated as candidate flows. If a specific skill bot is selected, user input is routed to that skill bot for further processing. If a system intent is selected, one or more actions are performed according to the selected system intent.

[0084] As described above, a skill (also known as a bot, chatbot, chatterbot, skillbot, or talkbot) is a computer program that can engage in conversations with end users. Bots can generally respond to natural language messages (e.g., questions or comments) through messaging applications that use natural language messaging. Businesses may use one or more bot systems to communicate with end users through messaging applications. Messaging applications, sometimes called channels, can be the end user's preferred messaging application, one that the end user has already installed and is familiar with. Therefore, end users do not need to learn a programming language or download and install a new application to chat with a bot system. Messaging applications may include, for example, over-the-top (OTT) messaging channels (e.g., Facebook Messenger, Facebook WhatsApp, WeChat, Line, Kik, Telegram, Talk, Skype, Slack, or SMS), virtual private assistants (e.g., Amazon Dot, Echo, or Show, Google® Home, Apple HomePod, etc.), mobile and web app extensions that extend native or hybrid / responsive mobile or web applications with chat capabilities, or voice-based input (e.g., devices or apps with interfaces that use Siri, Cortana, Google Voice, or other voice input for interaction).

[0085] In some examples, a bot system may be associated with a Unified Resource Identifier (URI). A URI can identify a bot system using a string. A URI can be used as a webhook for one or more messaging application systems. A URI may include, for example, a Unified Resource Location Specifier (URL) or a Unified Resource Name (URN). A bot system may be designed to receive messages (e.g., Hypertext Transfer Protocol (HTTP) post-call messages) from messaging application systems. HTTP post-call messages may be directed from messaging application systems to URIs. In some embodiments, messages may differ from HTTP post-call messages. For example, a bot system may receive messages from the Short Message Service (SMS). While the discussion herein may refer to communications that a bot system receives as messages, it should be understood by those skilled in the art that a message can be an HTTP post-call message, an SMS message, or any other type of communication between two systems.

[0086] End users can interact with bot systems through conversational interactions (sometimes called conversational user interfaces (UIs)), similar to conversations between people. In some cases, the interaction may involve the end user saying "Hello" to the bot, the bot responding "Hi," and the bot asking the end user how it can help. In some cases, the interaction may also be a transactional interaction with a banking bot, for example, transferring money from one account to another. The interaction may also be an informational interaction with a human resources (HR) bot, for example, checking a vacation balance. The interaction may also be an interaction with a retail bot, for example, discussing returning a purchased item or seeking technical support.

[0087] In some embodiments, the bot system can intelligently handle end-user interactions without interaction with the bot system's administrator or developer. For example, an end-user may send one or more messages to the bot system to achieve a desired goal. The messages may contain some content, such as text, emojis, audio, images, video, or other means of conveying the message. In some embodiments, the bot system can convert the content into a standardized format (e.g., a representational state transfer (REST) ​​call to an enterprise service with appropriate parameters) and generate a natural language response. The bot system may also prompt the end-user for additional input parameters or request other additional information. In some embodiments, the bot system may also initiate communication with the end-user rather than passively responding to end-user utterances.

[0088] A conversation with a bot can follow a specific conversational flow that includes multiple states. The flow can define what happens next based on the input. In some embodiments, a bot system can be implemented using a state machine that includes user-defined states (e.g., end-user intents) and actions to be taken in or from a given state. The conversation can take different paths based on end-user input, which can influence the decisions the bot makes about the flow. For example, in each state, based on end-user input, the bot can determine the end-user intent and decide on the appropriate next action to take.

[0089] An intent may include a goal that the end user wants to achieve. The intent maps the end user input to an action that the backend system can perform for the end user. Thus, based on a phrase uttered in natural language by the end user, the bot may map the end user utterance to a specific use case or job, such as ordering a pizza, getting an account balance, making a transfer, making a purchase, or making a return. Human conversation is often non-linear in nature. The end user may often branch into different states during the conversation. For example, if the end user wants to transfer funds from account A to a recipient, the end user may start the conversation with the bot system by, for example, asking the bot to pay for the recipient's dinner. The bot may respond, for example, "From which account?" The end user may choose a checking account, in which case they may realize they are not sure about the balance in the account. Thus, the end user may switch contexts and ask about the balance and recent transactions, etc. In other words, the end user may trigger changes in flow and state, for example, from a transfer to a balance check, and then to recent transactions. At some point, the end user may decide to revert to their original intent—to pay the recipient. Therefore, one task of the bot system is to dynamically determine the end user's intent from natural language utterances.

[0090] The bot may use a natural language processing (NLP) engine and / or a machine learning model (e.g., an intent classifier) ​​to map end-user utterances to specific intents. For example, a machine learning-based NLP engine may be trained to understand and classify natural language conversations from end-users, extract necessary information from the conversation, and take precise actions such as performing transactions or retrieving data from a backend system of records.

[0091] Figure 2 shows a distributed system 200 that implements a bot system to communicate with end users using a messaging application, according to one embodiment. System 200 may include a bot system 220, one or more messaging application systems 215, and one or more end-user devices such as one or more mobile devices 210. In some examples, the messaging application may be installed on an electronic device (e.g., a desktop computer, laptop, mobile device 210, etc.). While the discussion herein refers to mobile devices and messaging applications, those skilled in the art will recognize that any electronic device may be used, and any messaging platform or messaging application may be used, such as Facebook Messenger, Whatsapp Instant Messaging Software, WeChat Mobile Text and Voice Messaging Communication Service, Kik Messenger, Telegram Messenger, Skype Mobile Messenger, Short Message Service (SMS), or any other messaging application that provides a platform for end users to communicate. In other examples, the messaging application may run through a browser installed on the mobile device 210 (e.g., Google Chrome® browser, Mozilla® Firefox® browser, Internet Explorer browser). In some embodiments, two or more messaging applications may be installed on the end-user device to communicate through two or more messaging platforms (e.g., two or more messaging application systems 215).

[0092] The messaging application may be facilitated by a messaging platform such as a messaging application system 215. The mobile device 210 may be connected to the messaging application system 215 by a first network (for example, the internet). The messaging application system 215 may be a messaging platform provided by a third party such as Facebook, Tencent, Google, or Microsoft. The messaging application system 215 may manage content sent and received through the messaging application across multiple mobile devices or other end-user devices.

[0093] A bot system 220 (for example, implemented on one or more servers) may also be communicatively connected to the messaging application system 215 to send and receive messages. Communication between the messaging application system 215 and the bot system 220 may be via a second network (e.g., the Internet). The first and second networks may be the same network, or they may be similar or completely different networks. The messaging application system 215 may use the Internet to route content (e.g., messages or information from messages) from the mobile device 210 to the bot system 220. In some embodiments, the destination of the content (e.g., identification information of the bot system 220) may be included in the content as a nominal address. In some embodiments, the bot system 220 may also be configured to communicate with two or more messaging application systems 215.

[0094] As mentioned above, content exchanged between end users or between an end user and a bot system may include, for example, text, emojis, audio, media (e.g., images, videos, links), or any other method of conveying a message. For example, an example of a message received by bot system 220 from Facebook Messenger may include:

[0095]

number

[0096] The bot system 220 may receive content from the messaging application system 215 using a connector 230 that functions as an interface between the bot system 220 and the messaging application system 215. In some embodiments, the connector 230 may normalize the content from the messaging application system 215 so that the bot system 220 can analyze the content across different messaging application systems. The content normalization process may include formatting the content from each type of messaging application into a common format for processing. In some embodiments, the bot system 220 may include one or more connectors for each messaging application (e.g., Facebook Messenger, Whatsapp Instant Messaging Software, WeChat Mobile Text and Voice Messaging Communication Service, Kik Messenger, Telegram Messenger, and Skype Mobile Messenger, Short Message Service (SMS)). In some implementations, the connector 230 may route the content to a message inqueue 240. The message inqueue 240 may include a buffer (e.g., a first-in, first-out (FIFO) buffer) that stores content in the order it is received. In some embodiments, each connector 230 may be associated with one or more message inqueues.

[0097] The message inqueue 240 may send content to the message processor 250 when the message processor 250 becomes available. In some embodiments, the message processor 250 may pull content from the message inqueue 240. The message processor 250 may parse the message and determine the intent of the parsed message, as will be described in detail below. In some embodiments, the message processor 250 may include a natural language processor 252 and an intent determination subsystem 254. The natural language processor 252 may parse the message and perform certain semantic analysis, such as identifying a subject, predicate (e.g., an action), and / or object. The intent determination subsystem 254 may determine the end-user intent based on the parsed message. As described above, the intent may include the purpose of the message. For example, the purpose of the message may be to order a pizza, give instructions to a computer, send money, or ask a question about delivery. In some embodiments, parameters associated with the intent, which more specifically define or clarify the action to be taken, may also be extracted from the message by the natural language processor 252 and / or intent determination subsystem 254, and may be referred to as entities.

[0098] After the end-user intent is determined by the message processor 250 based on its content, the determined intent (and the parameters associated with the intent) may be sent to the action engine 260. The action engine 260 may be used to determine what action to take based on the intent (and the parameters associated with the intent) and the current state (or context) of the state machine described above, as described above. For example, the action engine 260 may send some outbound content as a response to the message out queue 270 and / or send a command to some enterprise service such as enterprise service 225, or retrieve information from some enterprise service. The message out queue 270 may send the outbound content to the connector 230. The connector 230 may then send the outbound content to a messaging application system indicated by the action engine 260, which may be the same as or different from the messaging application system 215. The messaging application system 215 may then forward the outbound content to a messaging application on the mobile device 210.

[0099] The bot system 220 may communicate with one or more enterprise services (e.g., enterprise service 225), one or more storage systems for storing and / or analyzing messages received by the bot system 220, or a content system for providing content to the bot system 220. Enterprise service 225 may communicate with one or more of the connector 230, the action engine 260, or any combination thereof. Enterprise service 225 may communicate with connector 230 in a similar manner to that of the messaging application system 215. Enterprise service 225 may send content to connector 230 so that it can be associated with one or more end users. Enterprise service 225 may also send content to connector 230 so that the bot system 220 can perform actions associated with end users. The action engine 260 may communicate with enterprise service 225 to retrieve information from enterprise service 225 and / or instruct enterprise service 225 to perform actions identified by the action engine 260.

[0100] In some embodiments, the bot system 220 may include one or more timers. One timer may cause the action engine 260 to send content to the end user using the connector 230 and the messaging application system 215 after a certain amount of time has elapsed. In some embodiments, one timer may send content to the bot system 220 as well as to the end user or enterprise service 225. For example, a timer may send a message to the bot system 220 so that it is analyzed in such a way that a message from an end user will be analyzed.

[0101] In one particular embodiment, an end user may use a mobile device 210 to send a message to the bot system 220 via a messaging application system 215. The message may include a greeting such as "Hello" or "Hi." The bot system may determine that a new conversation has been initiated with the end user and initiate a status machine. In some embodiments, the bot system may identify one or more characteristics of the end user. For example, the bot system may identify the end user's name using a profile associated with the end user on the messaging application system. Using one or more characteristics, the bot system may respond to the end user on the messaging application. The response may include a message to the end user that responds to a message received from the end user. For example, the response may include a greeting with the end user's name, such as "Hi Tom, how can I help you?" Depending on the company associated with the bot system, the bot system may work to achieve the company's goals. For example, if the bot system is associated with a pizza delivery company, the bot system may send a message to the end user asking if they would like to order a pizza. The conversation between the bot system and the end user may continue from there, going back and forth, until the bot system completes the conversation or the end user stops responding to the bot system.

[0102] In some embodiments, the bot system may initiate a conversation with an end user. The conversation initiated by the bot system may respond to a previous conversation with the end user. For example, the end user may have ordered a pizza in a previous conversation. The bot system may then initiate a conversation when the pizza is ready. In some embodiments, the bot system may determine that the pizza is ready when it receives instructions from the company associated with the bot system (for example, an employee sends a message to the bot system indicating that the pizza is ready). The conversation may include a message sent to the end user indicating that the pizza is ready.

[0103] In some embodiments, the bot system may send messages to end users on a different messaging application than the one that previously received the message. For example, the bot system may decide to send messages using Short Message Service (SMS) instead of Facebook Messenger. In such implementations, the bot system may integrate multiple messaging applications.

[0104] In some embodiments, the bot system may decide to initiate a conversation based on a timer. For example, the bot system may decide to have a one-week timer for the end user after a pizza has been ordered. The expiration of the one-week timer may cause the bot system to initiate a new conversation with the end user about ordering another pizza. The timer may be set by the company and implemented by the bot system.

[0105] As described above, in some embodiments, the action engine 260 may send commands to or retrieve information from several enterprise services 225. For example, when the bot system 220 (more specifically, the message processor 250) determines an intent to check a balance, the bot system 220 may determine which of several accounts (e.g., checking accounts or savings accounts) will have its balance checked. If the end user enters "What is the balance in my savings account?", the bot system 220 may extract "savings," send a command to the bank server to check the balance, and then send the received balance information to the end user via message. If the end user initially only says "What is the balance in my account?", the bot system 220 may send a message to the end user prompting the end user to further specify a particular account, or it may retrieve information about all of the end user's accounts and send those accounts to the end user for the end user to choose from.

[0106] In some embodiments, the bot system may maintain information between conversations. This information may be used later so that the bot system does not need to ask any questions each time a new conversation is initiated between the end user and the bot system. For example, the bot system may remember information about the end user's previous pizza order. In a new conversation, the bot system may send the end user a message asking if they want to order the same thing as last time.

[0107] In some embodiments, the bot system 220 may store information associated with end users in a cache. The cache may write to a database to store information after an outbound message has been sent from the connector 230 to the messaging application system. In other embodiments, the cache may write data at different times (e.g., after a specific event, after each event, after a certain amount of time, or any other metric for determining when to write to the database).

[0108] The bot system 220 may allow each component to be scaled when a slowdown is identified. For example, if the bot system 220 identifies that the number of messages arriving at connector 230 exceeds a threshold, one or more additional connectors may be added to connector 230. In addition, the number of message inqueues, message processors, action engine instances, and message outqueues may be increased depending on where the slowdown occurs. In such implementations, additional components may be added without requiring the addition of other additional components. For example, connectors may be added without requiring the addition of additional instances of the action engine. In some implementations, one or more components or parts of components of the bot system 220 may run on virtual machines. By running on virtual machines, additional virtual machines may be launched as desired.

[0109] As described above, building a bot system such as a user intent classification engine that can determine an end-user's intent based on end-user utterances is a challenging task, partly due to the nuances and ambiguities of natural language, as well as the dimensions of the input space (e.g., possible end-user utterances) and the size of the output space (number of intents). Therefore, it may be necessary to monitor, debug, and modify new bot systems to improve their performance and the user experience with them. Often, it can be difficult to more specifically identify the root causes of performance that falls below the desired level and to determine how to improve the bot system without using analysis or optimization tools.

[0110] In some situations, bot owners, developers, or administrators may want to monitor the bot's operational status and understand how the bot is being used and where end users are abandoning it, in order to improve the bot. For example, bot owners, developers, or administrators may want to know which bot conversations are successful and which are not in order to identify and diagnose elements of underperformance in the bot system.

[0111] In some embodiments, the analysis system may be integrated with the bot system. The analysis system may monitor events occurring during conversations between the end user and the bot system, aggregate and analyze the collected events, and provide information about the conversations graphically on a graphical user interface at different generalization levels, such as all conversations, different categories of conversations, and individual conversations. For example, the graphical user interface may display options for filtering or selecting certain types of conversations or individual conversations, and graphically display the selected information, for example, by visualizing the conversation path. The analysis system may also provide suggestions, options, or other information for improving the bot system.

[0112] Figure 3 shows an integrated system 300, according to one embodiment, which includes a bot system (such as bot system 220) and a bot analysis system for monitoring, analyzing, visualizing, and improving the performance of the bot system. As shown, the bot system may include a connector 330 and several bot engines 310, such as a dialog engine 312, an intent modeler 314, an entity resolver 316, and a custom component 318. The bot system may also include a database 340, a management application programming interface (API) 350, a user interface 354, and a UI server 352. The bot analysis system may include a collector 355, a concentration engine 360, a database 370, and a REST server 380. The bot analysis system may also include a user interface 392 and a UI server 390. The collector 355 of the bot analysis system may collect events 305 that occur in the bot system. Feedback 394 from the bot analysis system may be provided to the bot system via user interfaces 392 and 354.

[0113] Connector 330 may act as an interface between the bot system and one or more end users through one or more channels, such as channels 320 and 322. Each channel may be a messaging application, such as a messaging channel (e.g., Facebook Messenger, Facebook WhatsApp, WeChat, Line, Kik, Telegram, Talk, Skype, Slack, or SMS), a virtual private assistant (e.g., Amazon Dot, Echo, or Show, Google Home, Apple HomePod), a mobile and web app extension that extends a native or hybrid / responsive mobile or web application with chat functionality, or a voice-based input (e.g., a device or app with an interface that uses Siri, Cortana, Google Voice, or other voice input for interaction). In some embodiments, Connector 130 may normalize content from different channels so that the bot system can analyze content across different messaging application systems. Content normalization processing may include formatting content from each type of messaging application into a common format for processing. In some embodiments, the bot system may include one or more connectors for each channel.

[0114] The intent modeler 314 may be used to determine the end-user intent associated with an end-user utterance. In some embodiments, the intent modeler 314 may use a natural language processor to tag parts of speech (verbs, nouns, adjectives), discover headwords / stems (runs / running / ran -> run), and tag entities (Texas -> LOCATION) to determine the end-user intent based on one or more messages received by the bot system from the end-user. In some embodiments, the intent modeler 314 may normalize the message. For example, "Mary ran to Texas" may become "PERSON run to LOCATION". The intent modeler may also include logic to detect words with the same meaning within the end-user message. For example, if the training dataset includes "Mary ran to Texas" and "Bob walked to Detroit," and both are mapped to the same intent, and run / walk appears in the same set of intents, the intent modeler 314 may learn this for the intent resolution run=walk purpose. In one exemplary example, "Mary ran to Texas" could be "PERSON run to LOCATION," and "Bob walked to Detroit" could be "PERSON walk to LOCATION." In the illustrated example, these two sentences may be associated with the same intent because "a noun runs to a noun" is the same as "a noun walks to a noun" for the intent resolution purpose.In another example, both "I want to order a large cheese pizza" and "I want to order a small pepperoni pizza" could be normalized to "I want to order a Bots_PizzaSize Bots_Toppings pizza."

[0115] After normalization, the probability that a given word occurrence indicates a particular intent can be determined. In some cases, probabilities can be combined using basic probability arithmetic as if they were independent. For example, if "order" suggests a 20% probability of ordering a pizza and "pizza" suggests a 10% probability of ordering a pizza, the total probability is 1 - (1 - 0.2)(1 - 0.1) = 28%. Some probabilities may be based on the presence of a word, or on specific linguistic elements such as the presence of negation or personal pronouns.

[0116] Another level of rules may be template rules, which are combinations of words. In some examples, all sentences in a training dataset may automatically become rules once normalized. In such examples, the training dataset may contain a very small number of short sentences. Template rules may return a probability of 1. New rules may be generated from rules through an inductive process. For example, the following sentences, "How much did I spend last month on gas?" and "How much did I spend in May on food?" may belong to tracking spending. These sentences may be used to inductively draw a rule, "How much did I spend," which is a shared part between them. In other examples, the training dataset may contain the phrase "How much did I spend" to achieve the same result.

[0117] The above example allows for intent definitions to avoid duplication, such as variations on the named entity (e.g., "Send money to Sue" and "Send money to Bob"). However, similar sentences with one or two different words may be used for training. Similar sentences may allow the model to learn which words may have the same meaning with respect to intent resolution and which words may be common misspellings.

[0118] If a particular word or set of words (such as a verb) is important to the intent, the probability may be manipulated by having more examples that use such words (and their synonyms), and fewer examples that involve such words for other intents.

[0119] Examples can also be provided to prevent the model from making incorrect assertions. For example, certain subphrases or words that only appear in a particular intent can lead to incorrect assertions. Similarly, it can be prevented from using similar sentences belonging to different intents to synthesize broad rules for training purposes.

[0120] The entity resolver 316 can identify entities (e.g., objects) associated with an end-user intent. For example, in addition to end-user intents identified by the intent modeler 314, such as "order a pizza," the entity resolver 316 can resolve entities associated with the intent, such as pizza type and toppings.

[0121] The dialogue engine 312 may be used to process conversations between an end user and a bot system. For example, the dialogue engine 312 may respond to an end user's utterance based on the end user's intent identified by the intent modeler 314 and the entities associated with the end user's intent identified by the entity resolver 316. In some embodiments, the dialogue engine 312 may use a state machine that includes user-defined states (e.g., end user intents) and actions taken in or from a state to a state in order to process a conversation with the end user.

[0122] Custom component 318 may include customized modules for specific bot systems. For example, a financial bot may include custom components that can be used, for instance, to check balances, transfer funds, or pay bills.

[0123] Database 340 may be used to store data for the bot system, such as data for classification models and conversation logs. The management API 350 can be used by the bot system administrator or developer to manage the bot system, such as retraining classification models, editing intents, or otherwise modifying the bot system. The administrator or developer can manage the bot system using the user interface 354 and UI server 352.

[0124] Various events can be generated while a bot system is running. An event may be generated based on one or more instructions included in the bot system. For example, an event may be generated when the bot system enters a specific state, which is defined by the bot system's administrator or developer. Once an event is generated, it can be collected, stored, and analyzed by a bot analytics system. When capturing an event, additional information associated with the event may also be collected, which may indicate the current context in which the event was generated.

[0125] For example, a conversation event may be generated by the dialogue engine 312. The conversation event may include a message (called msg_received) received by the bot system from an end-user device. msg_received may include one or more of the following parameters or variables: the message content, the time the message was received by the bot system, the language of the received message, device properties (e.g., version or name), operating system properties (e.g., version or name), geolocation properties (e.g., Internet Protocol address, latitude, longitude, etc.), identification information (e.g., user ID, session ID, bot system ID, tenant ID, etc.), timestamps (e.g., device creation, device transmission, collector derived timestamps), and channels.

[0126] A conversational event may also include a message (called msg_sent) sent by the bot system to the end-user device. msg_sent may include one or more of the following: the content of the message (e.g., the text or HTML of the message), the time the message was sent by the bot system, the language of the message, the creator of the message (e.g., the bot system or the end-user device), device properties, operating system properties, browser properties (e.g., version or name), app properties (e.g., version or name), geolocation properties (e.g., Internet Protocol address, latitude, longitude, etc.), identification information (e.g., user ID, session ID, bot system ID, tenant ID, etc.), and channel (e.g., Facebook or Webhook).

[0127] The dialog engine 312 can also generate dialog state execution events. As described above, the dialog engine 312 can use a state machine to determine the flow of conversation with the end user. The state machine may include a set of states and rules for transitions between states. The dialog engine 312 can run the state machine for each end user conversation and generate a dialog state execution event for each state in which the dialog engine 312 takes a step to process the end user utterance. Attributes of a dialog state execution event may include, for example, the state name, component name, next action, entity match, intent match, variable, user query statement, response statement, execution time, communication language, device properties, operating system properties, geolocation properties, identification information, timestamp, channel, etc. The state name may be the name of the state currently being executed or an "error state". The component name may be the name of the bot component to be executed for the current state. The next action may be the next action to be executed. The entity match may be the entity to be resolved in the current message. The intent match may be the intent to be resolved by a score value. The variable may be a variable value for the current state. A query statement may be a message sent by the end user. A response statement may be a message sent to the end user. Execution time may be a timestamp of a completed state execution. Communication language may be the language of the messages in the conversation. Device and / or operating system properties may be associated with the end user interacting with the bot system. Browser and / or app properties may be associated with the end user interacting with the bot system. Geolocation properties may be the location of the end user interacting with the bot system.

[0128] An intent resolution event may occur as a result of the execution of the intent modeler 314. The intent modeler 314 can identify the end user's intent from a set of intents based on the end user's utterance, using a trained or otherwise defined classification model. The result of intent classification may be captured as an intent resolution event attribute, which may include, for example, the final intent classification result (e.g., the identified intent) and the confidence score associated with each intent in the set of intents.

[0129] Entity resolver 316 can generate entity resolver events. An entity is an object associated with an end-user intent. Entity definition rules may be determined when the bot system is created. For example, in addition to resolving an end-user intent such as "order a pizza," the bot system may also use entity resolver 316 to resolve associated entities such as pizza type, toppings, etc. Entity resolver events can be captured during entity resolution. Examples of attributes associated with an entity resolver event may include the entity name, applicable rules, search terms, resolved state, query statement, entity type, execution time, communication language, device properties, operating system properties, browser properties, app properties, geolocation properties, identification information, timestamp, channel, etc. The entity name may be the name of the entity currently being resolved. Applied rules may be, for example, precedent, succeeding, or aggregate. Search terms may be from, to, destination, origin, etc. The resolved state may be a dialog state resolved for the entity. The query statement may be a message containing entity values. The entity type may be a system or a derived entity. The execution time may be a timestamp of entity resolution. The communication language may be the language of the messages in the conversation. Device and / or operating system properties may be associated with the end user interacting with the bot system. Browser and / or app properties may be associated with the end user interacting with the bot system. The geolocation property may be the location of the end user interacting with the bot system.

[0130] Custom components can also generate events, such as predefined events or custom events. Predefined events may be properties that are captured when the custom component is executed. Examples of attributes for predefined events may include component name, event name, payload, execution time, communication language, device properties, operating system properties, browser properties, app properties, geolocation properties, identification information, timestamp, channel, etc. The component name may be the name of the custom component currently running. The event name may be called, called failed, replied to, replied failed, etc. The payload may be, in case of failure, reason for failure, stack trace, etc. The execution time may be a timestamp indicating when the event occurred. The communication language may be the language of the messages in the conversation. Device and / or operating system properties may be associated with the end user interacting with the bot system. Browser and / or app properties may be associated with the end user interacting with the bot system. The geolocation property may be the location of the end user interacting with the bot system.

[0131] Custom components can also emit custom events while they are running. Attributes of a custom event may include, for example, component name, event name, custom payload, execution time, communication language, device properties, operating system properties, browser properties, app properties, geolocation properties, identification information, timestamp, and channel. The component name may be the name of the currently running custom component. The event name may be a user-defined event name (e.g., balance search). The payload may be, for example, ("Amount": "USD 100", "Account": "Current"). The execution time may be a timestamp indicating when the event occurred. The communication language may be the language of the messages in the conversation. Device and / or operating system properties can be associated with the end user interacting with the bot system. Browser and / or app properties can be associated with the end user interacting with the bot system. The geolocation property may be the location of the end user interacting with the bot system.

[0132] Error and timeout events can also be generated by the bot system during execution. Error events can be generated when an error occurs. Timeout events can be generated when an end-user conversation has been inactive for a certain period of time that may be set in the channel.

[0133] The bot analysis system can collect events and additional information when the bot system interacts with an end user and generates corresponding events. For example, collector 355 can collect events and additional information and send the collected information to a queue. In some embodiments, collector 355 may be configurable and can be programmed to collect different events and / or event attributes as desired. For example, collector 355 may be configured to capture dialog state attributes, intent resolution attributes, entity resolution attributes, and error and timeout attributes. In some embodiments, collector 355 may also be configured to collect information about events 395 generated by systems other than the bot system.

[0134] The enrichment engine 360 ​​can validate and enrich the collected events and other information and write them to the database 370. For example, based on the collected IP addresses, the enrichment engine 360 ​​can determine the location of the end user associated with the IP address. As another example, the enrichment engine 360 ​​can extract specific features from the collected information, such as determining the web browser or channel used by the end user. The REST server 380 can analyze the enriched events and other information and generate various reports based on certain aggregate metrics 372. The reports can be displayed to the owner, administrator, or developer of the bot system on the user interface 392 via the UI server 390. The owner, administrator, or developer of the bot system may provide feedback 394 to the bot system to improve it.

[0135] Figure 4 is a simplified flowchart 400 illustrating an example of a process for developing skills in several embodiments. The process may include creating an intent in 410, training the skills in 420, creating entities in 430, integrating custom components in 440, creating a dialogue flow in 450, testing the skills in 460, routing to channels in 470, and considering insight reports for improving the skills in 480.

[0136] In 410, intents may be created for a skill. Intents describe various actions that the skill may help its end user complete. For example, if a skill enables a user to perform various banking transactions, intents for the skill might include, for example, "check balance" or "transfer money." Intents may not only describe what a skill can do, but may also be an integral part of the skill's intelligence. Each intent may have a set of typical user statements (i.e., utterances) associated with it, enabling the skill to recognize user input. These utterances may share the same meaning, but they may also be different (for example, "What's my savings account balance?" and "How much is in my checking account?").

[0137] In 420, the skill may be trained to infer user intents when parsing user input. Specifically, the skill may be trained using intents and their utterances (collectively, training data), so that the skill can resolve user input to one of the intents. The trained skill may recognize not only sample phrases belonging to each intent, but also similar phrases corresponding to each intent.

[0138] In 430, entities may be created for skills. In some embodiments, a skill may require some additional context to complete a user request. Several user requests may resolve to the same intent (for example, "What's my savings account balance?" and "How much is in my checking account?" will both resolve to the "Check balance" intent), but they are requesting different things. To clarify the requests, one or more entities may be added to the intent. Using the example of a banking skill, an entity "Account Type" that defines values ​​such as "Check" and "Regular" may enable the skill to parse the user request and respond appropriately.

[0139] In section 440, custom components may be integrated into a skill. Before integrating a component into a skill, the skill may be able to recognize user input but not be able to respond to it. A component may enable the skill to execute a function. A component may execute a function such as outputting text based on an intent resolved from an end-user message, or it may perform a task specific to a particular skill, such as checking an account balance.

[0140] In 450, a dialogue flow may be created. A dialogue flow describes how the skill responds when different intents are resolved. The dialogue flow defines the actions or behaviors that the skillbot takes, such as how the skillbot responds to user utterances, how the skillbot prompts the end user for input, and how the skillbot returns data. In some embodiments, the dialogue flow may be a flowchart written in a simple markdown language such as a version of YAML (e.g., OBotML).

[0141] In 460, skills may be tested after they have been developed and deployed in the digital assistant described above, for example. Skills may also be tested by having one or more conversations with the skill using one or more dialogue flows.

[0142] Optionally, in 470, a skill may be routed to one or more channels for user messaging and other functionalities. For example, if a skill is not added to a digital assistant, it may be added to one or more user messaging channels. Users may chat with the skill through one or more of these user messaging channels, such as various messaging platforms, copyrighted messaging apps, and web pages. The skill will run on any of these channels, whether text-only or capable of supporting scrolling of cards, UI elements, images, and other non-text content. In addition to user interface channels, a skill may be routed to other channels, such as channels linking the skill to a customer support system or channels routing notifications from external applications prompting the skill to initiate a conversation.

[0143] In section 480, developers may consider insight reports to improve skills. For example, developers may consider insight reports related to skills to understand whether customers are using the skill as intended. Insight reports may include, for example, higher-level usage metrics and conversational trends, individual views of intents, their execution paths, and conversational transcripts. Insight reports may provide different perspectives on how well the skill supports its customers and where the skill prevents customers from completing tasks. These insight reports may not only help developers quickly identify problems but also suggest user inputs that can improve the skill's intent resolution.

[0144] In some embodiments, different versions of a skill may be generated. In some embodiments, a skill may be a composite skill that includes two or more related skills to perform a more complex function as described above. For example, composite skill A may include skills B and C, and for example, the output from skill B may be used as an input to skill C.

[0145] As described above, bots may use machine learning models (e.g., intent classifiers) to map end-user utterances to specific intents. Despite widespread adoption, machine learning models, including neural network models, generally operate as black boxes that make inferences (outputs) by performing mathematical or logical operations on input data. The output of a machine learning model may include, for example, predictions or classifications and / or corresponding confidence levels. However, the output of a model or a particular inference may not provide insights into the model so that the user can understand the model's behavior and the reasoning behind the inference. Therefore, it can be difficult for the user to determine whether a model and / or a particular inference is reliable, and the user may choose to make decisions or take actions based on the inference, or to deploy the model. For example, when using machine learning for medical diagnosis or terrorism detection, one should not act blindly on predictions, as the results could be disastrous. It is also necessary to evaluate the entire model before deploying it. Currently, models are generally evaluated using accuracy metrics or other statistics based on available validation datasets. However, real-world data often differs significantly, and furthermore, evaluation metrics may not reflect product goals.

[0146] In the case of a classifier for a bot system, the bot owner or developer might want to understand, for example, why the classifier infers a particular user intent from certain user utterances while inferring different user intents from different user utterances, before deploying the classifier to the bot system. Gaining insights into the classifier may also help the bot developer reduce the training data (e.g., training utterances) used to train the classifier, or generate more useful training data more intelligently to improve the classifier.

[0147] In one embodiment, the bot system may include an intent classifier and an inference explanation engine. The inference explanation engine may provide insights into the intent classifier's inferences of individual user utterances. For example, the inference explanation engine may identify a word or combination of words from the utterance that causes the intent classifier to make an inference with a certain confidence level. The inference explanation engine may include, for example, a local proxy model or a scoped rule model.

[0148] The surrogate model is trained to approximate the predictions of the underlying black-box model. The purpose of an interpretable surrogate model is to approximate the predictions of the underlying model as accurately as possible using an interpretable surrogate model (e.g., a prediction function). The surrogate model may be obtained by (1) selecting a dataset which may be the same dataset or a subset of the same dataset or a new dataset from the same distribution used to train the underlying black-box model; (2) generating predictions for the selected dataset using the underlying black-box model; (3) selecting an interpretable model type (e.g., a linear model, a decision tree); (4) training an interpretable model using the selected dataset and the predictions from the underlying black-box model to generate the surrogate model; and (5) determining how well the surrogate model replicates the predictions of the underlying black-box model and retraining the surrogate model as necessary. The trained surrogate model may then be interpreted.

[0149] Instead of training a global surrogate model, Local Interpretable Model-Independent Explanation (LIME) focuses on training local surrogate models to explain individual predictions. In LIME, the goal is to understand why a machine learning model makes a particular prediction. LIME generates a new dataset containing variations of input samples (e.g., reordered samples) and corresponding predictions from the underlying machine learning model. An interpretable model is then trained using this new dataset. The interpretable model could be, for example, a least absolute void and selection operator (Lasso) or a decision tree. The trained interpretable model may be a good approximation (local fidelity) of the predictions of the underlying machine learning model for variations of input samples, but it does not need to be a good global approximation for all input samples. A local surrogate model may be obtained by (1) selecting a sample of interest, for which the predictions by the underlying machine learning model may need to be explained; (2) perturbing the sample of interest and generating predictions for the perturbed sample by the underlying machine learning model; (3) weighting the perturbed sample according to its proximity to the sample of interest; and (4) training a weighted interpretable model using the perturbed sample and the corresponding predictions from the underlying machine learning model. The predictions for the sample of interest by the underlying machine learning model may then be explained by interpreting the trained interpretable model.

[0150] An example of a scoped rule model is the anchoring technique. The anchoring technique can explain individual predictions of a black-box machine learning model (e.g., a classifier) ​​by finding a decision rule that sufficiently "anchores" the prediction. A rule anchors a prediction if changes in other features do not affect the prediction. One exemplary anchoring technique may utilize reinforcement learning techniques in combination with graph search algorithms to recover from local optima while minimizing the number of model calls (and therefore execution time). The anchoring technique employs a perturbation-based strategy to generate local explanations for predictions in a black-box machine learning model. Instead of using a surrogate model, the resulting explanations in the anchoring technique are expressed as easily understandable IF-THEN rules. The rules are scoped and reusable. For example, an anchor may include the concept of coverage, specifying other, possibly unknown, instances to which the rule might apply.

[0151] Finding an anchor may involve solving a search or multi-armed bandit (MAB) problem derived from the field of reinforcement learning. To find an anchor, a neighborhood or perturbation may be created and evaluated for instances to be described in one or more rounds. In each round, a perturbation matching a currently followed rule (e.g., an anchor) may be created and evaluated by calling the model. MAB may be used to efficiently explore and utilize different strategies (called arms in the analogy of a slot machine) using sequential selection. In a given setting, each candidate rule should be considered an arm that can be pulled. Each time a candidate rule is pulled, its respective neighborhood is evaluated, and thus further information may be obtained about the payoff of the candidate rule (e.g., the precision indicating how well the rule describes the instances to be described). If there is no statistical confidence that the payoff of the candidate rule exceeds a certain threshold, more samples may be used. In each subsequent round, the best candidate from the previous round may be extended with additional feature predicates that are not yet included in it.

[0152] The above process can be combined with a beam search process, which is a variation of graph search and breadth-first algorithms. The beam search process carries over the B best candidate rules from each round to the next round (where B is the beam width). These B best candidate rules are then used to create new rules. The beam search process may include up to feature count (x) rounds, since each feature may be included in the rule at most once. In each round i, the beam search process generates candidates with i predicates and selects their B best candidate rules. Therefore, by setting B higher, the algorithm may be more likely to avoid local optima, but it may use a large number of model calls and thus increase the computational load.

[0153] Techniques for explaining estimation results In some embodiments, a client-facing service for a client hosting a chatbot implements a technique for explaining inference results, as described herein. The technique described provides utility to the client (e.g., the entity hosting / owning the chatbot) by enabling the client to consider the chatbot's behavior and the results of that behavior using a comprehensive explanation. It opens up the "black box" of ML classifications that the customer should view in order to review and improve the chatbot system. In some embodiments, the technique is implemented as part of a client-facing system and interface in which the client may consider various aspects of the chatbots hosted by the client. For example, the client may, through the interface, review a list of the chatbots hosted by the client, as well as various reports and suggestions related to the functionality of those chatbots.

[0154] The chatbot list within the interface may further include information related to various conversations associated with various chatbots over a certain period. For example, a client may view various conversations, including utterances, received by a chatbot over a certain period. These various conversations, including utterances, may be associated with specific inference results or the absence of inference results predicted by the chatbot. Inference results are the result of the chatbot's interpretation of a particular utterance. For example, inference results may include, but are not limited to, the chatbot's classification of the utterance (e.g., prediction of intent / services corresponding to the utterance), or confidence thresholds for classification decisions (e.g., the confidence threshold level required for a particular intent / service to predictively associate with the utterance). The client derives significant value and usefulness from understanding how inference results are generated for a particular utterance.

[0155] Figure 5 is a simplified flowchart 500 illustrating an example of a process for explaining inference results according to one embodiment. The process shown in Figure 5 may be implemented by software (e.g., code, instructions, programs), hardware, or a combination thereof, executed by one or more processing units (e.g., processors, cores) of each system. The software may be stored on a non-temporary storage medium (e.g., on a memory device). The methods presented in Figure 5 and described below are intended to be illustrative and non-limiting. Figure 5 shows various processing steps occurring in a particular sequence or order, but this is not intended to be limiting. In certain alternative embodiments, those steps may be executed in some different order, or some steps may be executed in parallel. In some embodiments, such as the embodiments shown in Figures 1 to 4, the process shown in Figure 5 may be executed by a preprocessing subsystem to explain inference results according to one embodiment. The machine learning model may include, for example, a neural network model or any other machine learning model, and may be used, for example, as a classifier that classifies user intents based on user utterances.

[0156] In 510, a request is received to describe the inference results for an utterance. Specifically, the request is received from an entity to obtain a report about the inference results that the chatbot generates as a result of parsing the utterance. For example, a client hosting a chatbot may generate and send a request, which is received as part of process 500, to generate a report for the client's review that details the inference results output by the chatbot at some prior time when the utterance was entered.

[0157] As described herein, inference results are results or predictions that include some data about how a chatbot application parses and interprets an utterance. In some embodiments, a request describing an inference result is received from an entity such as a client hosting a chatbot application that derived an inference result about an utterance. For example, a chatbot may receive an utterance from a client's customer and, in response, take some action based on the customer's utterance. The request may be submitted, for example, via a command line or user interface accessible by the client. The process that takes the action generates inference data that is useful in describing how the chatbot application took the customer's utterance and took the action. The inference result may include, for example, a classification result attributable to the utterance based on the chatbot application's processing of the utterance, which is useful for the client to understand how the chatbot application works.

[0158] In some embodiments, a client-hosted chatbot application includes a classifier that classifies user intents based on user utterances. For example, the classifier may be a subsystem of the chatbot system that includes a machine learning model trained to take utterances as input and output a predicted classification of the utterances. In some embodiments, the classifier is a trained neural network machine learning model configured to map the inputs contained in the utterances to various nodes of a neural network model. The various nodes of the neural network model may then be processed to produce output results corresponding to the classification of the utterances. The classification may be, for example, an intent / service that the chatbot predicts the utterances will correspond to. In some embodiments, the classification results of the above inference results are generated by the classifier.

[0159] In some embodiments, the system stores data in computer memory in real time regarding the reasoning results and classifications made by the chatbot system. For example, when the chatbot receives an utterance, the system may store the received utterance. After the chatbot outputs prediction data, for example, regarding a predicted intent, the system may store the prediction data as part of the reasoning results. The client may review this information stored in the system to determine the utterance received by the chatbot and the resulting reasoning results that were generated. If the client determines that there is a problem with the chatbot, or is otherwise dissatisfied with the chatbot's functionality, the client may request a report explaining the reasoning results for the utterance. In some embodiments, the system that stores data regarding the utterance and reasoning results is the same system that performed process 500.

[0160] In 520, the classification result of the utterance is obtained from the classifier. Specifically, the classification result may be replicated when the utterance is input to the chatbot, using a classifier such as the chatbot system's classifier in 510 or a similar classifier. For example, if the utterance and chatbot system specified by the client in the reporting request are known, the utterance may be input to the chatbot system or a similar system, and the classification result may be obtained as output (e.g., an intent predicted to correspond to the utterance).

[0161] In some embodiments, the classification result is obtained by inputting the utterance into a classifier, for example, into a machine learning model included in the classifier. By obtaining the classification result, the system is able to determine the classification result as part of the inference result required, as described in step 510. The classification result may then be obtained as output for performing further steps of process 500. In some embodiments, the classification result is stored as part of a previous operation of the chatbot application utilizing the classifier. For example, when a client customer sends an utterance to the chatbot application, the chatbot application's classifier may generate a classification result and store the classification result in a local or external system, such as a client database. The classification result may then be retrieved from the database in 520.

[0162] In some embodiments, the classification results include classification data corresponding to predicted intents / services that correspond to utterances, as predicted by the chatbot. For example, the classification data may include data about the utterance, one or more predicted intents / services related to the utterance, and one or more confidence values ​​corresponding to the degree of confidence that one or more predicted intents / services correspond to the utterance. In some embodiments, the confidence values ​​correspond to the level of correspondence between nodes in a trained neural network machine learning model, which is used by the classifier when parsing the utterance.

[0163] In some embodiments, the classifier is a facsimile classifier that mimics the operation of the chatbot system. For example, a system may store the classifier as individual classifiers without corresponding chatbot models. The system may use the classifiers to obtain classification results using utterances without the entire chatbot system needing to parse the utterances. Since only the classifiers may be required to perform step 520, the system may store, for example, a trained machine learning model used by the chatbot system that corresponds to the classifiers. In some embodiments, the system hosting the facsimile classifiers is the same system that performs process 500.

[0164] Once a classification result is obtained, in 530, a new set of anchors containing anchor words related to the utterance is selected, and a synthetic utterance is generated by replacing non-anchor words in the utterance with similar words. Specifically, a subset of words that make up the utterance are grouped into a set of anchors that represent groupings of words that are relatively important for generating the inference result. Words that are not in the set of anchors are replaced with similar words to form a synthetic utterance. For example, one or more words in the utterance may be designated as anchor words that are predicted to be relatively more important for classification than other words in the utterance. Words that are not designated as anchors are then replaced with similar words to form a new synthetic utterance that is semantically different from the original utterance but contextually similar.

[0165] The result of substituting non-anchor words is the generation of a synthetic utterance that retains some similarity to the original utterance through the substitution of synonyms. In some embodiments, selecting one or more anchor words from an utterance involves determining one or more words in the utterance that correspond to a high degree of uniqueness (e.g., fewer known synonyms for a word or phrase). In some embodiments, the anchor word is the word least likely to be substituted within the utterance to form a similar utterance. For example, the utterance "I want to order a pizza." contains the words "I," "want," "to," "order," "a," and "pizza." In this phrase, the words "I" and "pizza" correspond to fewer alternative synonyms than the other words and are therefore more likely to be candidates for the anchor word. For example, the phrase "I desire to purchase one pizza." is similar to the phrase "I want to order a pizza." because the former is more likely to correspond to a similar intent than the latter. In contrast, the phrase "John wants to order a treehouse" is not a similar phrase, despite the fact that the words "want(s)," "to," "order," and "a" are retained as anchor words between the two phrases.

[0166] The selection of a new set of anchors may be carried out in any manner necessary to enable the generation of the synthesized utterances described herein. In some embodiments, selecting a set of anchors involves using a beamsort algorithm to select potential candidates for anchor words in the utterance. For example, one process may determine one or more best candidates for anchor words based on some heuristic, such as the uniqueness metric described above. One or more additional words may then be selected to constitute the set of anchors. All words not selected for the set of anchors are non-anchor words that are replaced by non-anchor words in the synthesized utterances. In some embodiments, a set of anchors is selected only if it satisfies a heuristic threshold for anchor selection (for example, the average uniqueness score of each word in a group must satisfy or exceed the uniqueness threshold before the group can be selected as the set of anchors).

[0167] The 530 steps enable the generation of new synthetic utterances that differ from the original utterance but may be used to test inference results regarding similarities and differences between utterances after classification. By retaining anchors as constants in these new utterances, the new utterances will reveal how specific inference results for an utterance are determined at the word level. For example, these new synthetic variations of an utterance may also be classified by the classifier to determine one or more predicted intents for the new utterance.

[0168] In step 540, the classification result of the synthesized utterance is obtained from the classifier, along with the confidence level of the selected set of anchors. Specifically, each synthesized utterance generated in step 530 is input into the classifier model to obtain a prediction of the classification result for that synthesized utterance. For example, for each synthesized utterance generated in step 530, the synthesized utterance is input into an ML model that operates as part of the classifier used in step 520 to obtain a classification result for the synthesized utterance. If the ML model is trained to output a certain predicted intent / skill given a given input utterance, then the ML model will output one or more predicted intents / skills given one or more synthesized utterances to be input into the ML model. Based on the predicted intents / skills for the synthesized utterances, a confidence value is determined for the set of anchors selected in step 530. The confidence value may correspond to any metric related to the generated synthesized utterances and the predicted intents / skills for those synthesized utterances. In some embodiments, the confidence value for a set of anchors is the percentage of generated synthesized utterances whose classification result is the same as the original classification result. For example, if 100 synthesized utterances are generated for a set of anchors, and 87 of these utterances are determined to correspond to predicted intents that match the predicted intent classification results obtained in 520, the confidence value for that set of anchors may be 87% or 0.87.

[0169] In some embodiments, each predicted classification may correspond to a confidence metric associated with a confidence level corresponding to a particular predicted intent / skill in an input utterance. For example, in some embodiments where the ML model is a trained neural network ML model, the confidence metric may be based on a numerically derived metric of correspondence between nodes in the trained neural network when processing the input utterance. In these embodiments, a higher confidence metric would correspond to a relatively higher degree of correspondence between nodes in the trained neural network when the predicted intent / skill is output. In some embodiments, the confidence value for a set of associated anchors may be derived in part from the confidence metric of each of the synthesized utterances produced for that set of associated anchors.

[0170] The classification results of synthesized utterances are useful for determining the aspects of a chatbot that are used to predict intents / skills based on utterances. For example, both the original and synthesized utterances may be associated with the same predicted intent / skill. As a result, a reasonable inference can be made that anchor words shared between the original and synthesized utterances are a major element in the inference results generated for the original utterance. Conversely, if the original and synthesized utterances do not share the same predicted intent / skill, a reasonable inference can be made that either anchor words shared between the original and synthesized utterances are not a major element in the inference results, or the ML model is malfunctioning when processing a particular word or phrase within the utterance. For these and similar reasons, the classification results of the generated synthesized utterances are extremely valuable in determining the inference results for the original utterance.

[0171] In step 550, the set of anchors associated with the highest confidence level is identified. Specifically, the confidence values ​​of the set of anchors corresponding to the highest confidence level may be selected to be included in the report to be sent in response to the request received in step 510. For example, process 500 may include generating multiple sets of synthesized utterances based on multiple selected sets of anchors. The set of anchors with the highest confidence value is most likely to be the most valuable when viewing the report on the inference results. Therefore, in step 550, the classification results and confidence levels generated and obtained in step 540 may be compared to confidence levels previously generated for other sets of anchors. The set of anchors with the highest confidence level among the multiple sets of anchors being compared, and the corresponding synthesized utterances, may be identified and stored for later inclusion in the report.

[0172] In some embodiments, instead of selecting a single highest confidence value among each set of anchors, multiple sets of anchors may be included in the report based on a threshold confidence level. For example, a threshold confidence value may be used as a reporting cutoff to determine whether a set of anchors is included in the report or not. Each set of anchors and its corresponding synthesized utterance is included in the report if the confidence level of each set of anchors exceeds the confidence value threshold.

[0173] The operations in 530-550 may be performed iteratively over multiple rounds, for example, in the beam search process described above, and different word combinations may be used as anchors in each round. In one example, each word of an utterance may be used as an anchor word to generate variations of the utterance as a synthetic utterance in the first round, the synthetic utterances may be classified by a classifier, and anchors with confidence levels may be selected after the first round. In some embodiments, in a second round, any combination of an anchor word selected in the first round and another word in the utterance may be used as an anchor word for a certain anchor to generate more variations as a synthetic utterance, the synthetic utterances may be classified by a classifier, the classification results may be compared with the classification results of the original utterance, and anchors with the highest confidence levels may be selected after the second round.

[0174] In some embodiments, in the second round, new combinations of anchor words not selected in the first round may be selected and used to generate synthesized utterances, which are then classified and compared as described above. The iterations of operations 530–550 may be stopped when all significant anchors have been selected and synthesized utterances have been generated for those anchors. For example, a uniqueness threshold may be employed as part of the beam search. If all sets of anchors satisfying the uniqueness threshold have been selected in 530, the iteration may be stopped at 550 and process 500 may proceed to 560.

[0175] In 560, a report may be generated based on the results of 550 across multiple rounds. The report may include one or more sets of anchors and the corresponding confidence levels for those sets of anchors. In some embodiments, the report may include classification results for synthesized utterances. Based on the report, the user may understand which words or combinations of words cause the classifier to infer a particular intent from the original utterance. For example, the report may show the original utterance "I want to order a pizza." along with a synthesized utterance such as "I desire to purchase one pizza." The report may show, for example, that both the original and synthesized utterances were predicted to correspond with similar intents, for example, an intent named "ORDER FOOD," with similar confidence levels. Thus, the user may infer from the report that the shared anchor words "I" and "Pizza" are key factors in determining the same intent and are the reason for the relatively high confidence values ​​for that particular set of anchors.

[0176] Table 1 below shows an example of the report description section in a JSON file. The report includes the classifier's inference result for the original utterance "money available". The report also includes, for each anchor, the variations of the original utterance generated using the anchor, the corresponding classification result by the classifier, and the respective precision determined based on the number of matches and mismatches between the classification result of the variations of the original utterance and the classification result of the original utterance. For example, if the anchor contains only the word "available", the variations of the original utterance may include "# available", "amenities available", "bicycles available", "+ available", "2017 available", etc. The classifier may classify these utterances as being associated with the intent "balances", which matches the classification result of the original utterance. However, the classifier may classify the utterances “commercially available,” “only available,” “universally available,” and “Currently available” as “unresolved intent,” which do not match the classification result of the original utterance. The report indicates an accuracy of approximately 47.5% for associating them with the anchor “available.” When the anchor contains both the words “money” and “available,” the accuracy is approximately 100%.

[0177] Table 1 (Explanatory Reporting Section in JSON)

[0178] [Table 1-1]

[0179] [Table 1-2]

[0180] [Table 1-3]

[0181] [Table 1-4]

[0182] [Table 1-5]

[0183] [Table 1-6]

[0184] [Table 1-7]

[0185] Figure 6 shows an embodiment of System 600 for illustrating inference results according to one embodiment. For example, System 600 may be a system for triggering an implementation of Process 500 or a similar process described herein. System 600 includes a user interface 605, a management API 610, an intent server 615, a gateway 620, a workflow engine 625 (e.g., a server), and a database or cloud storage device 630. The intent server 615 may include a classifier that infers the user's intent based on the user's utterance. The workflow engine 625 may execute an application 635 (e.g., in Python code) to perform the classification explanation described above, for example, with respect to Process 500 in Figure 5.

[0186] The client may use asynchronous calls to submit explanation requests along with classification queries (e.g., utterances) to the gateway 620 via the user interface 605 and the management API 610. For example, the client may use a client device to interact with the user interface 605 to view aspects of the chatbot, or otherwise interact with services associated with the chatbot. In some embodiments, the client may use system 600 to view utterances processed by the chatbot at a previous time and the inference results generated for those utterances. For example, the client may use a particular chatbot as part of its own services.

[0187] The client may notice that a particular utterance has been processed and that an unexpected inference result has been generated for that utterance (for example, the utterance "I would like to order a pizza" was predicted to correspond to a request to process a customer refund). The client may then request an explanation for the inference result based on the utterance, as described in 510. The client enters this request into user interface 605, which then routes the request to management API 610. Management API 610 may be an API that operates as part of a chatbot explanation system separate from the client device used by the client.

[0188] In some embodiments, upon receiving an asynchronous call by the management API 610, the management API 610 forwards the call to the intent server 615 for routing to the gateway 620. In other embodiments, upon receiving an asynchronous call by the management API 610, the management API 610 forwards the call directly to the gateway 620. When the intent server 615 receives the call, it forwards the call to the gateway 620. The gateway 620 provides a job identifier for the explanation request, and the job identifier is returned to the user via the intent server 615, the management API 610, and the user interface 605. The gateway 620 initiates a workflow to process the explanation request by forwarding the explanation request and its associated job identifier to the workflow engine 625. The job identifier may be an identifier corresponding to a report generation process, such as process 500.

[0189] The workflow engine 625 executes application 635 to perform classification explanations, for example, using the beam search technique described above. Application 635 may, in one or more rounds, generate new utterances (e.g., approximately 500-2000 new utterances on average) that are variations of the original utterance (i.e., the classification query in the explanation request) using various anchors, and supply the new synthesized utterances to the intent server 615. In some examples, application 635 uses a cache to ensure that only unique utterances are submitted to the intent server 615. For example, some of the synthesized utterances generated by application 635 may be known utterances previously organically received from a user of the chatbot system. The intent server 615 exposes an API to retrieve a list of intents and new utterances from the workflow engine 625. Based on the retrieved list of intents, the workflow engine 625 generates classification results for the new utterances. Application 635 may, according to a process such as process 500, determine the confidence level of an anchor based on the classification results in the previous round, select the anchor with the highest confidence level, determine a new anchor based on the selected anchor, generate a new query based on the new anchor, and supply the new query to intent server 615. The results, including the anchor, the new synthesized utterance, the classification result of the synthesized utterance, the confidence level associated with the anchor, or any combination thereof, may be sent to a database or cloud storage device 830 (e.g., object store) along with a job identifier for storage by workflow engine 825 and / or intent server 815.

[0190] Figure 7 shows an embodiment of system 700 for retrieving explanations of inference results according to one embodiment. Similar to system 600, system 700 may include a user interface 705, a management API 710, an intent server 715, a gateway 720, and a database or cloud storage device 730. A user may submit an explanation result request to gateway 720, along with a job identifier, using an asynchronous call, for example, via user interface 705 and management API 710. In some examples, upon receiving an asynchronous call by management API 710, management API 710 forwards the call to intent server 715 for routing to gateway 720. In other examples, upon receiving an asynchronous call by management API 710, management API 710 forwards the call directly to gateway 720. If intent server 715 receives the call, intent server 715 forwards the call to gateway 720. The gateway 720 retrieves results based on the job identifier, including the anchor, synthesized utterance, the classification result of the synthesized utterance, the confidence level associated with the anchor, or any combination thereof. The results are returned to the user via the intent server 715, the management API 710, and the user interface 705. Multiple explanation result requests can be executed simultaneously using multiple job identifiers.

[0191] Figure 8A shows an example of simulated CPU usage during an explanatory calculation in one embodiment. In the illustrated example, the utterance to be classified and explained is "another agent question." The execution time is approximately 234 seconds. Specifically, Figure 8A shows the level of central processing unit (CPU) resource usage for performing the explanation of the inference result compared to the time required to complete the request.

[0192] Figure 8B shows an example of simulated memory usage during an explanatory calculation according to one embodiment. Specifically, Figure 8B shows the level of memory resource usage for performing the explanation of the inference result compared to the time required to complete the request.

[0193] Figure 9A shows an example of simulated CPU usage during an explanatory calculation according to one embodiment. In the illustrated example, the utterance to be classified and explained is, "I would like to have a large cheese pizza with a tall glass of coke." The execution time is approximately 933 seconds. Specifically, Figure 9A shows the level of central processing unit (CPU) resource usage for performing the explanation of the inference result compared to the time required to complete the request.

[0194] Figure 9B shows an example of simulated memory usage during an explanatory calculation according to one embodiment. Specifically, Figure 9B shows the level of memory resource usage for performing the explanation of the inference result compared to the time required to complete the request.

[0195] Figure 10A shows an example of simulated CPU usage during an explanatory calculation according to one embodiment. In the illustrated example, the utterance to be classified and explained is "question". The execution time is approximately 101 seconds. Specifically, Figure 10A shows the level of central processing unit (CPU) resource usage for performing the explanation of the inference result compared to the time required to complete the request.

[0196] Figure 10B shows an example of simulated memory usage during an explanatory calculation according to one embodiment. Specifically, Figure 10B shows the level of memory resource usage for performing the explanation of the inference result compared to the time required to complete the request.

[0197] Techniques for generating training datasets As described above, generating reports that include instructions for synthesized utterances, predicted intents / skills for those synthesized utterances, and corresponding confidence values ​​for those synthesized utterances is useful in determining how the chatbot system is processing the received utterances. The reports may also be used to identify any flaws in the chatbot system. For example, a chatbot system may mishandle certain anchor words in an utterance and predict an inaccurate intent / skill to associate with that utterance. Eliminating the "black box" of ML decisions is very useful in identifying chatbot system flaws, but the chatbot system would then have to be retrained to prevent the inaccurate predictions from occurring again.

[0198] The process of training and retraining chatbots is difficult and resource-intensive. For example, areas of the chatbot ML model that are flawed must first be identified. Then, a training dataset that effectively mitigates these flaws must be generated. The training dataset includes a set of training utterances and the "ground truth" intents / skills corresponding to the training utterances. The ML model is then trained using the training utterances with the ground truth intents / skills to improve the model's ability to better predict intents / skills for utterances. Obtaining training datasets is difficult, and manually selecting and constructing training datasets is inefficient and resource-intensive.

[0199] Figure 11 is a simplified flowchart illustrating an example of a process for retraining a model using synthesized utterances according to one embodiment. The process shown in Figure 11 may be implemented by software (e.g., code, instructions, programs), hardware, or a combination thereof, executed by one or more processing units (e.g., processors, cores) of each system. The software may be stored on a non-temporary storage medium (e.g., on a memory device). The method presented in Figure 11 and described below is intended to be illustrative and non-limiting. Figure 11 shows various processing steps occurring in a particular sequence or order, but this is not intended to be limiting. In certain alternative embodiments, those steps may be executed in some different order, or some steps may be executed in parallel. In some embodiments, such as those shown in Figures 1 to 5, the process shown in Figure 11 may be performed by a preprocessing subsystem to hold a model using synthesized utterances according to one embodiment. The machine learning model may include, for example, a neural network model or any other machine learning model, and may be used, for example, as a classifier that classifies user intents based on user utterances.

[0200] More specifically, process 1100 describes the process for obtaining a training dataset and retraining the ML model to predict intents / skills based on input utterances. As described above, referring to Figure 5 and process 500, several synthetic utterances are generated corresponding to a set of anchors from the original utterances and the ML model that will operate as part of the chatbot system. The synthetic utterances generated as part of process 500 may be used to identify flaws in the ML model, specifically in handling the original and similar utterances. In this case, it is beneficial to retrain the ML model using the original and / or similar utterances, as well as the corresponding ground truth, to improve the model function. It will be understood that the synthetic utterances generated in steps 530-550 are good candidates for retraining the machine learning model.

[0201] Process 1100 may optionally begin at 550 of process 500, where the set of anchors with the highest confidence level is identified. In some embodiments, process 1100 begins when process 550 has completed the last iteration of blocks 530-550 and the set of anchors with the highest confidence level has been identified. The set of anchors and the corresponding synthesized utterances are then used as part of process 1100 to generate a training dataset for retraining the ML model. In some embodiments, process 1100 operates in parallel with 560 in Figure 5. Specifically, the synthesized utterances may be used in process 1100 at the same time that the report containing the set of anchors is generated.

[0202] Process 1100 continues at step 550 of process 500, where one or more utterances are received, each utterance corresponding to a confidence level among one or more confidence levels. In some embodiments, where process 1100 begins at step 550 of process 500, the one or more utterances are synthetic utterances generated as part of the report generation process. For example, in response to receiving a request describing an inference result, the system performing process 500 may also automatically generate a retraining dataset for a particular ML model associated with the utterances specified in the request.

[0203] In other embodiments, one or more utterances are utterances stored in computer memory at a previous point in time. For example, one or more utterances may be utterances corresponding to predicted intents / skills predicted by the ML model, which will be retrained according to process 1100. The utterances may be stored previously for future training of the ML model, for example, as part of a previous iteration of process 500.

[0204] Each utterance corresponds to a certain confidence level. As described herein, the confidence level represents the strength of the correlation of a set of anchors to a particular utterance, based on the proportion of synthesized utterances for that set of anchors that are classified into the same intent / skill as the particular utterance. For example, each synthesized utterance may correspond to a determined confidence value of the set of anchors used to generate each synthesized utterance. In some embodiments, each synthesized utterance independently corresponds to a certain confidence value. For example, the confidence level of a synthesized utterance may be determined when given a particular utterance as input and making a prediction of a particular intent / skill using the level of correspondence between nodes in a trained neural network ML model. In some embodiments, where process 1100 begins at step 550 of process 500, one or more synthesized utterances already correspond to a confidence level determined at step 540 of process 500.

[0205] In step 1120, one or more confidence values ​​received in step 1110 are compared to a threshold confidence value. Specifically, each of the one or more confidence values ​​is compared to the threshold confidence value to determine a subset of confidence values ​​that do not meet or exceed the threshold. In various embodiments, the threshold confidence value is a value that corresponds to a relatively acceptable level of correspondence for the predicted intent / skill for a particular utterance. For example, a training dataset is often most useful for training an ML model when the ML model has not already predicted the intent / skill associated with the utterances in that training dataset with a high degree of confidence. The result of training would be that, after training, the ML model predicts the intent / skill for similar utterances with a higher level of confidence. Therefore, it is most useful to select utterances corresponding to confidence levels below the desired confidence threshold to be included in the training dataset. Utterances corresponding to confidence levels above the threshold may be useful when training the ML model, but they will not have as efficient an impact on model training as utterances corresponding to confidence levels below the threshold.

[0206] In some embodiments, the confidence level threshold may be determined based on some input from entities associated with the chatbot system and the ML model. Specifically, the entity that owns or operates the chatbot system, and therefore the ML model, may specify a confidence threshold to optimize the generation of new training datasets. For example, a client hosting the chatbot system may decide to set the threshold confidence value to a value that allows for the generation of a sufficient training dataset, but keeps the training dataset sufficiently small to retrain the model within a certain amount of time. In some embodiments, the threshold confidence value may be suggested to the client for use based on metrics associated with the utterances received in 1110.

[0207] In step 1130, a subset of training utterances is determined, and each utterance in the training utterance subset corresponds to a confidence value less than or equal to the threshold confidence value. Any utterances within one or more received utterances that correspond to a confidence level less than or equal to the threshold confidence value will be included in the training utterance subset and will ultimately be used to retrain the model. For example, the threshold confidence value may be set to 0.80 (or 80% confidence). The confidence value corresponding to the first utterance of one or more utterances may be 0.90 (or 90% confidence that the utterance corresponds to a specific intent / skill). The confidence value corresponding to the second utterance may be 0.67 (or 67%). The first utterance is not determined to be in the training utterance subset because its corresponding confidence level is greater than the threshold confidence value. The second utterance is determined to be in the training utterance subset because its corresponding confidence level is less than or equal to the threshold confidence value. In some embodiments, the threshold may further specify a criterion for which utterances may be acquired, for example, that a subset of training utterances includes only those utterances corresponding to confidence values ​​less than or equal to the threshold confidence value. It will be understood by those skilled in the art that any criterion for the threshold may be used. For example, in an alternative embodiment, only utterances with probabilities above the threshold may be selected, in particular for the purpose of fine-tuning the ML model to predict specific intents / skills for utterances. For the client, it is often important to understand which utterances correspond to less reliable predictions in order to understand how to retrain the ML model. In some cases, the client may also request explanations for utterance classifications with higher confidence values ​​in order to understand why the model predicts intents / skills with higher confidence.

[0208] In step 1140, one or more training categories are determined that correspond to a subset of training utterances. Specifically, one or more training categories are ground truths associated with a subset of training utterances used to train the ML model. For example, the client may access a list of training utterance subsets. The client may then select ground truth intents / skills for each utterance in the training utterance subset to form a complete training dataset. In some embodiments, one or more training categories may be represented using an interactive list of possible intents / skills to which the utterance may correspond as a correct ground truth prediction by the ML model. Ground truth intents may be selected from an interactive list of one or more known intents associated with the ML model. Once selected, ground truth intents are used to create a new set of training data. An interactive list of ground truth intents to be associated with a subset of training utterances is described here and below, including in Figure 12.

[0209] In step 1150, the model is trained using a subset of training utterances based on one or more determined training categories. The new set of training data may be input into one or more ML models to train one or more ML models. For example, the determined subset of training utterances is used to retrain an ML model that has been trained according to the ground truth training categories determined in step 1140. One result is that utterances previously classified as inaccurately predicted intents / skills, as determined by reports such as those generated in step 560, will be used to train the ML model to more efficiently predict the correct intents / skills for similar utterances. Another result is that utterances previously classified as correctly predicted intents / skills, but with confidence levels below an acceptable level of confidence (i.e., threshold confidence value), will be used to further enhance the correct model prediction behavior.

[0210] In various embodiments not shown in Figure 11, one or more new synthetic utterances may be generated in 1110 that are similar to one or more received utterances and are based on a threshold confidence value. For example, one or more new synthetic utterances may be generated based on word or phrase substitution, similar to step 530 of process 500, in order to create a more comprehensive training dataset for retraining the model. The new synthetic utterances may be based on anchor substitution or other techniques as described above in 530. For example, one or more new synthetic utterances may be generated in response to determining that a subset of training utterances is below a threshold confidence value. These artificial utterances may be included in the set of retraining data and may correspond to determined training categories as set by the user. For example, if the training dataset must contain utterances of a certain size, one or more new synthetic utterances may be generated and used to supplement the training dataset. In other embodiments, the threshold confidence value may be automatically increased or decreased to meet the required number of utterances in the training dataset.

[0211] In various embodiments not shown in Figure 11, a subset of training utterances undergoes a context-based filtering process before being used to retrain the ML model. The context-based filtering process refines the utterances to fit the environment-specific context. For example, a customer service chatbot may belong to a context environment that prohibits vulgar or offensive utterance patterns. One or more acquired utterances and / or synthesized utterances may be filtered to remove all vulgar or offensive language from the utterances before retraining. For example, to reduce vulgar bias in a newly trained ML model, vulgar words may be replaced with similar, non-vulgar words before training the model. In another example, a machine learning model may belong to a context environment that prohibits the inclusion of personally identifiable information in any response to a query. A subset of training utterances may be filtered to replace any proper nouns or personally identifiable information within the utterances with general information / words. The filtering process may include any filtering to achieve the goals of the environment-specific training process.

[0212] Blocks 1110-1150 of process 1100 may be executed iteratively if a subset of training utterances is iteratively generated based on the results of previous retraining. For example, after retraining the model 1150, the utterances in the training dataset may be modified to generate a dataset of test utterances similar to the subset of training utterances. The test dataset may be input to the newly trained ML model to predict the classification for each utterance in the test dataset. The utterances in the test dataset and the corresponding confidence values ​​for the utterances may then be used to repeat step 1110 of process 1100 to further retrain the ML model. In various embodiments, operations 1110-1150 are repeated until the confidence values ​​for the generated utterances no longer fall below a threshold.

[0213] Figure 12 shows an exemplary interface for retraining a model using synthesized utterances according to one embodiment. Specifically, Figure 12 shows an exemplary interface 1200 for generating a training dataset for retraining a machine learning model. The interface shown in Figure 12 may be used by entities such as a client or user to generate a set of training data for improving the functionality of an ML model. Interface 1200 includes a threshold field 1210. The threshold field 1210 may be a field that displays a threshold confidence value used in process 1100. The threshold field 1210 may be an interactive portion of interface 1200 that a client may interact with to input a threshold confidence value. For example, as described above with reference to Figure 11, the threshold confidence value may be set to determine a subset of training utterances that do not correspond to a confidence level above the threshold confidence value (e.g., the “confidence score” shown in Figure 12).

[0214] As shown in Figure 12, one or more received utterances are displayed in a comprehensive list on the interface 1200. For example, as shown in Figure 12, the interface 1200 includes an utterance list 1220. In some embodiments, the utterance list 1220 is a list of one or more received utterances, such as the utterance received in step 1110 of process 1100. With respect to Figure 12, it should be noted that the utterances obtained and displayed in the utterance list 1220 may be variations of similar utterances and may be the result of the process for synthetic utterance generation described above with respect to process 500, specifically block 530. The interface 1200 also includes a predicted intent list 1230. The predicted intent list 1230 is displayed together with the utterance list 1220 to show the intents that the displayed utterances are predicted to correspond to as part of the comprehensive interface. For example, the utterances shown in the utterance list 1220 are shown together with the predicted intents in the intent list 1230 that were previously predicted in a classification result step such as 540 of process 500.

[0215] Interface 1200 includes a confidence score list 1240. The confidence score list 1240 may be displayed as part of a comprehensive interface, along with an utterance list 1220 and a predicted intent list 1230, to indicate the corresponding confidence levels related to the predicted intents and input utterances that are displayed together. Interface 1200 includes an interactive intent selector 1250. The interactive intent selector 1250 is an interactive element of interface 1200, which allows a client using the interface to select a ground truth intent corresponding to a corresponding utterance for inclusion in a training dataset, for example, as part of process 1100. The utterance selected in the interactive intent selector 1250 may be used as a ground truth intent for a corresponding utterance, and if that utterance is associated with a confidence score less than or equal to its confidence score, it may result in the inclusion of that utterance in a subset of training utterances. Thus, a client using interface 1200 may generate a comprehensive training dataset with utterances without having to manually create new training utterances from scratch.

[0216] Figure 13 shows an exemplary interface for retraining a model using synthesized utterances according to one embodiment. Specifically, Figure 13 shows the comprehensive interface 1200 of Figure 12 with additional interactive fields. The threshold input field 1300 is an interactive threshold field into which a specified threshold confidence value may be entered. The threshold input field 1300 may correspond to a field after the user has interacted with the threshold field 1210. A user of the interface shown in Figure 13 may use the interface to perform the methods described herein, including the process described in Figure 11. For example, the user may first specify in the threshold input field 1300 a threshold confidence value that will be used as part of process 1100, specifically as blocks 1120 and 1130. The entered threshold confidence values ​​are used to determine which subset of utterances to include in the training dataset for training the ML model.

[0217] In various embodiments not shown in Figure 13, buttons or other interactive elements may be associated with displayed utterances, which, when interacted with, trigger the generation of synthetic utterances similar to the displayed utterances. The client may select an interactive button or element on the interface to initiate the generation of a set of training datasets based on the input made by the client. In various embodiments, instead of only utterances associated with confidence values ​​below a threshold confidence value, only utterances manually selected by the client using checkboxes next to displayed utterances will be used in the retraining data. An example of generating synthetic utterances according to a particular embodiment is shown below in JavaScript Object Notation (JSON) script. For example, the generation of synthetic utterances described by the following script may be used in synthetic utterance generation processes such as 530 and / or process 1100.

[0218] Table 2: Explanation section for examples of generating synthesized utterances (in JSON).

[0219] [Table 2-1]

[0220] [Table 2-2]

[0221] [Table 2-3]

[0222] Specifically, the steps describe generating similar synthesized utterances based on a given utterance. These steps correspond to creating a set of synthesized utterances (utterances not manually written or taken from an utterance repository) for retraining the model. The synthesized utterances are generated as part of the retraining data generation and added to the data object "utterances_with_similar_predictions". The utterances are added with the "b'Add" command.

[0223] A combined interface for requesting reports and generating training datasets. Figure 14 shows an exemplary interface for illustrating inference results as part of a training dataset generation process, according to one embodiment. Specifically, Figure 14 shows an interface 1400, according to several embodiments, for displaying a comprehensive analysis of one or more inference results given an utterance. For example, Figure 14 includes an interactive interface 1400 that displays inference results associated with predictions made by a machine learning model given some input utterance. Interface 1400 displays a list of various inference results displayed according to some criteria. As shown in Figure 14, all utterances whose inference results include the intent "Unresolved Intent" matching the classification of the input utterance are displayed. The corresponding utterances are displayed on interface 1400 along with their corresponding inference results.

[0224] The inference result includes the intent predicted to most closely correspond to the utterance, along with other intents that likely correspond to a given utterance. For example, the first line displays the utterance "add another song to the Cita Romntica playlist" along with various aspects of the utterance as determined in the inference result. In addition to the "unresolved intents" predicted to correspond to the utterance, a "win rate" is also displayed. The win rate is a score corresponding to the predicted probability that the result intent corresponds to the utterance. In some embodiments, the win rate is the calculated difference between a first confidence value representing the primary intent / skill predicted to correspond to the utterance and a second confidence value representing the secondary intent / skill predicted to correspond to the utterance.

[0225] The inference report also includes an intent score chart showing a breakdown of potential predictable intents and the relative confidence level of correspondences between utterances and predicted intents. The inference report includes an interactive intent selection field. The interactive intent selection field may be presented adjacent to a suggestion button that, when interacted with, autofills the field with the suggested corresponding intent. The autofill field may be used to generate pairs of utterances and ground truth training categories to be included in a training dataset for retraining the model. For example, the client may select the checkbox next to the utterance, “Add another song to the Cita Romantica playlist.” The client can then press a button, such as the “Add Example” button on interface 1400, to trigger the generation of training data points containing the utterances and corresponding ground truth intents selected by the client. These training data points may then be included in a training dataset to train an ML model for a chatbot system in the future.

[0226] Interface 1400 also includes buttons for analyzing a queue of inference results. For example, instead of pressing the “Add Example” button to generate training data points, the client can press the “Add to Analyze Queue” button to request an explanatory report related to the inference result, given the corresponding utterance, as shown in process 500 in Figure 5.

[0227] Figure 15 shows an exemplary interface for describing inference results according to one embodiment. Specifically, Figure 15 shows an interface similar to interface 1400 after the inference results and corresponding utterances have been selected for introduction into the analysis queue. Notification 1500 may be included on the interface to inform the user that the selected inference results have been added to the queue of requested inference results. For example, by selecting the inference results using the checkboxes in interface 1400 and clicking the "Analyze Queue" button, a process such as process 500 may be started for the selected inference results and utterances and the corresponding utterances and ML models associated with them. Notification 1500 may then alert the user that the particular inference results are being analyzed and an explanatory report is being generated.

[0228] Figure 16 shows an exemplary interface for explaining inference results according to one embodiment. Specifically, Figure 16 shows an interface similar to interface 1400 / 1500 after an utterance has been analyzed by an utterance analysis tool, such as an utterance analysis tool utilizing process 500. Notification 1600 may be included on the interface to inform the user that the selected inference result / utterance has been analyzed and that the report is ready for viewing.

[0229] Figure 17 shows an exemplary interface for explaining inference results according to one embodiment. Specifically, Figure 17 shows an interface similar to interfaces 1400 / 1500 / 1600 while the explanation report is pending and has not yet finished generating. For example, while the client or user of the interface is waiting for the explanation report to be generated, the user may look at the displayed interface shown in Figure 17 to inform the user that the explanation analysis is not yet complete.

[0230] An example of submitting a request for an explanation of the utterance classification result, according to one embodiment, is shown below in a JavaScript® object notation (JSON) script.

[0231] Table 3: Explanation section for examples of generating synthesized utterances (in JSON).

[0232] [Table 3-1]

[0233] [Table 3-2]

[0234] [Table 3-3]

[0235] [Table 3-4]

[0236] In the example shown, a request may be submitted by entering a query (e.g., the utterance "money available") followed by the command "getexplain" (get explanation). A JavaScript Object Notation (JSON) file may be generated for the query and explanation request, as shown.

[0237] Figure 18 shows an example of the output of an explanation for utterance classification according to one embodiment. In the illustrated example, the utterance analyzed is “another agent question,” as shown in the table above. The output may include a link to the report 1810 and, for example, as part of process 500-530, a list of selected anchors 1820 and corresponding confidence levels 1830 for each set of anchors in the list of anchors 1820. For example, the confidence level associated with the anchor set “question” may be approximately 33.8%, corresponding to a confidence metric of 0.33 indicating that the correct intent was predicted for the input utterance. The confidence level associated with the anchor “question another,” using the anchor words “question” and “another,” is approximately 100%. This represents a response predicted with full accuracy given the anchor words (e.g., the ML model has been previously heavily trained on those anchor words).

[0238] Figure 19 is a simplified diagram of a distributed system 1900 for realizing one embodiment. In the illustrated example, the distributed system 900 includes one or more client computing devices 1902, 1904, 1906, and 1908 connected to a server 1912 via one or more communication networks 1910. The client computing devices 1902, 1904, 1906, and 1908 may be configured to run one or more applications.

[0239] In various examples, Server 1912 may be adapted to run one or more services or software applications that enable one or more embodiments of the embodiments described herein. In some examples, Server 1912 may also provide other services or software applications that may include non-virtual and virtual environments. In some examples, these services may be provided to users of client computing devices 1902, 1904, 1906, and / or 1908 as web-based services or cloud services, such as under the Software as a Service (SaaS) model. Users operating client computing devices 1902, 1904, 1906, and / or 1908 may access the services provided by these components by interacting with Server 1912 using one or more client applications.

[0240] In the configuration shown in Figure 19, server 1912 may include one or more components 1918, 1920, and 1922 that implement the functions performed by server 1912. These components may include one or more processors, hardware components, or software components that can be executed by a combination thereof. It should be recognized that a wide variety of system configurations are possible, which may differ from the distributed system 1900. Therefore, the example shown in Figure 19 is an example of a distributed system for implementing the example system and is not intended to be limiting.

[0241] A user may run one or more applications using client computing devices 1902, 1904, 1906, and / or 1908, which may generate one or more storage requests, which may then be processed in accordance with the teachings of this disclosure. A client device may provide an interface that allows a user of the client device to interact with the client device. A client device may also output information to the user through this interface. Although Figure 19 shows only four client computing devices, any number of client computing devices may be supported.

[0242] Client devices may include various types of computing systems, such as portable handheld devices, general-purpose computers like personal computers and laptops, workstation computers, wearable devices, game systems, thin clients, various messaging devices, sensors or other sensing devices. These computing devices may include various types and versions of software applications and operating systems (e.g., Microsoft Windows®, Apple Macintosh®, UNIX® or UNIX-like operating systems, Linux® or Linux-like operating systems, and various mobile operating systems, such as Google Chrome® OS, including Microsoft Windows Mobile®, iOS®, Windows Phone®, Android®, BlackBerry®, and Palm OS®). Portable handheld devices may include cellular phones, smartphones (e.g., iPhone®), tablets (e.g., iPad®), and personal digital assistants (PDAs). Wearable devices may include Google Glass® head-mounted displays and other devices. The game system may include various handheld game devices and internet-connected game devices (for example, Microsoft Xbox® game consoles with or without Kinect® gesture input devices, Sony PlayStation® systems, and various game systems provided by Nintendo®). Client devices may be capable of running a wide variety of applications, such as various internet-related applications and communication applications (for example, email applications and short message service (SMS) applications), and may use various communication protocols.

[0243] The communication network 1910 may be any type of network known to those skilled in the art that can support data communication using any of the various available protocols, including but not limited to TCP / IP (Transmission Control Protocol / Internet Protocol), SNA (System Network Architecture), IPX (Internet Packet Switching), AppleTalk®, etc. Just as an example, the communication network 1910 may include local area networks (LANs), Ethernet®-based networks, Token Ring, wide area networks (WANs), the Internet, virtual networks, virtual private networks (VPNs), intranets, extranets, public switched telephone networks (PSTNs), infrared networks, wireless networks (for example, wireless networks operating under any of the IEEE 802.11 protocol suites, Bluetooth®, and / or any other wireless protocols), and / or any combination of these and / or other networks.

[0244] Server 1912 may consist of one or more general-purpose computers, dedicated server computers (including, for example, PC (personal computer) servers, UNIX® servers, midrange servers, mainframe computers, rack-mount servers, etc.), server farms, server clusters, or other appropriate configurations and / or combinations. Server 1912 may include one or more virtual machines running a virtual operating system, or other computing architectures with virtualization. This could be, for example, one or more flexible pools of logical storage that can be virtualized to maintain virtual storage for the server. In various examples, Server 1912 may be adapted to run one or more services or software applications that provide the functions described above.

[0245] The computing system within Server 1912 may run one or more operating systems, including any of the above-mentioned operating systems, and commercially available server operating systems. Server 1912 may also run any of a variety of other server applications and / or middle-tier applications, including HTTP (Hypertext Transfer Protocol) servers, FTP (File Transfer Protocol) servers, CGI (Common Gateway Interface) servers, Java® servers, and database servers. Exemplary database servers include, but are not limited to, those commercially available from Oracle®, Microsoft®, Sybase®, IBM® (International Business Machines), and others.

[0246] In some implementations, server 1912 may include one or more applications for analyzing and organizing data feeds and / or event updates received from users of client computing devices 1902, 1904, 1906, and 1908. For example, data feeds and / or event updates may include, but are not limited to, Twitter® feeds, Facebook® updates, or real-time updates received from one or more third-party sources and continuous data streams, including real-time events related to sensor data applications, financial stock market boards, network performance measurement tools (e.g., network monitoring and traffic management applications), clickstream analysis tools, and automotive traffic monitoring. Server 1912 may also include one or more applications for displaying data feeds and / or real-time events via one or more display devices on client computing devices 1902, 1904, 1906, and 1908.

[0247] The distributed system 1900 may also include one or more data repositories 1914, 1916. In a particular example, these data repositories may be used to store data and other information. For example, one or more of the data repositories 1914, 1916 may be used to store information such as information related to storage virtual machines, information mapping application identities to selected storage virtual machines, and other information used by server 1912 when performing authentication functions. The data repositories 1914, 1916 may reside in various locations. For example, the data repository used by server 1912 may be located locally with server 1912, or it may be located remotely from server 1912 and communicate with server 1912 via a network-based connection or a dedicated connection. The data repositories 1914, 1916 may be of different types. In a particular example, the data repository used by server 1912 may be a database, such as a relational database, like a database provided by Oracle Corporation® and other manufacturers. One or more of these databases may be adapted to allow data to be stored, updated, and retrieved from the database in response to commands in SQL format.

[0248] In a particular example, one or more of the data repositories 1914 and 1916 may be used by the application to store application data. The data repositories used by the application may be of various types, such as a key-value store repository, an object store repository, or a general-purpose storage repository supported by the file system.

[0249] In certain examples, the functions described in this disclosure may be provided as services through a cloud environment. Figure 20 is a simplified block diagram of a cloud-based system environment system 2000 for realizing several embodiments. The cloud-based system environment system 2000 is a simplified block diagram of a cloud-based system environment that may provide various services as cloud services according to certain examples. In the examples shown in Figure 20, the cloud infrastructure system 2002 may provide one or more cloud services that a user may request using one or more client computing devices 2004, 2006, and 2008. The cloud infrastructure system 2002 may include one or more computers and / or servers, which may include those previously described with respect to server 1912. The computers within the cloud infrastructure system 2002 may be organized as general-purpose computers, dedicated server computers, server farms, server clusters, or any other appropriate arrangement and / or combination.

[0250] Network 2010 can facilitate data communication and exchange between client computing devices 2004, 2006, and 2008 and the cloud infrastructure system 2002. Network 2010 may include one or more networks. The networks may be of the same type or different types. Network 2010 may support one or more communication protocols, including wired and / or wireless protocols, to facilitate communication.

[0251] The example shown in Figure 20 is merely one example of a cloud infrastructure system and is not intended to be limiting. It should be understood that in several other examples, the cloud infrastructure system 2002 may have more or fewer components than those shown in Figure 20, may combine two or more components, or may have components in different configurations or arrangements. For example, while Figure 20 shows three client computing devices, any number of client computing devices may be supported in alternative examples.

[0252] The term "cloud service" is generally used to refer to services made available to users on demand via communication networks such as the Internet, through a service provider's system (e.g., Cloud Infrastructure System 2002). Typically, in a public cloud environment, the servers and systems that make up the cloud service provider's system are different from the customer's own on-premises servers and systems. The cloud service provider's system is managed by the cloud service provider. Therefore, customers can use the cloud services provided by the cloud service provider without purchasing separate licenses, support, or hardware and software resources for the service. For example, the cloud service provider's system can host applications, and users can order and use applications on demand via the Internet without purchasing infrastructure resources to run the applications. Cloud services are designed to provide easy and scalable access to applications, resources, and services. Several providers offer cloud services. For example, some cloud services, such as middleware services, database services, and Java® cloud services, are offered by Oracle Corporation® in Redwood Shores, California.

[0253] In certain examples, the cloud infrastructure system 2002 can provide one or more cloud services using various models such as a software as a service (SaaS) model, a platform as a service (PaaS) model, an infrastructure as a service (IaaS) model, including a hybrid service model. The cloud infrastructure system 2002 can include a suite of applications, middleware, databases, and other resources that enable the provisioning of various cloud services.

[0254] The SaaS model enables an application or software to be delivered to a customer as a service over a communication network such as the Internet without the customer purchasing the underlying hardware or software for the application. For example, by using the SaaS model, a customer can be enabled to access on-demand applications hosted by the cloud infrastructure system 2002. Examples of SaaS services provided by Oracle Corporation (registered trademark) include, but are not limited to, various services for human resources / capital management, customer relationship management (CRM), enterprise resource planning (ERP), supply chain management (SCM), enterprise performance management (EPM), analytics services, social applications, and the like.

[0255] The IaaS model is generally used to provide flexible computing and storage capabilities by providing infrastructure resources (such as servers, storage, hardware, and networking resources) to customers as cloud services. Various IaaS services are provided by Oracle Corporation (registered trademark).

[0256] The PaaS model is generally used to provide a platform and environmental resources as a service, enabling customers to develop, run, and manage applications and services without having to procure, build, or manage those resources themselves. Examples of PaaS services offered by Oracle Corporation® include, but are not limited to, Oracle Java Cloud Service (JCS), Oracle Database Cloud Service (DBCS), data management cloud services, and various application development solution services.

[0257] Cloud services are generally delivered in an on-demand, self-service, subscription-based, flexibly scalable, reliable, highly available, and secure manner. For example, a customer may order one or more services provided by the cloud infrastructure system 2002 through a subscription order. The cloud infrastructure system 2002 then provides the services requested in the customer's subscription order by performing processing. For example, a user may request the cloud infrastructure system to register an application as described above and to provide services to the application according to the application's specified requirements. The cloud infrastructure system 2002 may be configured to provide one cloud service or more cloud services.

[0258] The Cloud Infrastructure System 2002 can provide cloud services through various deployment models. In a public cloud model, the Cloud Infrastructure System 2002 may be owned by a third-party cloud service provider, and the cloud services are provided to the general public. These customers may be individuals or companies. In another example, under a private cloud model, the Cloud Infrastructure System 2002 may function within an organization (for example, within a corporate organization), and the services are provided to customers within this organization. For example, these customers may be various departments within the company, such as the human resources department or the payroll department, or individuals within the company. In yet another example, under a community cloud model, the Cloud Infrastructure System 2002 and the services provided may be shared among various organizations within the relevant community. Other models, such as hybrid models of the above models, may also be used.

[0259] The client computing devices 2004, 2006, and 2008 may be of different types and may be capable of operating one or more client applications. Users may interact with the cloud infrastructure system 2002, including requesting services provided by the cloud infrastructure system 2002, by using the client devices. For example, a user may use a client device to request authentication-related services as described in this disclosure.

[0260] In some examples, the processing performed by the Cloud Infrastructure System 2002 to provide services may include big data analysis. This analysis may involve using, analyzing, and processing large datasets to detect and visualize various trends, behaviors, relationships, etc., within this data. This analysis may be performed by one or more processors, possibly processing the data in parallel and running simulations using the data. For example, big data analysis may be performed by the Cloud Infrastructure System 2002 to determine which storage virtual machines should be selected for a particular application based on the application's stated authentication-related requirements. The data used in this analysis may include structured data (e.g., data stored in a database or data structured according to a structured model) and / or unstructured data (e.g., data blobs (binary large objects)).

[0261] As illustrated in the example in Figure 20, the cloud infrastructure system 2002 may include infrastructure resources 2030 used to facilitate the provisioning of various cloud services provided by the cloud infrastructure system 2002. Infrastructure resources 2030 may include, for example, processing resources, storage or memory resources, networking resources, etc. In a particular example, a storage virtual machine available to handle storage requested by an application may be part of the cloud infrastructure system 2002. In other examples, the storage virtual machine may be part of a different system.

[0262] In certain examples, resources may be grouped into resource sets or resource modules (also referred to as "pods") to facilitate the efficient provisioning of these resources to support various cloud services provided by Cloud Infrastructure System 2002 to different customers. Each resource module or pod may contain a pre-integrated and optimized combination of one or more types of resources. In certain examples, different pods may be pre-provisioned for different types of cloud services. For example, a first set of pods may be provisioned for a database service, and a second set of pods, which may contain different resource combinations than the pods in the first set, may be provisioned for a Java service, etc. For several services, resources allocated to provisioning these services may be shared among the services.

[0263] The Cloud Infrastructure System 2002 itself may internally use services 2032 that are shared by different components of the Cloud Infrastructure System 2002 and facilitate the provisioning of services by the Cloud Infrastructure System 2002. These internally shared services may include, but are not limited to, security and identity services, integration services, enterprise repository services, enterprise manager services, virus scanning and whitelisting services, high availability, backup and recovery services, services that enable cloud support, email services, notification services, and file transfer services.

[0264] The cloud infrastructure system 2002 may include multiple subsystems. These subsystems may be implemented in software, hardware, or a combination thereof. As shown in Figure 20, the subsystems may include a user interface subsystem 2012 that enables users or customers of the cloud infrastructure system 2002 to interact with the cloud infrastructure system 2002. The user interface subsystem 2012 may include various different interfaces, such as a web interface 2014, an online store interface 2016 where cloud services offered by the cloud infrastructure system 2002 are advertised and available for purchase by consumers, and other interfaces 2018. For example, a customer may use a client device to request one or more services (service request 2034) that the cloud infrastructure system 2002 offers using one or more of interfaces 2014, 2016, and 2018. For example, a customer may access the online store, browse the cloud services offered by the cloud infrastructure system 2002, and place a subscription order for one or more services offered by the cloud infrastructure system 2002 that the customer wishes to subscribe to. This service request may include information identifying the customer and one or more services that the customer wishes to request. For example, a customer may submit a request order for services provided by Cloud Infrastructure System 2002. As part of the order, the customer may provide information identifying the application for which the service is provided and one or more credentials for that application.

[0265] In a specific example, such as the example shown in Figure 20, the cloud infrastructure system 2002 may include an order management subsystem (OMS) 2020 configured to process new orders. As part of this process, the OMS 2020 may be configured to prepare the order for provisioning by creating a customer account if one does not already exist, receiving billing and / or account information from the customer to be used to charge the customer in order to provide the requested services to the customer, verifying the customer information, reserving the order for the customer after verification, and coordinating various workflows.

[0266] If properly validated, OMS2020 may invoke Order Provisioning Subsystem (OPS)2024, configured to provision resources for this order, including processing, memory, and networking resources. Provisioning may involve allocating resources for the order and configuring those resources to facilitate the services requested by the customer order. The way resources are provisioned for an order and the types of resources provisioned may depend on the type of cloud service ordered by the customer. For example, following a certain workflow, OPS2024 may be configured to determine the specific cloud service being requested and identify the number of pods that would have been pre-configured for that particular cloud service. The number of pods allocated for an order may depend on the size / volume / level / scope of the requested service. For example, the number of pods to allocate may be determined based on the number of users the service should support, the duration for which the service is requested, etc. The allocated pods may then be customized to suit the specific customer making the request in order to provide the requested service.

[0267] In certain cases, the setup phase process may be performed by the Cloud Infrastructure System 2002 as part of the provisioning process, as described above. The Cloud Infrastructure System 2002 can generate an application ID and select a storage virtual machine for the application from among the storage virtual machines provided by the Cloud Infrastructure System 2002 itself, or from storage virtual machines provided by other systems other than the Cloud Infrastructure System 2002.

[0268] The cloud infrastructure system 2002 may send a response or notification 2044 to the requesting customer to indicate when the requested service will be available. In some examples, the customer may be sent information (e.g., a link) that enables them to begin using and utilizing the benefits of the requested service. In certain examples, the response to the customer requesting the service may include an application ID generated by the cloud infrastructure system 2002 and information identifying the virtual machine selected by the cloud infrastructure system 2002 for the application corresponding to the application ID.

[0269] The Cloud Infrastructure System 2002 may provide services to multiple customers. For each customer, the Cloud Infrastructure System 2002 manages information related to one or more subscription orders received from the customer, maintains customer data related to the orders, and provides the requested services to the customer. The Cloud Infrastructure System 2002 may also collect usage statistics about the customer's use of the subscribed services. For example, statistics may be collected on the amount of storage used, the amount of data transferred, the number of users, and the amount of system uptime and system downtime. This usage information may be used to charge customers. Billing may be done, for example, on a monthly basis.

[0270] The cloud infrastructure system 2002 may provide services to multiple customers in parallel. The cloud infrastructure system 2002 may store information about these customers, which may include copyright information. In a particular example, the cloud infrastructure system 2002 includes an Identity Management Subsystem (IMS) 2028 configured to manage customer information and separate the managed information so that information about one customer cannot be accessed from information about another customer. The IMS 2028 may be configured to provide various security-related services, such as identity services, which may include services for managing information access management, authentication and authorization services, customer identity and roles and related capabilities, etc.

[0271] Figure 21 shows an example of a computer system 2100 for realizing several embodiments. In some examples, the computer system 2100 may be used to realize an application system, an access management system, a system in a data center, and any of the various servers and computer systems described above. As shown in Figure 21, the computer system 2100 includes various subsystems, including a processing subsystem 2104 that communicates with several other subsystems via a bus subsystem 2102. These other subsystems may include a processing acceleration unit 2106, an I / O subsystem 2108, a storage subsystem 2118, and a communication subsystem 2124. The storage subsystem 2118 may include non-temporary computer-readable storage media, including a computer-readable storage medium 2122 and system memory 2110.

[0272] The bus subsystem 2102 provides a mechanism for various components and subsystems of the computer system 2100 to communicate with each other as intended. Although the bus subsystem 2102 is schematically shown as a single bus, alternative examples of the bus subsystem may utilize multiple buses. The bus subsystem 2102 may be one of several types of bus structures, including a memory bus or memory controller, peripheral bus, local bus, etc., using one of various bus architectures. For example, such architectures may include the Industry Standard Architecture (ISA) bus, Micro Channel Architecture (MCA) bus, Enhanced ISA (EISA) bus, Video Electronics Standards Association (VESA) local bus, and Peripheral Component Interconnect (PCI) bus, which can be implemented as a mezzanine bus manufactured according to the IEEE P1386.1 standard.

[0273] The processing subsystem 2104 controls the operation of the computer system 2100 and may include one or more processors, application-specific integrated circuits (ASICs), or field-programmable gate arrays (FPGAs). The processors may include single-core or multi-core processors. The processing resources of the computer system 2100 can be organized into one or more processing units 2132, 2134, etc. A processing unit may include one or more processors, one or more cores from the same or different processors, a combination of cores and processors, or other combinations of cores and processors. In some examples, the processing subsystem 2104 may include one or more dedicated coprocessors, such as graphics processors or digital signal processors (DSPs). In some examples, some or all of the processing units in the processing subsystem 2104 may use customized circuitry, such as application-specific integrated circuits (ASICs) or field-programmable gate arrays (FPGAs).

[0274] In some examples, processing units within the processing subsystem 2104 may execute instructions stored in system memory 2110 or computer-readable storage medium 2122. In various examples, processing units may execute various programs or code instructions and maintain multiple programs or processes running concurrently. At any given time, some or all of the program code to be executed may reside in system memory 2110 and / or computer-readable storage medium 2122, which may contain one or more storage devices. Through appropriate programming, the processing subsystem 2104 may provide the various functions described above. In an example where the computer system 2100 is running one or more virtual machines, one or more processing units may be assigned to each virtual machine.

[0275] In a particular example, a processing acceleration unit 2106 may be optionally provided to perform customized processing to accelerate the overall processing performed by the computer system 2100, or to offload a portion of the processing performed by the processing subsystem 2104.

[0276] The I / O subsystem 2108 can include devices and mechanisms for inputting information into the computer system 2100 and / or outputting information from, or via, the computer system 2100. Generally, the use of the term "input device" is intended to include all conceivable types of devices and mechanisms for inputting information into the computer system 2100. User interface input devices can include, for example, a keyboard, a mouse or other pointing device such as a trackball, a touchpad or touch screen incorporated into a display, a scroll wheel, a click wheel, a dial, buttons, switches, a keypad, a voice input device with a voice command recognition system, a microphone, and other types of input devices. User interface input devices can also include motion sensing and / or gesture recognition devices such as the Microsoft Kinect (registered trademark) motion sensor, which enables a user to control and interact with the input device, a device that provides an interface for receiving input using gestures and voice commands such as the Microsoft Xbox (registered trademark) 360 game controller. User interface input devices can also include gesture recognition devices such as a Google Glass (registered trademark) blink detector that detects a user's eye movement (e.g., a "blink" while taking a photo and / or while making a menu selection) and converts the eye gesture into an input to the input device (e.g., Google Glass (registered trademark)). Additionally, user interface input devices can include a voice recognition sensing device that enables a user to interact with a voice recognition system (e.g., a Siri (registered trademark) navigator) via voice commands.

[0277] Other examples of user interface input devices may include, but are not limited to, three-dimensional (3D) mice, joysticks or pointing sticks, gamepads and graphic tablets, as well as auditory / visual devices such as speakers, digital cameras, digital camcorders, portable media players, webcams, image scanners, fingerprint scanners, barcode readers, 3D scanners, 3D printers, laser rangefinders, and eye-tracking devices. User interface input devices may also include medical imaging input devices such as computed tomography, magnetic resonance imaging, positional emission tomography, and medical ultrasound devices. User interface input devices may also include audio input devices such as MIDI keyboards and digital musical instruments.

[0278] Generally, the use of the term "output device" is intended to include all conceivable types of devices and mechanisms for outputting information from the computer system 2100 to a user or another computer. User interface output devices may include non-visual displays such as display subsystems, indicator lights, or audio output devices. Display subsystems may include flat panel devices such as those using cathode ray tubes (CRTs), liquid crystal displays (LCDs), or plasma displays, planning devices, touchscreens, etc. For example, user interface output devices may include, but are not limited to, a variety of display devices that visually convey text, graphics, and audio / video information, such as monitors, printers, speakers, headphones, car navigation systems, plotters, audio output devices, and modems.

[0279] The storage subsystem 2118 provides a repository or datastore for storing information and data used by the computer system 2100. The storage subsystem 2118 provides a tangible, non-temporary, computer-readable storage medium for storing basic programming and data configurations that provide some example functionality. Software (e.g., programs, code modules, instructions) that, when executed by the processing subsystem 2104, provides the above-described functionality may be stored in the storage subsystem 2118. The software may be executed by one or more processing units of the processing subsystem 2104. The storage subsystem 2118 may also provide authentication in accordance with the teachings of this disclosure.

[0280] The storage subsystem 2118 may include one or more non-temporary memory devices, including volatile and non-volatile memory devices. As shown in Figure 21, the storage subsystem 2118 includes system memory 2110 and computer-readable storage medium 2122. The system memory 2110 may include several memories, including volatile primary random access memory (RAM) for storing instructions and data during program execution, and non-volatile read-only memory (ROM) or flash memory for storing fixed instructions. In some implementations, a basic input / output system (BIOS), which includes basic routines to assist in the transfer of information between elements within the computer system 2100 during startup, etc., may typically be stored in ROM. Typically, the RAM contains data and / or program modules currently being operated and executed by the processing subsystem 2104. In some implementations, the system memory 2110 may include several different types of memory, such as static random access memory (SRAM) and dynamic random access memory (DRAM).

[0281] As an example, without limitation, as shown in Figure 21, the system memory 2110 may load running application programs 2112, program data 2114, and operating systems 2116, which may include various applications such as web browsers, middle-tier applications, and relational database management systems (RDBMS). As an example, the operating system 2116 may include Microsoft Windows®, Apple Macintosh®, and / or Linux operating systems, various commercially available UNIX® or UNIX-like operating systems (including, but not limited to, various GNU / Linux operating systems, Google Chrome® OS, etc.), and / or various versions of mobile operating systems such as iOS®, Windows Phone, Android® OS, BlackBerry® OS, and Palm® OS.

[0282] The computer-readable storage medium 2122 can store programming and data structures that provide several example functions. The computer-readable storage medium 2122 can provide storage for computer-readable instructions, data structures, program modules, and other data for the computer system 2100. Software (programs, code modules, instructions) that provides the above functions when executed by the processing subsystem 2104 may be stored in the storage subsystem 2118. As an example, the computer-readable storage medium 2122 may include non-volatile memory such as a hard disk drive, magnetic disk drive, optical disk drive such as a CD-ROM, DVD, or Blu-ray® disc, or other optical media. The computer-readable storage medium 2122 may also include, but is not limited to, a Zip® drive, flash memory card, Universal Serial Bus (USB) flash drive, Secure Digital (SD) card, DVD disc, digital videotape, etc. Computer-readable storage media 2122 may also include solid-state drives (SSDs) based on non-volatile memory such as flash memory-based SSDs, enterprise flash drives, and solid-state ROMs; SSDs based on volatile memory such as solid-state RAM, dynamic RAM, and static RAM; DRAM-based SSDs; magnetoresistive RAM (MRAM) SSDs; and hybrid SSDs that use a combination of DRAM and flash memory-based SSDs.

[0283] In certain examples, the storage subsystem 2118 may also include a computer-readable storage medium reader 2120 that can be further connected to the computer-readable storage medium 2122. The reader 2120 may be configured to receive and read data from memory devices such as disks, flash drives, etc.

[0284] In certain cases, computer system 2100 may support virtualization technologies, including but not limited to virtualization of processing and memory resources. For example, computer system 2100 may provide support for running one or more virtual machines. In certain cases, computer system 2100 may run programs such as hypervisors that facilitate the configuration and management of virtual machines. Each virtual machine may be allocated memory, computing resources (e.g., processors, cores), I / O, and networking resources. Each virtual machine typically runs independently of other virtual machines. A virtual machine typically runs its own operating system, which may be the same as or different from the operating system run by other virtual machines run by computer system 2100. Thus, potentially multiple operating systems may run simultaneously by computer system 2100.

[0285] The communication subsystem 2124 provides interfaces to other computer systems and networks. It functions as an interface for sending and receiving data between other systems and the computer system 2100. For example, the communication subsystem 2124 may enable the computer system 2100 to establish communication channels to one or more client devices over the internet in order to send and / or receive information with one or more client devices. For example, if the computer system 2100 is used to implement the bot system 120 shown in Figure 1, the communication subsystem may also be used to communicate with the application system and the system running the storage virtual machine selected for the application.

[0286] The communication subsystem 2124 may support both wired and / or wireless communication protocols. In some examples, the communication subsystem 2124 may include radio frequency (RF) transceiver components for accessing wireless voice and / or data networks (e.g., using cellular telephone technology, 3G, 4G, EDGE (High Speed ​​Data Rate for Global Evolution), or 5G), a Global Positioning System (GPS) receiver component, and / or other components. In some examples, the communication subsystem 2124 may provide wired network connectivity (e.g., Ethernet®) in addition to or instead of a wireless interface.

[0287] The communication subsystem 2124 can receive and transmit data in various formats. In some examples, in addition to other formats, the communication subsystem 2124 may receive input communications in the form of structured data feeds and / or unstructured data feeds 2126, event streams 2128, event updates 2130, etc. For example, the communication subsystem 2124 may be configured to receive (or transmit) data feeds 2126 in real time from users of other communication services such as social media networks and / or web feeds such as Twitter® feeds, Facebook® updates, Rich Site Summary (RSS) feeds, and / or real-time updates from one or more third-party sources.

[0288] In certain examples, the communication subsystem 2124 may be configured to receive data in the form of a continuous data stream, which may include an event stream 2128 and / or event update 2130 of real-time events that are inherently continuous or infinite and do not have a clear end. Examples of applications that generate continuous data include, for example, sensor data applications, financial stock market boards, network performance measurement tools (e.g., network monitoring and traffic management applications), clickstream analysis tools, and automotive traffic monitoring.

[0289] The communication subsystem 2124 may be configured to transmit data from the computer system 2100 to other computer systems or networks. This data may be transmitted in various different formats, such as structured and / or unstructured data feeds 2126, event streams 2128, and event updates 2130, to one or more databases that can communicate with one or more streaming data source computers connected to the computer system 2100.

[0290] The computer system 2100 may be one of many types, including handheld portable devices (e.g., iPhone® cellular phone, iPad® computing tablet, PDA), wearable devices (e.g., Google Glass® head-mounted display), personal computers, workstations, mainframes, kiosks, server racks, or other data processing systems. Because the nature of computers and networks is constantly changing, the description of the computer system 2100 shown in Figure 21 is intended only as a specific example. Many other configurations are possible, having more or fewer components than the system shown in Figure 21. Based on the disclosures and teachings provided herein, those skilled in the art will understand other methods and / or ways to realize various examples.

[0291] While specific examples have been described, various variations, modifications, alternative configurations, and equivalents are possible. The examples are not limited to operation within a specific data processing environment, but can freely operate within multiple data processing environments. Furthermore, although the examples were illustrated using a specific set of transactions and steps, it should be apparent to those skilled in the art that this is not intended as limitation. While some flowcharts illustrate operations as sequential processes, many of these operations may be executed in parallel or concurrently. In addition, the order of operations may be re-specified. Processes may have additional steps not shown in the diagrams. The various features and aspects of the above examples may be used individually or together.

[0292] Furthermore, while specific examples have been described using particular combinations of hardware and software, it should be understood that other combinations of hardware and software are also possible. The specific examples may be implemented using hardware alone, software alone, or a combination thereof. The various processes described herein may be implemented on the same processor or on different processors in any combination.

[0293] Where it is stated that a device, system, component, or module is configured to perform a particular operation or function, such configuration can be achieved, for example, by designing electronic circuits to perform the operation; by programming programmable electronic circuits (such as a microprocessor) to perform the operation; by executing computer instructions or code, or any combination thereof, stored in a non-temporary memory medium, or by executing a processor or core. Processes can communicate using a variety of techniques, including but not limited to conventional techniques for inter-process communication, different pairs of processes may use different techniques, and the same pair of processes may use different techniques at different times.

[0294] This disclosure provides specific details to ensure that the examples are fully understood. However, the examples can be implemented without these specific details. For example, well-known circuits, processes, algorithms, structures, and techniques are shown without unnecessary details in order to avoid ambiguity in the examples. This specification provides only exemplary examples and is not intended to limit the scope, applicability, or configuration of other examples. Rather, the above description of the examples provides a description that enables the implementation of various examples for those skilled in the art. Various modifications are possible within the scope of the function and configuration of the elements.

[0295] Therefore, the specification and drawings should be considered illustrative rather than restrictive. However, it will be clear that additions, reductions, deletions, and other modifications and changes may be made to them without departing from the broader spirit and scope set forth in the claims. Thus, specific examples have been described, but these are not intended to be limiting. Various variations and equivalents are within the scope of the attached claims.

[0296] While the above specification illustrates aspects of the disclosure with reference to specific examples, those skilled in the art will recognize that the disclosure is not limited thereto. The various features and aspects of the above disclosure may be used individually or together. Furthermore, the examples can be used in a variety of environments and applications beyond those described herein, without departing from the broader spirit and scope of the specification. Therefore, the specification and drawings should be considered illustrative rather than restrictive.

[0297] In the above description, the method is presented in a specific order for illustrative purposes. It should be understood that in alternative examples, the method may be executed in a different order than described. It should also be understood that the above method may be executed by hardware components, or it may be embodied in a sequence of machine-executable instructions, which, when used, may cause a machine such as a general-purpose or dedicated processor or logic circuit programmed with such instructions to execute the method. These machine-executable instructions may be stored on one or more machine-readable media, or other types of machine-readable media suitable for storing electronic instructions, such as CD-ROMs or other types of optical discs, floppy disks, ROMs, RAMs, EPROMs, EEPROMs, magnetic or optical cards, or flash memory. Alternatively, these methods may be executed by a combination of hardware and software.

[0298] Where a component is described as being configured to perform a particular operation, such configuration may be achieved, for example, by designing an electronic circuit or other hardware to perform a particular operation, by programming a programmable electronic circuit (e.g., a microprocessor or other suitable electronic circuit) to perform a particular operation, or by any combination thereof.

[0299] While examples illustrating this application are described in detail here, it should be understood that the concepts of the present invention can be embodied and adopted in various other aspects, and that the claims are intended to be interpreted to include such variations, unless limited by the prior art.

Claims

1. A method implemented by a computer, The chatbot system uses a machine learning model to classify speech and determine the results. The chatbot system includes determining a plurality of anchor sets based on the utterance, where each of the plurality of anchor sets corresponds to one or more anchor words of the utterance, and the machine learning model is configured to output a classification result based on the input utterance. The above method further applies to each of the anchor sets: The chatbot system includes generating one or more synthesized utterances corresponding to the anchor set based on the utterance, and each of the one or more synthesized utterances includes one or more anchor words of the anchor set. The generation of one or more synthesized utterances includes generating a plurality of synthesized utterances, each of which comprises one or more anchor words of the anchor set and one or more substituted words, and each of the substituted words of the synthesized utterance is a variation of a non-anchor word in the utterance. The above method further applies to each of the anchor sets: The chatbot system includes determining one or more classification results for one or more synthesized utterances and determining a confidence value, and determining the confidence value for the anchor set includes determining one or more matches between the classification result for the utterance and the one or more classification results for one or more synthesized utterances. The above method further, The chatbot system includes generating a report that includes a representation of a particular set of anchors among the plurality of anchor sets, wherein the particular set of anchors corresponds to the highest confidence value among the plurality of anchor sets, and is implemented by a computer.

2. The above method further, The chatbot system determines one or more specific synthesized utterances corresponding to the specific anchor set based on the specific anchor set, The chatbot system includes generating a training dataset for training the machine learning model based on one or more specific synthesized utterances, wherein the training dataset includes a subset of the one or more specific synthesized utterances, and the method further includes The chatbot system includes training the machine learning model using the training dataset, The computer-based method according to claim 1, wherein the machine learning model is further configured to output one or more classification results based on one or more synthesized utterances included in the training dataset.

3. The training dataset is generated by selecting the subset of one or more specific synthetic utterances based on a comparison between the confidence value and threshold confidence value of the subset of one or more specific synthetic utterances. The method implemented by a computer according to claim 2, wherein the confidence value of the subset is obtained by calculating the proportion of synthesized utterances that are determined to be a match among the one or more synthesized utterances corresponding to the subset.

4. Determining the confidence value for each of the aforementioned anchor sets is: A computer-based method according to any one of claims 1 to 3, comprising the chatbot system calculating the proportion of synthesized utterances that are determined to be a match among the one or more synthesized utterances corresponding to the anchor set.

5. A computer-based method according to any one of claims 1 to 4, wherein determining the plurality of anchor sets includes selecting the plurality of anchor sets in one or more rounds using beam search technology.

6. The computer-implemented method according to any one of claims 1 to 5, wherein the report includes a representation of a plurality of anchor sets, each of the plurality of anchor sets corresponding to a confidence value greater than a threshold confidence value.

7. Furthermore, the computer-based method according to any one of claims 1 to 6, comprising the chatbot system receiving a request to generate the report about the utterance, and determining the plurality of anchor sets being performed in response to receiving the request to generate the report.

8. It is a chatbot system, One or more processors, The system comprises a non-temporary computer-readable storage medium that stores computer-executable instructions, and when the computer-executable instructions are executed by one or more processors, the one or more processors cause the execution of an operation, and the operation is This involves using a machine learning model to classify speech and determine the results of that classification. The operation further includes determining a plurality of anchor sets based on the utterance, each of which anchor sets corresponds to one or more anchor words of the utterance, and the operation further includes For each of the aforementioned anchor sets, The machine learning model is configured to output a classification result based on the input utterance, wherein each of the one or more synthesized utterances includes one or more anchor words from the anchor set, and the machine learning model is configured to generate one or more synthesized utterances corresponding to the anchor set based on the input utterance. The generation of one or more synthesized utterances includes generating a plurality of synthesized utterances, each of which comprises one or more anchor words of the anchor set and one or more substituted words, and each of the substituted words of the synthesized utterance is a variation of a non-anchor word in the utterance. The aforementioned operation further, For each of the aforementioned anchor sets, This includes determining one or more classification results for the one or more synthesized utterances and determining a confidence value, and determining the confidence value for the anchor set includes determining one or more matches between the classification result for the utterance and the one or more classification results for the one or more synthesized utterances. The aforementioned operation further, A chatbot system comprising generating a report that includes a representation of a specific anchor set among the plurality of anchor sets, wherein the specific anchor set corresponds to the highest confidence value among the plurality of anchor sets.

9. The one or more processors perform further operations, and these operations are: Based on the aforementioned specific anchor set, one or more specific synthesized utterances corresponding to the aforementioned specific anchor set are determined, The operation further includes generating a training dataset for training the machine learning model based on one or more specific synthesized utterances, wherein the training dataset includes a subset of the one or more specific synthesized utterances, and the operation further includes This includes training the machine learning model using the aforementioned training dataset. The chatbot system according to claim 8, wherein the machine learning model is further configured to output one or more classification results based on utterances included in the training dataset.

10. The training dataset is generated by selecting the subset of one or more specific synthetic utterances based on a comparison between the confidence value and threshold confidence value of the subset of one or more specific synthetic utterances. The chatbot system according to claim 9, wherein the confidence value of the subset is obtained by calculating the proportion of synthesized utterances that are determined to be a match among the one or more synthesized utterances corresponding to the subset.

11. Determining the confidence value for each of the aforementioned anchor sets is: A chatbot system according to any one of claims 8 to 10, comprising calculating the proportion of synthesized utterances that are determined to be a match among the one or more synthesized utterances corresponding to the anchor set.

12. A method implemented by a computer, The process includes generating one or more synthesized utterances based on an utterance, wherein each of the one or more synthesized utterances includes one or more shared words with the utterance and corresponds to a certain confidence value among one or more confidence values. The generation of one or more synthesized utterances includes generating a plurality of synthesized utterances, each synthesized utterance comprising one or more shared words and one or more substituted words, each of the substituted words in the synthesized utterance being a variation of a non-shared word in the utterance. The confidence value corresponding to each of the synthesized utterances is obtained using a machine learning model configured to output a confidence value based on the input synthesized utterance. The machine learning model includes a neural network model used for syntactic analysis of utterances, and is configured to map the input synthesized utterances to multiple nodes of the neural network model. The confidence value of the synthesized utterance is obtained based on a numerically derived metric corresponding to the nodes mapped for that synthesized utterance. The above method further, For each of the aforementioned synthesized utterances, the confidence value of the synthesized utterance is obtained using the neural network model of the synthesized utterance. The above method further, The process involves comparing one or more confidence values ​​with a threshold confidence value, The method further includes determining a subset of the one or more synthesized utterances based on the comparison, wherein the determination includes selecting a synthesized utterance for the subset of the one or more synthesized utterances if the synthesized utterance corresponds to a confidence value less than or equal to the threshold confidence value, and the method further includes The method includes receiving from a user of an interactive interface a selection of one or more training categories and a selection of one or more synthesized utterances corresponding to each of the one or more training categories, wherein the selection of one or more synthesized utterances includes selection from a subset of synthesized utterances, and the method further includes: A computer-implemented method comprising generating a training dataset for training the machine learning model based on the selected subset of the one or more synthesized utterances and the selected one or more training categories.

13. The computer-implemented method according to claim 12, further comprising training the machine learning model based on the training dataset.

14. The machine learning model is constructed as an intent classifier including a plurality of model parameters to be learned by using an objective function, and training the machine learning model includes minimizing or maximizing the objective function, which measures the difference between a predicted intent and a correct intent, as described in claim 13, a computer-implemented method.

15. The method implemented by a computer according to any one of claims 12 to 14, wherein the threshold confidence value is received from the user of the interactive interface by inputting the threshold confidence value into an interactive interface element.

16. A program that causes a computer to perform the method according to any one of claims 1 to 7 and 12 to 15.

Images