Multifactorial Modeling for Natural Language Processing

Abstract

Techniques for a system are disclosed, including techniques for multi-factor modeling to train and utilize chatbot systems for natural language processing. In one embodiment, a method includes receiving a set of utterance data corresponding to a natural language based query and determining one or more intents for the chatbot, the one or more intents corresponding to possible contexts for the natural language based query and associated with a skill for the chatbot, the method further includes generating one or more intent classification datasets, each intent classification dataset being associated with a probability that the natural language query corresponds to an intent of the one or more intents, the method further includes generating one or more transformed datasets, each corresponding to a skill of the one or more skills, determining a first skill from the one or more skills based on the one or more transformed datasets, and processing the set of utterance data based on the determined first skill to resolve the natural language based query.

Description

[Technical field]

[0001] Priority claim This application claims the benefit of and priority to U.S. Provisional Patent Application 63 / 139,693, filed January 20, 2021, which claims the benefit of and priority to U.S. Nonprovisional Patent Application 17 / 578,170, filed January 18, 2022, which is incorporated herein by reference in its entirety for all purposes.

[0002] FIELD OF THEINVENTION The present disclosure relates generally to chatbot systems, and more particularly to techniques for multi-factor model training for chatbot systems in natural language processing. [Background technology]

[0003] background Instant messaging capabilities and automated chat platforms are efficient solutions to modern customer service problems. Organizations can leverage these implementations to provide timely and agile service to customers without investing valuable human capital in individual user inquiries. Modern automated chat platforms may utilize a parent "chatbot" to handle customer service requests. The parent chatbot may coordinate with multiple child chatbots that are delegated more specialized tasks. However, selecting the correct chatbot to which a task should be delegated is a difficult and potentially costly decision. Certain chatbots are designed to handle only a specific range of inquiries, and a particular chatbot may handle specialized tasks with much less efficiency than another chatbot. The inability to select the most efficient chatbot to answer customer inquiries may result in poor performance, inefficient resource utilization, and frustration for the customer base. Summary of the Invention [Problem to be solved by the invention]

[0004] For example, some chatbots may include several "skill" classes that help the chatbot process and respond to queries. These queries may take the form of "utterances" in a natural language-based format that simulates a text conversation or request. The utterances are processed to determine the most appropriate skills that the chatbot should employ to resolve the utterance and respond to the query. For example, the "order pizza" skill class is selected for the chatbot to respond to the natural language query "order me a pizza."

[0005] Slight variations in the context, intonation, spelling, tone, and / or setting of a customer inquiry can result in the selection of the wrong chatbot / skill for a particular task. Errors in selecting a chatbot can compound rapidly when an organization runs hundreds or thousands of automated inquiry responses every day. Simple methods of selecting a chatbot, such as one-to-one mapping of words to a particular chatbot, may not allow for proper contextual analysis and do not take into account the complexities of a conversation.

[0006] To aid in skill selection, a chatbot may use machine learning models to process an utterance and output the most appropriate skill for response to the utterance. The selection of skills to aid in answering a query can be based on contextual and lexical analysis of the query given to the organization. Thus, a chatbot is trained using single-factor selection machine learning techniques to select a skill given an input utterance. Single-factor models directly map inputs, e.g., a received utterance, to a desired output, e.g., a predicted skill for processing the utterance. Single-factor machine learning techniques are more time-efficient in predicting skills from input utterances, but such single-factor models do not take into account important context, intonation, orthography, tone, or intent of the utterance. For example, an utterance of "I ordered a pizza and it didn't arrive" is intended to be targeted to a customer service chatbot skill, but may instead be directed to a pizza ordering chatbot skill based on the partial phrase "ordered a pizza." As a result, training machine learning models using more context-agnostic methods such as direct skill selection does not provide the appropriate level of contextual learning required for an efficient chatbot model. Thus, standard machine learning techniques that directly determine the appropriate skill class to resolve an utterance are often insufficient.

[0007] overview Techniques for multi-factor modeling for training and utilizing chatbot systems for natural language processing are disclosed. [Means for solving the problem]

[0008] In one embodiment, a computer-implemented method includes a computing device receiving a set of utterance data corresponding to a natural language based query interpretable by a chatbot; and the computing device determining one or more intents for the chatbot, each of the one or more intents corresponding to a possible context for the natural language based query and associated with a skill of one or more skills for the chatbot; the method further includes the computing device generating one or more intent classification datasets based on the one or more intents using an intent classifier model, each intent classification dataset being associated with a probability that the natural language query corresponds to an intent of the one or more intents; the method further includes the computing device generating one or more transformed datasets based on the one or more intent classification datasets using a transformation mask model, each of the one or more transformed datasets corresponding to a skill of the one or more skills; the method further includes the computing device determining a first skill from the one or more skills based on the one or more transformed datasets; and processing the set of utterance data to resolve the natural language based query based on the determined first skill.

[0009] In one embodiment, the set of utterance data is received from a client device interacting with an automated digital assistant in communication with a computing device.

[0010] In one embodiment, the transformation mask model includes one or more transformation values, each transformation value corresponding to a ratio of at least one intent to at least one skill. In an additional embodiment, the set of utterance data includes at least an indication of a ground truth skill of the one or more skills, and the method further includes the computing device comparing the ground truth skill to the first skill to generate one or more training loss values, and the computing device modifying the one or more transformation values ​​based on the training loss values.

[0011] In one embodiment, the intent classifier model is a machine learning model configured to receive as input a set of utterance data and output one or more intent classification datasets, the set of utterance data including at least an indication of a ground truth skill of the one or more skills, and the method further includes the computing device comparing the ground truth skill to the first skill to generate one or more training loss values, and the computing device training the machine learning model with the one or more training loss values ​​by altering a structure of the machine learning model based on the training loss values.

[0012] In one embodiment, the method further includes the computing device generating second skill data using a skill classifier model, the skill classifier model being a machine learning model configured to receive as input a set of utterance data and output a predicted skill from the one or more skills, the method further includes the computing device comparing the first skill and the second skill to generate one or more training loss values, the one or more training loss values ​​corresponding to one or more differences between the first skill and the second skill, and the method further includes the computing device training the machine learning model using the one or more training loss values ​​by modifying a structure of the machine learning model based on the training loss values.

[0013] In one embodiment, the method includes processing the set of utterance data to generate a natural language-based response to the natural language-based query based on the determined first skill, and transmitting the natural language-based response to the client device.

[0014] In one embodiment, the method further includes the computing device processing the speech data to generate refined speech data in a format for input to the intent classifier model, where the refined speech data is input to the intent classifier model to generate one or more intent classification data sets.

[0015] In one embodiment, each transformed data set of the one or more transformed data sets includes a probability value corresponding to a likelihood that the speech data corresponds to a skill of the one or more skills, and determining the first skill includes selecting the skill corresponding to the transformed data set having a highest probability value from the set of one or more probability values.

[0016] In one embodiment, the first skill includes generating one or more average probability values ​​for each skill based on the probability values ​​associated with the intent classification dataset, and selecting a skill associated with the highest average probability value from among the one or more average probability values.

[0017] Some embodiments of the present disclosure include a computer program product tangibly embodied in a non-transitory machine-readable storage medium and including instructions configured to cause one or more data processors to perform some or all of one or more of the methods disclosed herein.

[0018] Some embodiments of the present disclosure include a system including one or more data processors. In some embodiments, the system includes a non-transitory computer-readable storage medium including instructions that, when executed on the one or more data processors, cause the one or more data processors to perform some or all of one or more methods and / or some or all of one or more processes disclosed herein.

[0019] The techniques described above and below may be implemented in a number of ways and in a number of contexts. Some example implementations and contexts are provided with reference to the following drawings, as described in more detail below. However, the following implementations and contexts are only a few of many. [Brief description of the drawings]

[0020] [Figure 1] 1 is a simplified block diagram of a distributed environment incorporating an illustrative embodiment; [Diagram 2] FIG. 1 is a simplified block diagram of a parent bot (MB) system according to some embodiments. [Diagram 3] FIG. 1 is a simplified block diagram of a Skillbot system according to an embodiment. [Figure 4] FIG. 1 is a simplified block diagram of a computing system implementing a speech data processing system, according to one embodiment. [Diagram 5] FIG. 1 is a simplified block diagram of a computing system that implements a transformation model, according to one embodiment. [Figure 6A] FIG. 1 is a simplified block diagram of a computing system that implements a transformation model, according to one embodiment. [Figure 6B] FIG. 1 is a simplified block diagram of a computing system that implements a transformation model, according to one embodiment. [Figure 7A] FIG. 2 is a simplified block diagram of an intent classifier configured as an artificial neural network, according to one embodiment. [Figure 7B]FIG. 1 is a simplified block diagram of a skill classifier configured as an artificial neural network, according to one embodiment. [Figure 8] FIG. 1 illustrates a process flow for selecting a context class using a transformation model to respond to an utterance, according to various embodiments. [Figure 9] FIG. 1 is a simplified block diagram of an exemplary selection of a skill bot by intent classification techniques, according to an embodiment. [Figure 10] 1 shows a simplified diagram of a distributed system for implementing various embodiments. [Figure 11] FIG. 1 is a simplified block diagram of one or more components of a system environment in which services provided by one or more components of an embodiment system are provided as cloud services, in accordance with various embodiments. [Figure 12] 1 illustrates an exemplary computer system that may be used to implement various embodiments. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0021] Detailed Description In the following description, for purposes of explanation, specific details are set forth in order to provide a thorough understanding of particular embodiments. It will be apparent, however, that various embodiments may be practiced without these specific details. The figures and description are not intended to be limiting. The term "exemplary" is used herein to mean "serving as an example, instance, or illustration." Any embodiment or design described herein as "exemplary" is not necessarily to be construed as preferred or advantageous over other embodiments or designs.

[0022] A particular skill of a chatbot is designed to process and respond only to a particular range of utterances, and a particular skill may only solve specialized tasks that another skill cannot solve. For example, several skills of a chatbot may be related to somewhat similar tasks (e.g., operating an automated system at a pizza place), but each task is tailored to process and respond to a particular type of utterance (e.g., initiating a new pizza order, modifying an existing order, providing customer support, providing information about business transactions and hours of operation, etc.). The inability to select the most efficient skill to answer a customer inquiry may result in poor performance, inefficient resource utilization, and frustration for the customer base.

[0023] As mentioned above, natural language specific elements such as the context, intonation, orthography, tone, and / or setting of a customer query may cause incorrect chatbot / skill selection for a particular task. To assist in selecting the most appropriate skill, the chatbot may use machine learning models to process the utterance and output the most appropriate skill for responding to the utterance. The selection of skills to assist in answering the query may be based on a contextual and lexical analysis of the query given to the digital assistant. For example, the utterance is sent to a "skill classifier" which uses machine learning techniques to analyze the utterance and determine the specific skill for resolving the utterance. Single-factor modeling techniques for skill selection directly map inputs such as natural language utterances to predicted skills for resolving the utterance. Single-factor modeling is a useful technique because it may enable a quick prediction of skills that are prioritized when responding to customers utilizing an automated digital assistant. However, single-factor machine learning techniques for skill selection based on processing of an utterance may not take into account important context, intonation, orthography, tone, or intent of the utterance. For example, the utterance "I ordered a pizza and it didn't arrive" might be intended for a customer service chatbot, but might instead be directed to a pizza ordering chatbot skill based on the phrase "ordered a pizza."

[0024] Training a chatbot model to select skills based on directly mapping utterances to possible skills may not train the chatbot model adequately to handle the context and intonation of the utterance effectively enough to select skills with sufficient accuracy. The inadequacy of single-factor skill-based selection is evident in both the training and prediction phases of a chatbot system. For example, training for a machine learning model of a chatbot system that selects and determines skills directly from the input utterance to respond to the utterance lacks sufficient contextual details to build an appropriate classification model. An undertrained chatbot utilizing single-factor modeling may then proceed to select inappropriate skills to resolve the utterance when interacting with a real client / customer.

[0025] Techniques for multi-factor modeling for natural language processing are described herein to train and deploy chatbots / skills to process and respond to natural language based queries. Multi-factor modeling techniques enable improved training and processing of input data, such as utterance data, to better predict skills for processing utterances and train skill classifier models to do so. Multi-factor models can also be trained to provide more accurate output data for training single-factor models, such as skill classifiers, or for processing contextually complex utterances to respond to customer / client inquiries.

[0026] The multi-factor modeling technique as described herein utilizes a structural class hierarchy of the chatbot system, including a set of skills for each chatbot and a set of intents for each skill. The skills and intents are described below with reference to Figures 1-3. A multi-factor machine learning model is used to process the natural language query to output an intermediate contextual data set representing the intent classification of the utterance. The intermediate intent classification may then be utilized to determine the skills for processing the utterance or to further improve the training of the skill classification of the chatbot system. Thus, the multi-factor modeling technique allows for the introduction of contextually determined intent data that is taken into account in the final selection of chatbot skills for processing the utterance. Prior to the selection of the final skills for processing the utterance, various transformations may be applied to the intermediate intent classifications. For example, transformation "masks" are applied to the intermediate intent classification data to map the intent classifications to the corresponding skill classifications. Thus, the skill classifications generated by the multi-factor modeling technique include a more contextually relevant processing of the original utterance compared to single-factor modeling techniques such as direct skill prediction from the utterance.

[0027] The techniques described herein further improve the training and optimization of single-factor models that can be deployed for customer interactions. As mentioned above, single-factor models alone are insufficient to handle low-grained contextual linguistics and intonations in natural language. Attempting to refine these single-factor models with basic supervised machine learning techniques may somewhat increase the accuracy of single-factor models in predicting skills, but does not allow for context-based training. The use of multi-factor models to select more contextually accurate skills from a set of utterance data, and comparing the contextually predicted skills with predicted skills from single-factor models, will yield context-based loss data that is used to retrain single-factor models in a manner that single supervised learning cannot. Thus, multi-factor modeling not only produces more accurate predictions when processing natural language queries, but also trains time-efficient single-factor models to better predict skills given a set of input utterance data.

[0028] As described herein, "utterance data" may include data related to a natural language-based query provided by a customer or client for resolution by an automated chatbot system. For example, an utterance may include the natural language-based query "I would like to order a pizza."

[0029] As described herein, a "context class" is a chatbot "skill" that relates to a response category and / or subroutine for resolving an utterance by the chatbot. A context skill class may include instructions, configurations, or other functionality for processing and responding to an utterance in a natural language based format. For example, a chatbot may include a skill such as a "pizza ordering skill." Each chatbot may include multiple "skills" for resolving and responding to utterances.

[0030] As described herein, a "context subclass" is a chatbot "intent" that relates to a response subcategory and / or subroutine for resolving an utterance by a chatbot through a particular skill. A context intent class may include instructions, configurations, or other functionality for processing and responding to an utterance in a natural language based format according to the skill to which the intent belongs. For example, a chatbot may include a skill such as "pizza ordering skill," which may further include an intent such as "place a new order from a customer." Each skill may include or correspond to multiple "intents" for resolving and responding to an utterance, and each intent may correspond to a different subtask / context.

[0031] As described herein, a "translation model" is a set of translation data that represents a correspondence between one or more intents and one or more skills of a chatbot model. For example, a translation model is a dynamically trained set of mapping data to map a "contextual dataset" of intent score values ​​output by a machine learning model to a "translated" dataset of skill scores for selecting skills to respond to a natural language-based query. The mapping data may take the form of a matrix, where the rows and columns of the matrix correspond to skills and intents, respectively, and vice versa.

[0032] Exemplary Chatbot System A bot (also referred to as a skill, chatbot, chatterbot, or talkbot) is a computer program that can conduct a conversation with an end user. A bot can generally respond to natural language messages (e.g., questions or comments) through a messaging application using natural language messages. A business can use one or more bot systems to communicate with end users through messaging applications. The messaging application is called a channel, and is the end user's preferred messaging application that the end user already has installed and is familiar with. Thus, the end user does not need to download and install a new application to chat with a bot system. Messaging applications can include, for example, over-the-top (OTT) messaging channels (e.g., Facebook Messenger, Facebook WhatsApp, WeChat, Line, Kik, Telegram, Talk, Skype, Slack, or SMS), virtual private assistants (e.g., Amazon Dot, Echo, or Show, Google® Home, Apple HomePod, etc.), mobile and web app extensions that extend native or hybrid / responsive mobile apps or web applications with chat capabilities, or voice-based input (e.g., devices or apps with an interface that uses Siri, Cortana, Google Voice, or other voice input for interaction).

[0033] In some examples, the bot system is associated with a Uniform Resource Identifier (URI). The URI can identify the bot system using a string of characters. The URI is used as a web hook for one or more messaging application systems. The URI can include, for example, a Uniform Resource Locator (URL) or a Uniform Resource Name (URN). The bot system is designed to receive a message (e.g., a HyperText Transfer Protocol (HTTP) post call message) from the messaging application system. The HTTP post call message is directed to the URI from the messaging application system. In some embodiments, the message is different from the HTTP post call message. For example, the bot system may receive a message from a Short Message Service (SMS). Although the discussion herein may refer to a communication received by the bot system as a message, it should be understood that the message may be an HTTP post call message, an SMS message, or any other type of communication between the two systems.

[0034] End users can interact with bot systems through conversational interactions (sometimes called conversational user interfaces (UIs)), much like interactions between people. In some cases, the interactions may involve the end user saying "Hello" to the bot, and the bot responding with "Hi" and asking the end user how it can assist the end user. In some cases, the interactions may also be transactional interactions with a banking bot, e.g., transferring money from one account to another; informational interactions with an HR bot, e.g., checking a vacation balance; or interactions with a retail bot, e.g., discussing returning a purchased item or seeking technical support.

[0035] In some embodiments, the bot system can intelligently handle end user interactions without interaction with an administrator or developer of the bot system. For example, an end user may send one or more messages to the bot system to achieve a desired goal. The messages may include some content, such as text, emojis, audio, images, video, or other methods of conveying a message. In some embodiments, the bot system can convert the content into a standardized format (e.g., a representational state transfer (REST) ​​call to an enterprise service with appropriate parameters) and generate a natural language response. The bot system can also prompt the end user for additional input parameters or request other additional information. In some embodiments, the bot system can also initiate communication with the end user rather than passively responding to end user utterances. Various techniques are described herein for identifying explicit invocations of the bot system and determining inputs for the invoked bot system. In some embodiments, the explicit invocation analysis is performed by the parent bot based on detection of an invocation name in the utterance. In response to detection of the invocation name, the utterance is refined for input to a skill bot associated with the invocation name.

[0036] A conversation with a bot can follow a particular conversational flow that includes multiple states. The flow can define what happens next based on the input. In some embodiments, the bot system is implemented using a state machine that includes user-defined states (e.g., end user intents) and actions to be taken at or from the states. The conversation can take different paths based on the end user input, which can affect the decisions the bot makes about the flow. For example, at each state, based on the end user input or utterance, the bot can determine the end user intent to determine the appropriate action to take next. Here, and in the context of utterances, the term "intent" refers to the intent of the user who gave the utterance. For example, a user intends to engage a bot in a conversation to order a pizza, and the user's intent may be expressed by the utterance "order a pizza". The user's intent may be directed to a particular task that the user wants the chatbot to perform on the user's behalf. Thus, an utterance can be expressed as a question, command, request, etc. that reflects the user's intent. An intent may include a goal that the end user wants to achieve.

[0037] In the context of chat configuration, the term "intent" is used herein to refer to configuration information for mapping user utterances to specific tasks / actions or categories of tasks / actions that a chatbot can perform. To distinguish between utterance intents (i.e., user intents) and chatbot intents, the latter may be referred to herein as "bot intents." A bot intent may include a set of one or more utterances associated with the intent. For example, an intent for ordering pizza may have various permutations of utterances that express a desire to place a pizza order. These associated utterances may be used to train the chatbot's intent classifier, which then allows the intent classifier to determine whether an input utterance from a user matches the pizza ordering intent. A bot intent is associated with one or more dialog flows for initiating a conversation with a user in a state. For example, a first message for a pizza ordering intent may be the question, "What kind of pizza would you like?" In addition to the associated utterances, a bot intent may further include a named entity associated with the intent. For example, a pizza ordering intent may include variables or parameters used to perform the task of ordering a pizza, such as topping 1, topping 2, type of pizza, pizza size, pizza quantity, etc. The values ​​of the entities are typically obtained through conversation with a user.

[0038] FIG. 1 is a simplified block diagram of an environment 100 incorporating a chatbot system according to certain embodiments. The environment 100 includes a digital assistant builder platform (DABP) 102, which allows users of the DABP 102 to create and deploy digital assistant or chatbot systems. The DABP 102 can be used to create one or more digital assistants (or DAs) or chatbot systems. For example, as shown in FIG. 1, a user 104 representing a particular business can use the DABP 102 to create and deploy a digital assistant 106 for users of the particular business. For example, a bank can use the DABP 102 to create one or more digital assistants for use by customers of the bank. Multiple businesses can use the same DABP 102 platform to create digital assistants. As another example, an owner of a restaurant (e.g., a pizza shop) can use the DABP 102 to create and deploy a digital assistant that enables customers of the restaurant to order food (e.g., order pizza).

[0039] For purposes of this disclosure, a "digital assistant" is an entity that helps a user of the digital assistant accomplish various tasks through natural language conversation. A digital assistant may be implemented solely with software (e.g., a digital assistant is a digital entity implemented with programs, codes, or instructions executable by one or more processors), with hardware, or with a combination of hardware and software. A digital assistant may be embodied or implemented in various physical systems or devices, such as a computer, a mobile phone, a watch, an appliance, a vehicle, etc. A digital assistant may also be referred to as a chatbot system. Thus, for purposes of this disclosure, the terms digital assistant and chatbot system are interchangeable.

[0040] A digital assistant, such as digital assistant 106 built using DABP 102, can be used to perform a variety of tasks through natural language-based conversations between the digital assistant and its user 108. As part of the conversation, the user may provide one or more user inputs 110 to the digital assistant 106 and obtain responses 112 from the digital assistant 106. A conversation can include one or more of the inputs 110 and responses 112. Through these conversations, the user can request one or more tasks to be performed by the digital assistant 106, and in response, the digital assistant 106 is configured to perform the user-requested task and respond to the user with an appropriate response.

[0041] The user input 110 is generally in a natural language format and is referred to as an utterance. The user utterance 110 may be in a text format, such as when the user types a sentence, a question, a piece of text, or even a single word, and provides it as input to the digital assistant 106. In some embodiments, the user utterance 110 may be in a voice input or speech format, such as when the user says or speaks something that is provided as input to the digital assistant 106. The utterance is typically in a language spoken by the user 108. For example, the utterance is in English or some other language. If the utterance is in a voice format, the voice input is converted into a text format utterance in that particular language, and the text utterance is then processed by the digital assistant 106. Various voice-to-text processing techniques are used to convert the voice or auditory input into a text utterance, which is then processed by the digital assistant 106. In some embodiments, the voice-to-text conversion is performed by the digital assistant 106 itself.

[0042] An utterance, which may be a text utterance or a voice utterance, may be a fragment, a sentence, multiple sentences, one or more words, one or more questions, a combination of the aforementioned types, etc. The digital assistant 106 is configured to apply natural language understanding (NLU) techniques to the utterance to understand the meaning of the user input. As part of the NLU processing on the utterance, the digital assistant 106 is configured to perform processing to understand the meaning of the utterance, which involves identifying one or more intents and one or more entities that correspond to the utterance. Upon understanding the meaning of the utterance, the digital assistant 106 may perform one or more actions or operations in response to the understood meaning or intent. For purposes of this disclosure, it is assumed that the utterance is a text utterance provided directly by a user 108 of the digital assistant 106 or is the result of a conversion of an input voice utterance into text format. However, this is not intended to be limiting or restrictive in any manner.

[0043] For example, user 108 input may request that a pizza be ordered by providing an utterance such as "I want to order a pizza." Upon receiving such an utterance, digital assistant 106 is configured to understand the meaning of the utterance and take appropriate action. The appropriate action may include responding to the user with a question requesting user input regarding, for example, the type of pizza the user wants to order, the size of the pizza, any toppings on the pizza, etc. The responses provided by digital assistant 106 may also be in natural language form, typically in the same language as the input utterance. As part of generating these responses, digital assistant 106 may perform natural language generation (NLG). Through a conversation between the user and digital assistant 106, in order for the user to order a pizza, the digital assistant may guide the user to provide all the necessary information to order the pizza, and then, at the end of the conversation, have the pizza ordered. Digital assistant 106 may end the conversation by outputting information to the user indicating that the pizza has been ordered.

[0044] At a conceptual level, the digital assistant 106 performs various processes in response to utterances received from a user. In some embodiments, this process involves a series of processing steps or a pipeline of processing steps, including, for example, understanding the meaning of the input utterance (sometimes referred to as natural language understanding (NLU)), determining an action to be performed in response to the utterance, causing the action to be performed if appropriate, generating a response to be output to the user in response to the user utterance, outputting the response to the user, etc. NLU processing can include parsing the received input utterance to understand the structure and meaning of the utterance, and refining and reconstructing the utterance to develop a more understandable form (e.g., logical form) or structure for the utterance. Generating a response may include using NLG techniques.

[0045] The NLU processing performed by a digital assistant such as digital assistant 106 may include various NLP-related processing such as sentence analysis (e.g., tokenization, reordering, identifying part-of-speech tags for sentences, identifying named entities in sentences, generating dependency trees to represent sentence structure, splitting sentences into clauses, analyzing individual clauses, resolving anaphoric forms, performing chunking, etc.). In an embodiment, the NLU processing, or portions thereof, is performed by digital assistant 106 itself. In some other embodiments, digital assistant 106 may use other resources to perform portions of the NLU processing. For example, the syntax and structure of an input spoken sentence is identified by processing the sentence with parsing, part-of-speech tagging, and / or named entity recognition. In one implementation, for English, parsing, part-of-speech tagging, and named entity recognition, such as those provided by the Stanford Natural Language Processing (NLP) Group, are used to analyze sentence structure and syntax. These are provided as part of the Stanford CoreNLP toolkit.

[0046] Although the various examples provided in this disclosure show English speech, this is meant as an example only. In certain embodiments, the digital assistant 106 can also process speech in languages ​​other than English. The digital assistant 106 may provide subsystems (e.g., components that implement NLU functionality) configured to perform processing for different languages. These subsystems are implemented as pluggable units that can be invoked using service calls from the NLU core server. This makes the NLU processing flexible and extensible for each language, including allowing for different orders of processing. Language packs are provided for each language, and the language packs can register a list of subsystems that can be served from the NLU core server.

[0047] A digital assistant, such as the digital assistant 106 shown in FIG. 1, can be made available or accessible to its user 108 through a variety of different channels, such as, but not limited to, through an application, through a social media platform, through various messaging services and applications, and through other applications or channels. A single digital assistant can have several channels configured for it, so that it can run on and be accessed by different services simultaneously.

[0048] A digital assistant or chatbot system typically includes or is associated with one or more skills. In an embodiment, these skills are individual chatbots (called skillbots) that interact with a user and are configured to fulfill a particular type of task, such as tracking inventory, submitting a timecard, creating an expense report, ordering food, checking a bank account, making a reservation, purchasing a widget, etc. For example, in the embodiment shown in FIG. 1, the digital assistant or chatbot system 106 includes skills 116-1, 116-2, etc. For purposes of this disclosure, the term "skill" is used synonymously with the term "skillbot."

[0049] Each skill associated with a digital assistant helps a user of the digital assistant complete a task through a conversation with the user, where the conversation can include a combination of text or auditory input provided by the user and responses provided by the skill bot. These responses can be in the form of text or auditory messages to the user and / or with simple user interface elements (e.g., selection lists) that are presented to the user for the user to make a selection.

[0050] There are various ways in which skills or skillbots can be associated with or added to a digital assistant. In one example, a skillbot can be developed by a company and then added to a digital assistant using DABP 102. In another example, a skillbot can be developed and created using DABP 102 and then added to a digital assistant created using DABP 102. In yet another example, DABP 102 provides an online digital store (referred to as a "skill store") that offers multiple skills directed to a wide range of tasks. Skills offered through the skill store may also expose various cloud services. To add a skill to a digital assistant created using DABP 102, a user of DABP 102 can access the skill store via DABP 102, select the desired skill, and indicate that the selected skill is to be added to the digital assistant created using DABP 102. Skills from the skill store can be added to a digital assistant as is or in modified form (e.g., a user of DABP102 may select and clone a particular skillbot offered by the skill store, customize or modify the selected skillbot, and then add the modified skillbot to a digital assistant created with DABP102).

[0051] A variety of different architectures are used to realize digital assistant or chatbot systems. For example, in one embodiment, a digital assistant created and deployed using DABP 102 is realized using a parent bot / child (or sub) bot paradigm or architecture. According to this paradigm, a digital assistant is realized as a parent bot that interacts with one or more child bots, which are skill bots. For example, in the embodiment shown in FIG. 1, the digital assistant 106 includes a parent bot 114 and skill bots 116-1, 116-2, etc., that are child bots of the parent bot 114. In certain embodiments, the digital assistant 106 itself may act as a parent bot.

[0052] A digital assistant realized according to the parent-child bot architecture allows a user of the digital assistant to interact with multiple skills through a unified user interface, i.e., through the parent bot. When a user engages with the digital assistant, the user input is received by the parent bot. The parent bot then performs processing to determine the meaning of the user input utterance. The parent bot then determines whether the task requested by the user in the utterance can be handled by the parent bot itself, and if not, the parent bot selects an appropriate skill bot to handle the user request and routes the conversation to the selected skill bot. This allows a user to converse with the digital assistant through a common single interface and still provide the ability to use several skill bots configured to perform specific tasks. For example, in the case of a digital assistant developed for an enterprise, the parent bot of the digital assistant may interface with skill bots having specific functions, such as a CRM bot for performing functions related to customer relationship management (CRM), an ERP bot for performing functions related to enterprise resource planning (ERP), an HCM bot for performing functions related to human capital management (HCM), etc. In this way, end users or consumers of the digital assistant only need to know how to access the digital assistant through a common parent bot interface, and behind the scenes, multiple skill bots are provided to handle user requests.

[0053] In one embodiment, in a parent bot / child bot infrastructure, the parent bot is configured to be aware of an available list of skill bots. The parent bot may have access to various available skill bots and, for each skill bot, metadata identifying each skill bot's capabilities, including tasks that can be performed by each skill bot. Upon receiving a user request in the form of an utterance, the parent bot is configured to identify or predict, from multiple available skill bots, a specific skill bot that can best accommodate or process the user request. The parent bot then routes the utterance (or a portion of the utterance) to that specific skill bot for further processing. Thus, control flows from the parent bot to the skill bot. The parent bot may support multiple input and output channels.

[0054] The embodiment of FIG. 1 shows a digital assistant 106 with a parent bot 114 and skill bots 116-1, 116-2, and 116-3, but this is not intended to be limiting. The digital assistant can include various other components (e.g., other systems and subsystems) that provide the functionality of the digital assistant. These systems and subsystems can be realized in an implementation using only software (e.g., code, instructions stored on a computer-readable medium and executable by one or more processors), only hardware, or a combination of software and hardware.

[0055] DABP 102 provides infrastructure and various services and features that enable a user of DABP 102 to create a digital assistant including one or more skill bots associated with the digital assistant. In some cases, a skill bot can be created by cloning an existing skill bot, for example, by cloning a skill bot provided by a skill store. As described above, DABP 102 provides a skill store or skill catalog that provides multiple skill bots for performing various tasks. A user of DABP 102 can clone a skill bot from the skill store. Modifications or customizations are made to the cloned skill bot as needed. In some other cases, a user of DABP 102 creates a skill bot from scratch using tools and services provided by DABP 102. As described above, a skill store or skill catalog provided by DABP 102 may provide multiple skill bots for performing various tasks.

[0056] In certain embodiments, at one high level, creating or customizing a skillbot includes the following steps: (1) Set up the settings for the new skill bot (2) Configure one or more intents for the skill bot (3) Set one or more entities for one or more intents (4) Train the Skillbot (5) Create a dialogue flow for your skill bot (6) Add custom components to your skill bot as needed (7) Test and deploy the skill bot. Each step will be briefly described below.

[0057] (1) Set Settings for a New Skillbot - Various settings are configured for a skillbot. For example, a skillbot designer can specify one or more call names for the skillbot being created. These call names can then be used by a user of the digital assistant to explicitly call the skillbot. For example, a user can enter a call name in the user's utterance to explicitly call the corresponding skillbot.

[0058] (2) Set one or more intents and associated example utterances for the skillbot - A skillbot designer specifies one or more intents (also called bot intents) for the skillbot being created. The skillbot is then trained based on these specified intents. These intents represent categories or classes that the skillbot is trained to infer about input utterances. Upon receiving an utterance, the trained skillbot infers the intent of the utterance, and the inferred intent is selected from a predefined set of intents used to train the skillbot. The skillbot then takes an appropriate action to respond to the utterance based on the intent inferred for the utterance. In some cases, the intents for a skillbot represent tasks that the skillbot can perform for a user of the digital assistant. Each intent is given an intent identifier or intent name. For example, for a skillbot trained for banking, the specified intents for the skillbot may include "CheckBalance", "TransferMoney", "DepositCheck", etc.

[0059] For each intent defined for a skillbot, the skillbot designer may also provide one or more example utterances that are representative and illustrative of that intent. These example utterances are meant to represent utterances that a user may input to the skillbot for that intent. For example, for a balance inquiry intent, example utterances may include "What's my savings account balance?", "How much is in my checking account?", "How much money do I have in my account?", etc. Thus, various permutations of typical user utterances are specified as example utterances for the intent.

[0060] The intents and their associated example utterances are used as training data to train the skill bot. A variety of different training techniques are used. As a result of this training, a predictive model is generated that is configured to take an utterance as input and output an intent inferred for the utterance by the predictive model. In some cases, the input utterance is provided to an intent analysis engine that is configured to predict or infer an intent for the input utterance using the trained model. The skill bot may then take one or more actions based on the inferred intent.

[0061] (3) Setting one or more entities for one or more intents - In some instances, additional context is required to enable the skill bot to respond appropriately to a user utterance. For example, there are situations in which a user input utterance resolves to the same intent in the skill bot. For example, in the above example, the utterances "What's my savings account balance?" and "How much is in my checking account?" both resolve to the same balance inquiry intent, but these utterances are different requests that want different things. To disambiguate such requests, one or more entities are added to the intent. Using the banking skill bot example, an entity called AccountType that defines values ​​called "checking" and "saving" may enable the skill bot to parse the user request and respond appropriately. In the above example, the utterances resolve to the same intent, but the value associated with the AccountType entity is different for the two utterances. This allows the skill bot to perform possibly different actions for two utterances even though they resolve to the same intent. One or more entities may be specified for a particular intent configured for the skill bot. Thus, entities are used to add context to the intent itself. Entities help to more fully describe the intent and allow the skill bot to complete the user request.

[0062] In one embodiment, there are two types of entities: (a) built-in entities provided by DABP 102, and (2) custom entities that may be specified by a skill bot designer. Built-in entities are generic entities that can be used with a wide variety of bots. Examples of built-in entities include, but are not limited to, entities related to time, date, address, number, email address, duration, circulation period, currency, phone number, URL, etc. Custom entities are used for more customized applications. For example, for a banking skill, an AccountType entity is defined by the skill bot designer to enable various banking transactions by checking user input for keywords such as current, regular, and credit card, etc.

[0063] (4) Train the skillbot - The skillbot is configured to receive user input in the form of utterances, parse or otherwise process the received input, and identify or select an intent associated with the received user input. As described above, the skillbot must be trained for this. In an embodiment, the skillbot is trained based on the intents configured for the skillbot and example utterances associated with the intents (collectively, training data) such that the skillbot can resolve the user input utterance to one of the skillbot's configured intents. In a particular embodiment, the skillbot is trained with the training data and uses a predictive model that allows the skillbot to identify what the user is saying (or, in some cases, what the user is trying to say). DABP 102 provides a variety of different training techniques that may be used by the skillbot designer to train the skillbot, including various machine learning based training techniques, rule-based training techniques, and / or combinations thereof. In an embodiment, a portion of the training data (e.g., 80%) is used to train the skillbot model, and another portion (e.g., the remaining 20%) is used to test or validate the model. Once trained, the trained model (sometimes called a trained skill bot) can then be used to process and respond to user utterances. In some cases, the user utterance is a question that requires only a single answer and does not require further conversation. To address such situations, we define a Q&A (Question and Answer) intent for the skill bot. This allows the skill bot to output a reply to a user request without the need to update the dialog definition. A Q&A intent is generated similarly to a normal intent. The dialog flow for a Q&A intent may differ from the dialog flow for a normal intent.

[0064] (5) Create a Dialog Flow for a Skill Bot - The dialog flow specified for a skill bot describes how the skill bot reacts when different intents for the skill bot are resolved in response to received user input. The dialog flow defines the behavior or actions taken by the skill bot, such as how the skill bot responds to user utterances, how the skill bot prompts the user for input, and how the skill bot returns data. The dialog flow is like a flowchart that the skill bot follows. Skill bot designers specify the dialog flow using a language such as Markdown language. In one embodiment, a version of YAML called OBotML is used to specify the dialog flow for a skill bot. The dialog flow definition for a skill bot serves as a model of the conversation itself, allowing the skill bot designer to choreograph the interaction between the skill bot and the user that the skill bot serves.

[0065] In one embodiment, a skill bot's dialog flow definition includes three sections: (a) Context Section (b) Default transition section (c) Status section.

[0066] Context Section - In the context section, the skill bot designer can define variables that will be used in the conversation flow. Other variables named in the context section include, but are not limited to, variables for error handling, variables for built-in or custom entities, user variables that allow the skill bot to recognize and persist user preferences, etc.

[0067] Default Transitions Section - Transitions for a skill bot can be defined in the dialog flow states section or the default transitions section. Transitions defined in the default transitions section act as fallbacks and are triggered when there is no applicable transition defined in a state or the conditions required to trigger a state transition are not met. The default transitions section can be used to define routing that allows the skill bot to gracefully handle unexpected user actions.

[0068] State Section - A dialog flow and its associated behaviors are defined as a series of temporary states that govern the logic within the dialog flow. Each state node in a dialog flow definition names a component that provides the functionality needed at that point in the dialog. In this way, we build states around the components. States contain component-specific characteristics and define transitions to other states that are triggered after the component is executed.

[0069] Special case scenarios are handled using the state section. For example, you might want to give a user the option to temporarily exit a first skill that the user is working on and do something in a second skill within the digital assistant. For example, if the user is engaged in a conversation with a shopping skill (e.g., the user has made some selections for a purchase), the user might want to jump to a banking skill (e.g., the user might want to make sure they have enough money for the purchase) and then return to the shopping skill to complete the user's order. To address this, an action in the first skill can be configured to initiate an interaction with a second, different skill in the same digital assistant and then return to the original flow.

[0070] (6) Adding Custom Components to a Skillbot - As described above, a state specified in a dialog flow for a skillbot nominates a component that provides the required functionality corresponding to that state. The component enables the skillbot to perform the functionality. In an embodiment, DABP 102 provides a set of pre-configured components to perform a wide range of functions. A skillbot designer can select one or more of these pre-configured components and associate them with a state in the dialog flow for the skillbot. A skillbot designer can also create custom or new components using tools provided by DABP 102 and associate the custom components with one or more states in the dialog flow for the skillbot.

[0071] (7) Testing and Deploying Skillbots - DABP 102 provides several features that allow skillbot designers to test the skillbots they are developing. The skillbots can then be deployed and included in a digital assistant.

[0072] While the above description describes how to create a skill bot, similar techniques can also be used to create a digital assistant (or parent bot). At the parent bot or digital assistant level, built-in system intents are set for the digital assistant. These built-in system intents are used to identify common tasks that the digital assistant itself (i.e., the parent bot) can handle without invoking a skill bot associated with the digital assistant. Examples of system intents defined for the parent bot include: (1) Exit: applies when the user signals in the digital assistant that they want to end the current conversation or context; (2) Help: applies when the user asks for help or direction; (3) UnresolvedIntent: applies to user input that does not match well with the Exit and Help intents. The digital assistant also stores information about one or more skill bots associated with the digital assistant. This information allows the parent bot to select a specific skill bot to process an utterance.

[0073] At the parent bot or digital assistant level, when a user inputs a phrase or utterance into the digital assistant, the digital assistant is configured to process to determine how to route the utterance and associated conversation. The digital assistant determines this using a routing model, which can be rule-based, AI-based, or a combination thereof. The digital assistant uses the routing model to determine whether the conversation corresponding to the user input utterance should be routed to a specific skill for processing, should be processed by the digital assistant or parent bot itself according to built-in system intents, or should be processed as a different state in the current conversation flow.

[0074] In certain embodiments, as part of this processing, the digital assistant determines whether the user input utterance explicitly identifies a skillbot with its call name. If the call name is present in the user input, it is treated as an explicit call of the skillbot corresponding to the call name. In such a scenario, the digital assistant can route the user input to the explicitly called skillbot for further processing. In the absence of a specific or explicit call, in one embodiment, the digital assistant evaluates the received user input utterance and calculates a confidence score for the system intent and skillbot associated with the digital assistant. The calculated score for the skillbot or system intent represents the likelihood that the user input represents a task that the skillbot is configured to perform or represents a system intent. The system intent or skillbot whose associated calculated confidence score exceeds a threshold (e.g., a Confidence Threshold routing parameter) is selected as a candidate for further evaluation. The digital assistant then selects a specific system intent or skillbot from the identified candidates for further processing of the user input utterance. In a particular embodiment, after one or more skill bots are identified as candidates, the intents associated with those candidate skills are evaluated (according to the intent model for each skill) and a confidence score is determined for each intent. In general, intents with a confidence score above a threshold (e.g., 70%) are treated as candidate intents. If a particular skill bot is selected, the user utterance is routed to that skill bot for further processing. If a system intent is selected, one or more actions are performed by the parent bot itself according to the selected system intent.

[0075] FIG. 2 is a simplified block diagram of a parent bot (MB) system 200, according to an embodiment. The MB system 200 can be implemented in software only, hardware only, or a combination of hardware and software. The MB system 200 includes a pre-processing subsystem 210, a multiple intent subsystem (MIS) 220, an explicit invocation subsystem (EIS) 230, a skill bot invocation unit 240, and a data store 250. The MB system 200 shown in FIG. 2 is merely an example of an arrangement of components in a parent bot. Those skilled in the art will recognize many possible variations, alternatives, and modifications. For example, in some implementations, the MB system 200 may have more or fewer systems or components than those shown in FIG. 2, may combine two or more subsystems, or may have subsystems in different configurations or arrangements.

[0076] The pre-processing subsystem 210 receives an utterance "A" 202 from a user and processes the utterance through a language detector 212 and a language parser 214. As discussed above, the utterance may be provided in a variety of ways, including as audio or text. The utterance 202 may be a fragment, a complete sentence, multiple sentences, etc. The utterance 202 may include punctuation. For example, if the utterance 202 is provided as audio, the pre-processing subsystem 210 may convert the speech to text using a speech-to-text converter (not shown) that inserts punctuation marks, e.g., commas, semicolons, periods, etc., into the resulting text.

[0077] The language detection unit 212 detects the language of the utterance 202 based on the text of the utterance 202. Since each language has its own grammar and semantics, the manner in which the utterance 202 is processed depends on the language. The language differences are taken into account when analyzing the syntax and structure of the utterance.

[0078] The language parser 214 parses the utterance 202 to extract part-of-speech (POS) tags for individual linguistic units (e.g., words) in the utterance 202. POS tags include, for example, noun (NN), pronoun (PN), verb (VB), etc. The language parser 214 may also tokenize the linguistic units of the utterance 202 (e.g., to convert each word into a separate token) and lemmatize the words. A lemma is the primary form of a set of words represented in a dictionary (e.g., "run" is a lemma for run, runs, ran, running, etc.). Other types of preprocessing that the language parser 214 may perform include chunking of compound expressions, for example, combining "credit" and "card" into a single expression "credit_card". The language parser 214 may also identify relationships between words in the utterance 202. For example, in some embodiments, the language parser 214 generates a dependency tree that indicates which parts of the utterance (e.g., particular nouns) are direct objects, which parts of the utterance are prepositions, etc. The results of the processing performed by the language parser 214 form the extracted information 205, which is provided as input to the MIS 220 along with the utterance 202 itself.

[0079] As mentioned above, utterance 202 may include multiple sentences. For purposes of detecting multiple intents and explicit invocations, utterance 202 may be treated as a single unit even if it includes multiple sentences. However, in an embodiment, preprocessing may be performed, for example by preprocessing subsystem 210, to identify a single sentence among multiple sentences for multiple intent analysis and explicit invocation analysis. In general, the results generated by MIS 220 and EIS 230 are substantially the same whether utterance 202 is processed at the level of individual sentences or as a single unit including multiple sentences.

[0080] The MIS 220 determines whether the utterance 202 represents multiple intents. Although the MIS 220 can detect the presence of multiple intents in the utterance 202, the processing performed by the MIS 220 does not involve determining whether the intent of the utterance 202 matches any intent configured for the bot. Instead, the processing to determine whether the intent of the utterance 202 matches a bot intent may be performed by the intent classifier 242 of the MB system 200 (e.g., as shown in the embodiment of FIG. 3) or by an intent classifier of the skill bot. The processing performed by the MIS 220 assumes that there is a bot (e.g., a particular skill bot or the parent bot itself) that can process the utterance 202. Thus, the processing performed by the MIS 220 does not require knowledge of what bots are in the chatbot system (e.g., the identity of the skill bot registered with the parent bot) or what intents are set for a particular bot.

[0081] To determine that an utterance 202 includes multiple intents, the MIS 220 applies one or more rules from a set of rules 252 in the data store 250. The rules applied to the utterance 202 depend on the language of the utterance 202 and may include a sentence pattern that indicates the presence of multiple intents. For example, a sentence pattern may include a conjunction that connects two parts of a sentence (e.g., a conjunct), both parts corresponding to separate intents. If the utterance 202 matches the sentence pattern, it can be inferred that the utterance 202 represents multiple intents. Note that an utterance with multiple intents does not necessarily have different intents (e.g., intents directed to different bots, or different intents within the same bot). Instead, the utterance may have separate instances of the same intent, such as "order pizza using payment account X, then order pizza using payment account Y."

[0082] As part of determining that utterance 202 represents multiple intents, MIS 220 also determines what portions of utterance 202 are associated with each intent. For each intent expressed in the multiple intent utterance, MIS 220 constructs a new utterance for separate processing in place of the original utterance, e.g., utterance “B” 206 and utterance “C” 208, as shown in FIG. 2. Thus, original utterance 202 may be split into two or more separate utterances that are handled one at a time. MIS 220 determines which of the two or more utterances should be processed first using extracted information 205 and / or from an analysis of utterance 202 itself. For example, MIS 220 may determine that utterance 202 includes a marker word indicating that a particular intent should be handled first. The newly formed utterance corresponding to this particular intent (e.g., one of utterances 206 or utterance 208) will be sent first for further processing by EIS 230. After the conversation triggered by the first utterance has ended (or been temporarily interrupted), the next highest priority utterance (e.g., the other of utterance 206 or utterance 208) may then be sent to EIS 230 for processing.

[0083] The EIS 230 determines whether the received utterance (e.g., utterance 206 or utterance 208) includes a call name of the skill bot. In an embodiment, each skill bot in the chatbot system is assigned a unique call name that distinguishes the skill bot from other skill bots in the chatbot system. A list of call names can be maintained in the data store 250 as part of the skill bot information 254. When the utterance includes a word that matches the call name, the utterance is considered to be an explicit call. If the bot is not explicitly called, the utterance received by the EIS 230 is considered an implicit call utterance 234 and input to the parent bot's intent classifier (e.g., intent classifier 242) to determine which bot to use to process the utterance. In some examples, the intent classifier 242 determines that the parent bot should process the implicit call utterance. In other examples, the intent classifier 242 determines the skill bot to route the utterance for processing.

[0084] The explicit invocation feature provided by the EIS 230 has several advantages. It can reduce the amount of processing that the parent bot must perform. For example, if there is an explicit invocation, the parent bot may not have to perform any intent classification analysis (e.g., using the intent classifier 242) or may have to perform reduced intent classification analysis to select a skill bot. Thus, the explicit invocation analysis may enable the selection of a particular skill bot without relying on intent classification analysis.

[0085] There are also situations where there is overlap in functionality between multiple skillbots. This can occur, for example, when the intents handled by two skillbots overlap or are very close to each other. In such situations, it may be difficult for a parent bot to identify which of multiple skillbots to select based on intent classification analysis alone. In such scenarios, an explicit invocation disambiguates the specific skillbot to be used.

[0086] In addition to determining that the utterance is an explicit invocation, EIS 230 is responsible for determining whether any portion of the utterance should be used as input to the explicitly invoked skill bot. In particular, EIS 230 may determine whether a portion of the utterance is not associated with an invocation. EIS 230 may make this determination through analysis of the utterance and / or analysis of extracted information 205. Instead of sending the entire utterance received by EIS 230, EIS 230 may send the portion of the utterance that is not associated with an invocation to the invoked skill bot. In some examples, the input to the invoked skill bot is formed by simply removing any portion of the utterance that is associated with an invocation. For example, "I want to order a pizza using Pizza Bot" may be shortened to "I want to order a pizza" because "using Pizza Bot" pertains to the invocation of the Pizza Bot but not to any processing performed by the Pizza Bot. In some examples, EIS 230 may reformat the portion to be sent to the invoked bot, for example to form a complete sentence. Thus, EIS 230 determines not only that there is an explicit call, but also what to send to the skill bot when there is an explicit call. In some instances, there may be no text to input to the called bot. For example, if the utterance was "Pizza Bot," EIS 230 may determine that the Pizza Bot is being called, but there is no text to be processed by the Pizza Bot. In such a scenario, EIS 230 can indicate to the skill bot caller 240 that there is nothing to send.

[0087] The skillbot invoker 240 invokes the skillbot in various manners. For example, the skillbot invoker 240 can invoke the bot in response to receiving an indication 235 that a particular skillbot has been selected as a result of an explicit invoke. The indication 235 can be sent by the EIS 230 along with an input for the explicitly invoked skillbot. In this scenario, the skillbot invoker 240 hands over control of the conversation to the explicitly invoked skillbot. The explicitly invoked skillbot determines an appropriate response to the input from the EIS 230 by treating the input as an independent utterance. For example, the response can be to perform a particular action or to start a new conversation in a particular state, where the initial state of the new conversation depends on the input sent from the EIS 230.

[0088] Another manner in which the skillbot invoker 240 can invoke a skillbot is by implicit invocation using the intent classifier 242. The intent classifier 242 can be trained using machine learning and / or rule-based training techniques to determine the likelihood that an utterance represents a task that a particular skillbot is configured to perform. The intent classifier 242 is trained with different classes, one class for each skillbot. For example, each time a new skillbot is registered with a parent bot, a list of example utterances associated with the new skillbot can be used to train the intent classifier 242 to determine the likelihood that a particular utterance represents a task that the new skillbot can perform. The parameters (e.g., a set of values ​​for the parameters of the machine learning model) generated as a result of this training can be stored as part of the skillbot information 254.

[0089] In an embodiment, the intent classifier 242 is realized using a machine learning model, as described in more detail herein. Training the machine learning model may include inputting at least a subset of utterances from example utterances associated with various skill bots to generate, as an output of the machine learning model, an inference about which bot is the correct bot to process any particular training utterance. For each training utterance, an indication of the correct bot to use for that training utterance is provided as ground truth information. The behavior of the machine learning model can then be adapted (e.g., through backpropagation) to minimize the difference between the generated inference and the ground truth information.

[0090] In certain embodiments, the intent classifier 242 determines a confidence score for each skill bot registered with the parent bot, indicating the likelihood that the skill bot can process an utterance (e.g., an implicit invocation utterance 234 received from the EIS 230). The intent classifier 242 may also determine a confidence score for each configured system-level intent (e.g., help, exit). If a particular confidence score meets one or more conditions, the skill bot invoker 240 will invoke the bot associated with the particular confidence score. For example, a threshold confidence score value may need to be met. Thus, the output 245 of the intent classifier 242 is either an identification of a system intent or an identification of a particular skill bot. In some embodiments, in addition to meeting the threshold confidence score value, the confidence score must exceed the next higher confidence score by a certain winning margin. Imposing such conditions allows routing to a particular skill bot when the confidence scores of multiple skill bots each exceed the threshold confidence score value.

[0091] After identifying the bot based on the evaluation of the confidence score, the skillbot invoker 240 hands over the processing to the identified bot. In the case of a system intent, the identified bot is a parent bot. Otherwise, the identified bot is a skill bot. Furthermore, the skillbot invoker 240 will determine what to provide as an input 247 to the identified bot. As described above, in the case of an explicit invoke, the input 247 can be based on a portion of the utterance that is not associated with the invoke, or the input 247 can be nothing (e.g., an empty string). In the case of an implicit invoke, the input 247 can be the entire utterance.

[0092] The data store 250 comprises one or more computing devices that store data used by various subsystems of the parent bot system 200. As described above, the data store 250 includes the rules 252 and the skill bot information 254. The rules 252 include, for example, rules for determining by the MIS 220 when an utterance represents multiple intents and how to split an utterance representing multiple intents. The rules 252 further include rules for determining by the EIS 230 which part of an utterance that explicitly invokes a skill bot should be sent to the skill bot. The skill bot information 254 includes the call names of the skill bots in the chatbot system, for example, a list of the call names of all the skill bots registered to a particular parent bot. The skill bot information 254 also includes information used by the intent classifier 242 to determine a confidence score for each skill bot in the chatbot system, for example, parameters of a machine learning model.

[0093] 3 is a simplified block diagram of a Skillbot system 300 according to certain embodiments. The Skillbot system 300 is a computing system that may be implemented in software only, hardware only, or a combination of hardware and software. In some embodiments, such as the embodiment shown in FIG. 1, the Skillbot system 300 can be used to implement one or more Skillbots within a digital assistant.

[0094] The skillbot system 300 includes an MIS 310, an intent classifier 320, and a conversation manager 330. The MIS 310 is similar to the MIS 220 of FIG. 2 and provides similar functionality, including being operable to determine (1) whether an utterance represents multiple intents, and if so, (2) how to split the utterance into separate utterances for each of the multiple intents using rules 352 in a data store 350. In an embodiment, the rules applied by the MIS 310 to detect multiple intents and split the utterance are the same as the rules applied by the MIS 220. The MIS 310 receives the utterance 302 and the extracted information 304. The extracted information 304 is similar to the extracted information 205 of FIG. 1 and can be generated using the language parser 214 or a language parser local to the skillbot system 300.

[0095] The intent classifier 320 may be trained in a manner similar to the intent classifier 242 discussed above in connection with the embodiment of FIG. 4, and as described in further detail herein. For example, in certain embodiments, the intent classifier 320 is implemented using a machine learning model. The machine learning model of the intent classifier 320 is trained for a particular skill bot using at least a subset of example utterances associated with that particular skill bot as training utterances. The ground truth for each training utterance will be the particular bot intent associated with that training utterance.

[0096] The utterance 302 may be received directly from a user or may be provided via a parent bot. When the utterance 302 is provided through a parent bot, for example as a result of processing through the MIS 220 and the EIS 230 in the embodiment shown in FIG. 4, the MIS 310 may be bypassed to avoid repeating the processing already performed by the MIS 220. However, if the utterance 302 is received directly from a user, for example during a conversation that occurs after routing to a skill bot, the MIS 310 may process the utterance 302 to determine whether the utterance 302 represents multiple intents. If the utterance 302 represents multiple intents, the MIS 310 applies one or more rules to split the utterance 302 into separate utterances for each intent, for example, utterance "D" 306 and utterance "E" 308. If the utterance 302 does not represent multiple intents, the MIS 310 forwards the utterance 302, without segmentation, to the intent classifier 320 for intent classification.

[0097] The intent classifier 320 is configured to match the received utterance (e.g., utterance 306 or 308) with an intent associated with the skillbot system 300. As explained above, a skillbot can be configured with one or more intents, each of which includes at least one example utterance associated with the intent and used to train the classifier. In the embodiment of FIG. 2, the intent classifier 242 of the parent bot system 200 is trained to determine the confidence scores of individual skillbots and the confidence scores of system intents. Similarly, the intent classifier 320 can be trained to determine the confidence scores of each intent associated with the skillbot system 300. The classification performed by the intent classifier 242 is at the bot level, while the classification performed by the intent classifier 320 is at the intent level and thus is of finer granularity. The intent classifier 320 has access to the intent information 354. The intent information 354 includes, for each intent associated with the skillbot system 300, a list of utterances that represent the meaning of the intent and are typically associated with tasks that can be performed by the intent. The intent information 354 can further include parameters that are generated as a result of training on this list of utterances.

[0098] The conversation manager 330 receives as an output of the intent classifier 320 an indication 322 of the particular intent identified by the intent classifier 320 as the best match for the utterance input to the intent classifier 320. In some instances, the intent classifier 320 is unable to determine any match. For example, the confidence score calculated by the intent classifier 320 may fall below a threshold confidence score value if the utterance is directed to a system intent or to an intent of a different skill bot. When this occurs, the skill bot system 300 may refer the utterance to the parent bot for processing, e.g., for routing to a different skill bot. However, if the intent classifier 320 is successful in identifying the intent within the skill bot, the conversation manager 330 begins a conversation with the user.

[0099] The conversation initiated by the conversation manager 330 is specific to the intent identified by the intent classifier 320. For example, the conversation manager 330 is realized using a state machine configured to execute a certain dialog flow for the identified intent. The state machine can include a default starting state (e.g., for when the intent is invoked without any additional input) and one or more additional states, each state having associated therewith an action to be performed by the skill bot (e.g., perform a purchase transaction) and / or a dialog to be presented to the user (e.g., questions, responses). Thus, the conversation manager 330 can determine an action / dialog 335 upon receiving an instruction 322 identifying an intent, and can determine the additional action or dialog in response to a subsequent utterance received during the conversation.

[0100] The data store 350 comprises one or more computing devices that store data used by various subsystems of the skillbot system 300. As shown in Figure 3, the data store 350 includes rules 352 and intent information 354. In certain embodiments, the data store 350 can be integrated into a parent bot or digital assistant data store, such as the data store 250 of Figure 2.

[0101] Exemplary Data Processing System FIG. 4 is a simplified block diagram of a data processing system 400 according to some embodiments. The configuration shown in FIG. 4 is an exemplary embodiment of a data processing system that may be utilized to perform all or a portion of the methods described herein, but the system shown in FIG. 4 is not the only system that may perform these methods. The data processing system 400 may be implemented in software only, hardware only, or a combination of hardware and software. The data processing system 400 includes a communication portal 402. The communication portal 402 is any hardware, software, or instructions that enable communication between an external entity and the data processing system 400. In various embodiments, the communication portal 402 enables the exchange of messages between a user and a chatbot implemented by the data processing system 400. For example, a user may send an incoming utterance 406 to the data processing system 400 as part of an interaction with the chatbot. The incoming utterance 406 is received and / or interpreted according to user communication instructions 404. The user communication instructions 404 is any set of instructions or steps that enable the communication portal to capture the incoming utterance 406 or to send an outgoing interaction 410. In various embodiments, the chatbot routing instructions are implemented to send outgoing interactions 410 associated with the chatbot to a user.

[0102] When the incoming utterance 406 is received by the communication portal, it is routed to the speech converter 412. The speech converter 412 is any entity within the data processing system 400 that can convert the incoming utterance 406 into speech data 416. In various embodiments, the speech converter 412 includes conversion instructions 414, which may include instructions or steps for converting the incoming utterance 406 into speech data 416. The speech converter 412 changes the incoming utterance 406, received in a natural language based format, into input data that is received and analyzed by a classifier model, such as an intent classifier. For example, the incoming utterance 406 is speech data in the same format as received from a client / customer interacting with an automated digital assistant to resolve an utterance using a chatbot. The speech converter 412 utilizes the conversion instructions 414 to generate corresponding speech data 416 in a format that can be input to the classifier model. This can be performed when the format of the incoming utterance 406 is not acceptable as an input for the classifier model. In various embodiments, the speech data 416 from the speech converter 412 is sent to an intent classifier 418. The intent classifier 418 may classify, quantify, or otherwise construct intent data 420 corresponding to the incoming utterance 406. In various embodiments, the intent data 420 may correspond to one or more probabilities that a particular intent corresponds to the incoming utterance 406. The intent classifier is an artificial neural network that employs nodes including node parameters or weights that are adjusted during a training sequence, for example, to better output one or more probabilities for future speech data 416 received. For example, a training data set having some "ground truth" data is received at the intent classifier 418, and probabilities generated therefrom are compared to "desired" intents specified by the ground truth data to generate losses that will be used to retrain the machine learning model.

[0103] The intent data 420 is sent from the intent classifier 418 to a conversion model 422. The conversion model may be an entity that exists independently within the data processing system 400 or may be stored and implemented by a separate component, such as the data store 438. The conversion model 422 converts the intent data 420 into skill data 424 that is sent to a skill repository 426. In various embodiments, the skill data 424 may correspond to one or more probabilities that a particular skill or child chatbot will respond to the incoming utterance 406 based on the input intent data 420. In various embodiments, the conversion model 422 performs one or more mathematical transformations on the intent data 420 to create the skill data 424. Once the skill data 424 is transformed by the conversion model 422, it is sent to the skill repository 426. The skill repository 426 is any entity within the data processing system 400 that stores, implements, or otherwise manipulates the skill data 424 to implement the methods described herein. The skill repository 426 includes skill selection instructions 428. The skill selection instructions 426 are any set of instructions or steps that aid in the selection of a skill or chatbot in response to the incoming utterance 406. For example, the skill selection instructions 428 may trigger the selection of a chatbot and / or skill for interacting with the user in response to parsing the skills data 424.

[0104] In various embodiments, the speech converter 412 may send the utterance data 416 directly to the skills repository 426 without first passing the data through the intent classifier 418 and / or the conversion model 422. In various additional embodiments, the skill classifier 430 may take the utterance data 416 and create skill data according to instructions separate from those of the intent classifier 418 and the conversion model 422. The comparison is used to retrain either the intent classifier 418, the conversion model 422, and / or the skill classifier 430. This represents a comparison of predicted skills based on a single-factor modeling technique and predicted skills based on a multi-factor modeling technique. The input of the utterance data 416 to the intent classifier 418 and then the conversion model 422 represents a multi-factor modeling approach to predicting skills for processing an utterance. In contrast, the direct input of the utterance data 416 to the skill classifier 430 represents a single-factor modeling approach to predicting skills for processing an utterance. The difference between the skills predicted using both approaches can be compared to generate a loss that can be used to retrain the skill classifier 430 to better predict skills in the case of single-factor modeling, such as at predicting time when interacting with a client / customer.

[0105] The skill repository 426 includes a chatbot repository 432. The chatbot repository 432 is any repository or storage that includes data or instances of one or more chatbots implemented to handle interactions. The chatbot repository 432 may generally include a parent chatbot 434 that can interact with a user of the data processing system 400. The chatbot repository 432 may further include child chatbots 436A-436N that correspond to skills in the chatbot ecosystem.

[0106] The data processing system 400 may further include a data store 438. The data store 438 is any storage, memory, or medium capable of storing data related to the operation of the data processing system 400. For example, the data store 438 may include a model repository 446 that may store one or more conversion models for converting intent data to skill data. The data store 438 may further include rules 440. The rules 440 are any set of rules, steps, instructions, computer readable code, or other logic that will implement the methods described herein. The data store 438 may further include training instructions 442. The training instructions 442 are any set of steps or instructions that cause training of the conversion model 422 or any other conversion model used to convert the intent data 420 to skill data 424. The data store 438 may further include skillbot information 444. The skillbot information 444 may include any data corresponding to a chatbot, a skill, or an intent. For example, skillbot information 444 may include information about child chatbots 436A-436N, the skills corresponding to those chatbots, and / or the intents associated with those skills and chatbots. Data store 438 is coupled to any of the entities in data processing system 400 discussed above, including those shown in FIG.

[0107] Exemplary Multi-Factor Modeling Embodiments FIG. 5 is a simplified block diagram of a computing system implementing a conversion model, according to an embodiment. Specifically, FIG. 5 illustrates the flow of an incoming dialogue through a data processing system utilizing multi-factor modeling for natural language processing. In some embodiments, the computing system illustrated in FIG. 5 is utilized as part of a prediction process to determine skills to be used to respond to an utterance. For example, the computing system may receive an incoming utterance 406 from a client / customer and select skills to be used to process the incoming utterance 406 and / or generate a response to the incoming utterance 406. In other embodiments, the computing system illustrated in FIG. 5 is utilized as part of a training process to determine skills to be compared to ground truth skills to refine the illustrated multi-factor model. For example, the determined skills output by the computing system of FIG. 5 may be compared to ground truth skills contained in some training data, and the difference between the determined skills and the ground truth skills may be used to retrain components of the computing system. At the same time, the determined intent output by the intent classifier 418 may be compared to ground truth intents contained in any training data, and the differences between the determined intent and the ground truth intent may be used to retrain components of the intent classifier 418.

[0108] As shown in FIG. 5, the illustrated block diagram relates to a flow implemented by a computing device or system to generate skill data 424 from some raw incoming utterance 406. The generated skill data 424 includes information that may be used to select a skill, such as a set of skill scores based on the original incoming utterance 406. For example, the incoming utterance 406 may include a query related to ordering a pizza from a pizzeria chatbot, and the skill data 424 includes several skill "scores" generated as a result of a multi-factor modeling technique by the intent classifier 418 and the transformation model 422.

[0109] In embodiments that include a prediction process, the incoming utterance 406 includes data related to an utterance generated by a client / customer entity to be processed by an appropriate skill to respond to the client / customer. In this case, the goal of the multi-factor modeling process illustrated in FIG. 5 is to utilize the information contained in the incoming utterance 406 to predict a skill for responding to a natural language based query also contained in the incoming utterance 406. The predicted skill may then responsively take some action based on the incoming utterance 406 and / or generate a natural language based outgoing response to respond to the customer / client's query.

[0110] In an embodiment with a training process, the incoming utterance 406 includes data related to the utterance and ground truth skills that will be used to refine the multi-factor model shown in FIG. 5. In this case, the purpose of the multi-factor modeling process shown in FIG. 5 is to attempt to predict a skill corresponding to the incoming utterance that would otherwise be used to process the incoming utterance 406 during a prediction time. The predicted skill may be compared to the ground truth skill to determine a delta between the predicted skill and the ground truth skill (the skill that would be predicted if the multi-factor model operated perfectly). The delta may be used to refine elements of the multi-factor model, such as the intent classifier 418 and / or the transformation model 422. At the same time, the predicted intent from the classifier 418 may be compared to the ground truth intent to determine a delta between the predicted intent and the ground truth intent. The delta may be used to refine elements of the multi-factor model, such as the intent classifier 418.

[0111] In some embodiments, the highest score or highest average score of the skills in the skills data 424 is utilized to select a predicted skill for processing the incoming utterance 406 during the prediction or training process. The incoming utterance 406 is first transformed in the speech converter 412 to create speech data 416 that is input to the intent classifier 418. For example, a natural language query related to ordering a pizza is decomposed into new data including individual words, groupings of words, contextual details of words, and the utterance as a whole. The intent classifier 418 may utilize a model, such as an artificial neural network machine learning model, to transform the utterance data 416 into intent data 420 associated with one or more "scores" for each intent. For example, the generated score or scores may relate to the probability that the incoming utterance 406 is associated with a particular intent from a superset of intents associated with one or more chatbot systems. An example of an intent classifier 418 utilizing an artificial neural network is discussed further below with reference to FIG. 7A.

[0112] The resulting intent data 420 is input to a transformation model 422 that includes a mask 500, which is a transformation model that takes the intent data 420 as an input and produces a set of transformed skill data 424 as an output. In various embodiments, the mask 500 is a mapping data set, such as a matrix, that represents a correspondence between the data sets in the intent data 420 and the resulting skills. Thus, the mask 500 is applied to the intent data 420 to produce skill data 424 that can be interpreted by the skill selection instructions 428 to select a predicted skill based on the multi-factor modeling of the intent classifier 418 and the transformation model 422. In some embodiments, the mask 500 is a dynamically trained transformation value array that maps the intent scores of the intent data 420 to the skill scores of the skill data 424. The skill data 424 is utilized to select a skill to process the incoming utterance 406 according to some instructions / criteria. In some embodiments, the skills data 424 and / or the selected skills are used to retrain various models, such as the speech converter 412, the intent classifier 418, the conversion model 422, etc. (i.e., a training process). In other embodiments, the skills data 424 is used to select the skills that will be used to analyze and respond to the incoming utterance 406 to resolve a customer / client query (i.e., a prediction process).

[0113] More specifically, as shown in FIG. 5, incoming utterances 406 are received from a user and input to a speech converter 412. The incoming utterances 406 are in a raw format and may have one or more variables or characteristics. For example, the incoming utterances 406 may include parameters X corresponding to one or more utterances received from the user (e.g., a query in a natural language-based format), parameters Y(i) corresponding to a number of labels indicating ground truth intents in the superset, and parameters Y(s) corresponding to a number of labels indicating ground truth skills in the superset. For example, parameters X include one or more natural language-based queries (e.g., "I would like to order a pizza"), parameters Y(i) include a set of intents to be considered by the model, such as input classifiers (e.g., pizza ordering intent, order editing intent, etc.), and parameters Y(s) include a set of skills for which one of the skills (e.g., an ordering skill, which may be either ordering a pizza or editing a previous pizza order) is to be predicted based on the multi-factor modeling technique presented in FIG. 5.

[0114] The incoming utterance data includes elements X, Y(i), and Y(s) as parameters for generating predictions of skills as part of the training or prediction process described herein. For example, one or more natural language queries X are the natural language queries included in the incoming utterance 406, the intent superset Y(i) is the set of all intents that may be considered as part of the intent classifier 418, and the skill superset Y(s) is the set of all skills that may be predicted from the skill data 424 output by the multi-factor model. During the training process, X may be all natural language queries for which ground truth skills are known. The parameters Y(i) and Y(s) may be predefined by a training process being performed, and the predicted skill will be at least one of a superset of the skill Y(s); the delta between the ground truth intent and the predicted intent data 420 may be used to update the parameters of the intent classifier 418; the delta between the ground truth skill and the predicted skill data 424 (if different) may be used to update the parameters of the intent classifier 418 or the conversion model 422. During the prediction process, X may be all natural language queries generated by the customer / client, and Y(i) and Y(s) may be selected based on the utterance source (e.g., a customer / client interacting with a chatbot via an automated digital assistant implemented by a pizza restaurant may use only a specific set of intents and skills for customer interactions).

[0115] The incoming utterances 406 are then processed by a speech converter 412 to convert the raw dialogue data into speech data 416 that is recognizable or more easily readable by the multi-factor model shown in Figure 5. For example, one or more natural language queries X may be converted to exclude any data except simple ASCII character data that contains the query. In another example, one or more natural language queries X may be converted to a vectorized format in which each individual word of the query is a value in an input vector.

[0116] The utterance data 416 may then be sent to an intent classifier 418, which will utilize the utterance data 416 as an input and generate intent data 420 as an output. The intent data is denoted as Y'(i), which corresponds to the scores calculated for the set of intent labels. In various embodiments, the intent classifier 418 utilizes a machine learning model, such as an artificial neural network, to take the utterance data 416 and output the intent data 420. In this case, Y'(i) may be a vector of values ​​or scores associated with the output nodes of the artificial neural network. In some embodiments, Y'(i) may be calculated as part of a formula using features of one or more natural language query X. The formula for determining Y'(i) may be as follows:

[0117]

number

[0118] where Y'(i) is the intent data 420 including a set of vectorized scores for each intent in Y(i), "sigmoid()" is a sigmoid function that returns the calculated sigmoid of the feature parameters, SF is a scaling factor that is predetermined, e.g., by hyperparameter tuning or by a data scientist managing the chatbot model, distance() is a function that calculates the relative mathematical distance between two values, W is a transformation matrix learned during model training that indicates the centroid of each intent class, and transform(X) is a transformation of one or more natural language-based queries from the utterance data 416 into a format that can be compared to W.

[0119] Thus, the computed intent data 420, which includes a vectorized dataset of values ​​corresponding to each intent in Y(i), is represented by Y'(i). These values ​​may be used in determining skill data 424 via a transformation model as part of a multi-factor modeling process to determine predicted skills for processing the input utterance 406. In embodiments that implement a prediction process, the output intent data 420 is then input to a transformation model 422 to continue the prediction process.

[0120] In an embodiment implementing a training process, the intent data 420 may be input to a conversion model to continue the skill prediction process described herein. Additionally, when the intent classifier 418 is a machine learning model, a supervised machine learning process may occur. For example, the incoming utterance 406 may further include a ground truth intent that corresponds to a desired intent that the intent classifier selects given the input utterance data 416. A difference between one or more ground truth intents Y(i) and the intent data 420 represented by Y'(i) is calculated, which is referred to as Loss(i), and indicates the degree of error of the intent classifier in predicting the ground truth intent. The vectorized values ​​in Y'(i) may be compared to one or more ground truth intents to determine whether the vectorized values ​​accurately represent the weighted importance of the ground truth intents. For example, the loss of an intent is calculated by the following formula:

[0121]

number

[0122] where Loss(i) is a total loss calculated based on the difference between the values ​​in y'(i) and the ground truth intent y(i), w0, w1, w2, and w3 are weight values ​​predetermined, e.g., by hyperparameter tuning or by a data scientist managing the chatbot model, cross_entropy_loss() is a function that returns the cross entropy loss of the ground truth label y(i) and the logistic distribution of the intent data 418 denoted by logit_y'(i), mse_loss() is a function that returns the mean squared error loss of the values ​​in the ground truth intents y(i) and y'(i), margin_loss() is a function that returns a margin-based loss classification for the values ​​in y'(i), and threshold_loss() is a function that returns a threshold-based loss of the values ​​in the ground truth intents y(i) and y'(i).

[0123] In some embodiments, the loss calculated during training of the intent classifier 418 may be used to retrain the intent classifier. For example, when the intent classifier 418 is a machine learning model, the loss value may be used to modify parameters of operation for the intent classifier 418 in proportion to the calculated loss. If the intent classifier 418 is an artificial neural network, the loss function may be used to modify weighted parameters of nodes in the neural network to refine the model for generation of more accurate intent data 420 during future processing. In some embodiments, the loss calculated for the intent classifier 418 is further utilized as part of a generic loss value, as described below.

[0124] The intent data 420 output from the intent classifier 418 is input to a transformation model 422 to generate skill data 424. The transformation model may include one or more masks 500, which are any data, function, process, or formula that transforms the intent data 420 to generate skill data 424. For example, as shown in FIG. 5, the mask 500 corresponds to a matrix of values ​​that maps one or more intent labels from the intent data 420 to one or more skill labels. For example, given a chatbot system that recognizes seven intents and three skills, a 7×3 mask matrix of values ​​is generated to map the intents to skills. The output of the transformation model 422 utilizing the masks 500 is skill data 424. The skill data is denoted as Y′(s). For example, the formula for determining Y′(s) from Y′(i) is:

[0125]

number

[0126] where Y'(i) is the incoming intent data 420 and M is a mask matrix. In some embodiments, the mask matrix may be converted into a format acceptable for mathematical operations involving the vectorized format of Y'(i). For example, the mask may be inverted on one or more axes to be added to or multiplied with the vectorized format of Y'(i) to compute a vector sum or product of Y'(s). In this manner, the mask matrix may be altered from a standard matrix format to map particular rows or columns of the matrix to rows or columns of Y'(i) to create a vectorized output Y'(s). The output Y'(s) is thus a vector of skill probabilities corresponding to the probability that one or more natural language queries X correspond to any skill in a set of skills.

[0127] The skill corresponding to the vectorized format of Y'(s) may be selected as the most appropriate skill for processing the utterance based on some instructions applied to the skills data 424, such as skill selection instructions 428. For example, the skill selection instructions may result by selecting the corresponding score value in Y'(s) that is the maximum score, minimum score, highest sum score, highest average score, etc. of each vectorized score. Since some skills are associated with multiple intents, there are many ways in which skills are selected or how Y'(s) is generated. In some embodiments, Y'(s) is a vectorized version of the skills data 424 that includes only the highest relative intent value for each corresponding skill. The following formula illustrates this embodiment:

[0128]

number

[0129] where Y'(s) is the vectorized skill data 424, M is a mask matrix, s1-s3 are three scores corresponding to possible skills in Y(s) that may be selected to process the utterance, and y1-y7 are seven scores corresponding to intents in Y(i). As shown by the above equation, the vector of intent scores Y'(i) is multiplied by the mask matrix M to form a three row vector, each corresponding to a skill s1-s3. Each vector row may contain several values, each value representing the likelihood that an intent corresponds to the incoming utterance 406 output by the intent classifier 418, since the mask matrix M contains only values ​​of 0 and 1. From these three vector rows, only the highest value of each vector may be selected and placed into a final vector of values ​​Y'(s) corresponding to the highest value for each skill from the three row vectors.

[0130] As an example, the three row vectors generated by the matrix multiplication of Y'(i) with M may have the highest values ​​in each row as y2, y4, and y6, each corresponding to one of the three skills s1-s3. The resulting vector Y'(s) is [y2, y4, y6]. The vector Y'(s) may then be used to predict the "best" skill based on the values ​​in Y'(s). The predicted skill is the skill that is predicted to be the best skill for processing and responding to the incoming utterance 406. An example of an instruction for selecting the predicted skill may include selecting the largest value in the vector Y'(s). For example, a formula for selecting the best predicted skill for responding to the incoming utterance 406 is given by:

[0131]

number

[0132] where Y(best) is the predicted skill selected based on the highest value selected among Y'(s). It will be appreciated that many different combinations of selections and rules may be applied to predict a skill to respond to an input utterance. In some embodiments, Y'(s) is a vector containing a value of each possible skill in Y(s), whose value is the average of each associated intent value in Y'(i). The predicted skill will correspond to the highest average value of a skill in Y'(s). In some embodiments, Y'(s) is a vector containing a value of each possible skill in Y(s), whose value is the sum of each associated intent value in Y'(i). The predicted skill will correspond to the highest average value of a skill in Y'(s).

[0133] In some embodiments, a machine learning model, such as skill classifier 430, may be utilized to select a skill vector Y'(s) given the input of skills data 424, instead of applying instructions to form Y'(s). An example of applying a machine learning model to form Y'(s) is described below with reference to FIG. 6A.

[0134] Similar to training the intent classifier 418 above, during the training process, the difference between the ground truth skill Y(s) and the predicted skill Y'(s) is referred to as Loss(s) and indicates the degree of error of the multi-factor modeling system in predicting the correct skill. The difference between the ground truth skill Y(s) and the predicted skill Y'(s) associated with the incoming utterance 406 is referred to as Loss(s) and indicates the degree of error of the transformation model in predicting the correct skill to utilize for an action. The skill loss is calculated by the following formula:

[0135]

number

[0136] where y(s) is the ground truth skill and y'(s) is the predicted skill. This formula is similar to the formula for Loss(i) above, except that the difference between the predicted skill and the ground truth skill is applied to determine Loss(s). In various embodiments, Loss(i) may be combined with Loss(s) to form a total loss for skill prediction that accounts for losses in both intent scoring and skill prediction. With the calculated Loss(i) and Loss(s), the total loss of the data processing system is calculated as follows:

[0137]

number

[0138] where lambda is a scalar representing a regularization rate to apply to the loss of the computed intent. According to various embodiments, the data processing system may modify any aspect of the multi-factor model by normalizing the Loss(total) value to the training format and inputting the value into the training model. For example, the Loss(total) calculated during analysis of the training text data is normalized and used to modify the conversion model 422 and / or the mask 500 to better map known intents to known skills.

[0139] The calculated losses, whether Loss(I), Loss(s), Loss(total) or any combination of those losses, may be used to retrain the elements of the multi-factor model shown in Figure 5. In some embodiments where the intent classifier is a machine learning model, the machine learning model includes one or more parameters of an operation for converting the utterance data 416 to the intent data 420. For example, the intent classifier may be an artificial neural network machine learning model that includes one or more nodes and node parameters for converting an input, such as the utterance data 416, to an output, such as the intent data 420. This embodiment is illustrated below with reference to Figure 7A.

[0140] The parameters of the machine learning model utilized as part of the intent classifier 418 may be altered in proportion to the calculated loss as defined above. For example, parameters of particular nodes of the artificial neural network may be reweighted in proportion to the calculated loss. Specifically, to improve the machine learning model, parameters corresponding to particular nodes associated with the ground truth intent / skill may be increased in proportion to the loss, and parameters corresponding to particular nodes associated with the predicted intent / skill may be decreased in proportion to the loss.

[0141] In some embodiments, values ​​of mask 500 may be altered based on the calculated loss. For example, mask 500 may include values ​​between 0 and 1, and those values ​​may be altered based on the calculated loss. For example, values ​​of mask 500 corresponding to predicted skill may be decreased proportionally to the loss, or values ​​of mask 500 corresponding to ground truth skill may be increased proportionally to the loss.

[0142] 6A is a simplified block diagram of a computing system implementing a transformation model, according to an embodiment. Specifically, FIG. 6A illustrates an alternative configuration of the components of the data processing system shown in FIG. 5, including a skill classifier 430 as part of a multi-factor modeling system. The skill classifier 430 may be implemented as both a single-factor model when receiving utterance data 416, and as part of a multi-factor model when receiving skill data 424 from the transformation model 422.

[0143] As mentioned above, single-factor modeling is an efficient way to predict skills to respond to incoming utterances, but does not predict skills with the same accuracy as multi-factor models. As shown in FIG. 6A, utterance data may be used in a single-factor model, i.e., skill classifier 430, to generate prediction data 610 corresponding to predicted skills to process incoming utterances 406. Because the prediction data 610 is generated by the skill classifier 430 using direct input of utterance data 416, skills may be predicted more quickly than the multi-factor modeling shown in FIG. 5. However, the single-factor modeling performed by the skill classifier 430 is not as accurate as the contextual multi-factor modeling performed by the intent classifier 418 and the transformation model 422.

[0144] During the training phase, single-factor modeling and multi-factor modeling processes are performed to retrain the skill classifier 430 to improve the single-factor modeling process. For example, the incoming utterance data 406 may undergo multi-factor modeling as described with reference to FIG. 5. In some embodiments, as shown in FIG. 6A, the mask 600 is a single vector mask instead of a matrix mask that is applied to the intent data 420 to form the skill data 424. For example, each element of the vectorized intent data may be multiplied by a corresponding element in the vector mask 600, and the resulting product value will be inserted into the vectorized skill data represented by Y'(I→S).

[0145] Once the vector of values ​​Y'(I→S) is calculated, the vector is used to calculate a predicted skill as part of a multi-factor modeling process. In some embodiments, a set of rules is applied to the vector Y'(I→S) to determine a predicted skill, similar to the embodiment described with reference to FIG. 5. In some embodiments, the vector Y'(I→S) is input to a skill classifier 430 to output a predicted skill. For example, the skill classifier 430 may include one or more skill prediction models, including one model for converting the input utterance data 416 to predicted data 610 (single-factor modeling) and another model for converting a vector of skill values ​​to predicted data 610 (as part of the multi-factor modeling process). Although the input utterance data 416 and the skill data 424 are different types of inputs, the output of the predicted skill as part of the prediction data 610 is the same, and thus the output of the single-factor modeling process may be compared to the output of the multi-factor modeling process.

[0146] As described above, the skills predicted by single-factor and multi-factor modeling of the same incoming utterance 406 can be used to refine the single-factor model to increase accuracy while retaining the speed of single-factor prediction. The predicted skills can be compared to determine the difference between the predicted skills for each separate process. This difference is used to identify a loss function or difference that is used to retrain the single-factor model, in this case the skill classifier 430. In various embodiments, a cost function may be used when multiple iterations of training are applied to the system described with reference to Figures 5, 6A, and 6B. For example, multiple loss functions may be calculated for each iteration of training and used to determine the cost function. The loss function / cost function may be used to retrain the model.

[0147] As described herein, the predicted data 610 generated by the single-factor modeling process will be more accurate when compared to the derived predicted data 610 generated through single-factor modeling when the utterance data 416 is directly input to the skill classifier, due to the use of more contextually accurate intent data and trainable transformation models. However, single-factor modeling is more time-efficient in that skills can be predicted without the various steps of multi-factor modeling. Therefore, it is useful to train a single-factor model such as the skill classifier 430 using the difference in output between multi-factor modeling and single-factor modeling to improve the single-factor modeling capabilities of the chatbot system.

[0148] As an example, an utterance is received: "I ordered a pizza and it wasn't cooked so it was terrible." The chatbot utilizes two skills: a pizza ordering skill and a customer service skill. A poorly trained single-factor model, such as a skill classifier, might simply select the pizza ordering skill based on the words "ordered" and "pizza" in the utterance, where customer service is the correct skill to process the utterance. A finer context analysis using multi-factor modeling would be more likely to select the correct skill. For example, the customer service skill may include an intent for unsatisfied customer comments. This intent is highly correlated with the utterance by the words "ordered" and "pizza," as well as "wasn't cooked" and "was terrible." This intent may also determine context specific to the utterance, such as the fact that the utterance is written in the past tense, possibly addressing customer service. In this example, the multi-factor modeling process provides the correct predicted skill relative to the single-factor modeling process. The difference between the skill predictions may be used to retrain the skill classifier, for example, decreasing a parameter in the skill classifier 430 that weights the correlation between the words "order" and "pizza" and pizza ordering skill, and increasing a parameter that weights the correlation between the words "wasn't" and "terrible" and customer service skill.

[0149] Thus, when the input utterance data 416 is part of the single-factor modeling process, the resulting losses / differences between predicted skills are used to retrain the skill classifier 430 to provide more contextually accurate prediction data 610. Thus, the block diagram shown in FIG. 6A illustrates a hybrid contextual model that may efficiently train and utilize the skill classifier 430 during the prediction phase. As mentioned above, the prediction phase (or “deployment phase”) is the phase in which a chatbot including the modeling techniques described herein is used to interact with real clients / customers and resolve real natural language queries sent to the chatbot system as part of communications with an automated digital assistant utilized by the customers / clients. During this phase, the model is effectively “deployed” to process and respond to the needs of the customers / clients. While the training phase may utilize predictions to retrain the model, the predictions used during the prediction phase will be used to respond to real-world customers / clients. The client experience is improved both when queries are resolved quickly and accurately. Thus, a contextually well-trained single-factor model such as the skill classifier 430 is a useful asset for a chatbot during the prediction phase. In some cases where the query issued by the customer / client is very complex or the single-factor model is not sufficiently trained, the multi-factor model may be utilized to predict the skill to process the request and handle the query to increase contextual accuracy at the expense of time-based efficiency. For example, a very complex natural language query may be received and the speech converter may determine, during conversion to speech data 416, that the language in the utterance may be too complex for the skill classifier to provide an accurate prediction. In response, the speech converter may send the utterance data 416 to the intent classifier 418 to begin the multi-factor modeling process.

[0150] FIG. 6B is a simplified block diagram of a computing system implementing a conversion model, according to an embodiment. As shown in FIG. 6B, mask 620 is a more complex mask than mask 610 in which skills are predicted from skill data 424 after providing a more contextually accurate set of skill data 424. As shown in FIG. 6B, the mask may include values ​​that are not Boolean, i.e., 0 and 1. For example, mask 620 may include values ​​that correspond to training modification values, such as multiples or floating point numbers, that are frequently converted according to training inputs according to the methods described herein. These values ​​of mask 620 may be more fine-tuned to affect a more accurate conversion of intent data 420 to skill data 424. As described above, mask 620 includes values ​​that map intent scores in intent data 420 to corresponding skills in the set of skills. In some cases, intents may not carry the same weight as other intents when determining predicted skills. For example, the "promotion offer" intent may be used much less than the "pizza order" intent. As a result, within mask 620, the values ​​representing the correspondence between promotional offer intents and skills may be much lower than the values ​​representing the correspondence between pizza ordering intents and skills.

[0151] 7A is a simplified block diagram of an intent classifier configured as an artificial neural network, according to some embodiments. Specifically, FIG. 7A illustrates an exemplary embodiment of an intent classifier 418 configured to utilize an artificial neural network to take an utterance vector as input and output one or more intent scores. As described herein, the utterance vector 700 is a form of utterance data 416 derived from the speech converter 412, and the intent scores 740(A)-740(N) are a form of intent data 420 or context data set output by the intent classifier 418.

[0152] As shown in FIG. 7A, the utterance vector 700 includes a set of one or more vector data 701(A)-701(N). The vector data may include any data derived from a speech / natural language based query and in a format acceptable as input to the intent classifier 418. The vector data 701 may include, for example, individual words, groupings of words, context, font, punctuation, environment, customer, or any other data related to the utterance received from the customer / client. The utterance vector 700 is mapped to a set of input nodes 710(A)-710(N) in the artificial neural network of the intent classifier 418. Each of these input nodes 710 is a node that includes node parameters / weights that determine the relative weight that each input is given in the initial stages of the artificial neural network.

[0153] The input nodes 710(A)-710(N) are mapped to additional layers of the artificial neural network via one or more edges. For example, some edges emanating from the input nodes 710(A)-710(N) lead to additional nodes in a hidden layer, each of which may have corresponding node parameter / weight values. For example, as shown in FIG. 7A, the input nodes 710(A)-710(N) are mapped to a hidden layer 720(A) of the artificial neural network. It will be understood that an artificial neural network is a network that includes one-to-one edge relationships of nodes in each successive layer or any subset of edges therein.

[0154] Values ​​passed from input nodes 710(A)-710(N) are passed successively through multiple hidden layers 720(A)-720(N) to progress through the neural network. It will be appreciated that node parameters / weights, and possibly edge weights between nodes, are modified according to one or more machine learning training methods. Once the hidden layer 720(N) processes the data from the previous hidden layer, the data is passed to one or more output nodes 730(A)-730(N). The output nodes 730(A)-730(N) may correspond to one or more final values ​​of possible context subclasses, such as intents for a chatbot, generated as a result of the artificial neural network. Thus, the artificial neural network of the intent classifier 418 may generate one or more probability scores related to the probability that the input utterance vector 700 corresponds to a particular intent for processing and responding to the utterance. These probabilities are represented by intent scores 740(A)-740(N) that are derived from output nodes 730(A)-730(N). The use of intent scores to select skills is explored further below with respect to FIG.

[0155] 7B is a simplified block diagram of a skill classifier configured as an artificial neural network according to some embodiments. Specifically, FIG. 7B illustrates an example embodiment of a skill classifier 430 configured to take utterance vectors as input and output predicted skills as part of single-factor modeling using an artificial neural network.

[0156] FIG. 7B is a simplified block diagram of a skill classifier 430 configured as an artificial neural network, according to some embodiments. Specifically, FIG. 7B illustrates an example embodiment of a skill classifier 430 configured to utilize an artificial neural network to take an utterance vector as input and output a predicted skill for processing an utterance. For example, the skill classifier 430 may take an utterance vector 700 similar to the way the intent classifier 418 takes an utterance vector. The skill classifier 430 may also include a number of input nodes 750(A)-750(N) and a number of hidden layers 760(A)-760(N) that proceed through the neural network. It will be appreciated that the node parameters / weights, and possibly the weights of the edges between the nodes, are modified according to one or more machine learning training methods. The final hidden layer 760(N) may then send the values ​​contained therein to a score selector 770. The score selector 770 may utilize one or more score selection criteria and values ​​received from the hidden layer 760(N) to determine a predicted skill 780 to be output by the skill classifier. For example, the score selector 770 may receive one or more score values ​​for several possible skills from which a predicted skill 780 may be selected and may utilize criteria that specify that the highest score is selected to select the predicted skill 780.

[0157] FIG. 8 illustrates a diagram showing a process flow for determining a context class using a conversion model to respond to an utterance, according to various embodiments. The process illustrated in FIG. 8 is implemented in software (e.g., code, instructions, programs) executed by one or more processing units (e.g., processors, cores) of the respective system, hardware, or a combination thereof. The software is stored on a non-transitory storage medium (e.g., on a memory device). The method presented in FIG. 8 and described below is intended to be exemplary and non-limiting. FIG. 8 illustrates various processing steps occurring in a particular sequence or order, but this is not intended to be limiting. In certain alternative embodiments, the steps may be performed in some different order or some steps may be performed in parallel. In various embodiments, the steps detailed in process 800 are performed by the systems and models discussed herein with respect to FIGS. 1-7B.

[0158] The process 800 begins at 810 when a set of utterance data is received that includes one or more natural language queries. For example, the utterance data is received as part of an incoming utterance 406 and is received at a communication portal 402 of the data processing system 400. After receiving the incoming utterance 406, the data processing system processes the data and prepares the utterance for input to one or more context models to predict skills to process and resolve the utterance. In some embodiments, the utterance data may indicate that the utterance is being processed as part of a training phase for a chatbot model and / or a classifier model to refine the model for a future prediction phase. In some embodiments, the utterance data may indicate that the utterance is from an automated digital assistant being utilized by a customer or client and therefore should be used to provide a prediction of a skill corresponding to the utterance to respond to the customer / client query during the prediction phase. In various embodiments, in response to receiving the utterance, the utterance is converted by a data converter, such as speech converter 412, to convert the data into a speech vector, such as speech vector 700, for input to a classifier, such as intent classifier 418.

[0159] At 820, one or more intents for the utterance data are determined. The one or more intents are possible intents that may be used to process and generate a response to the utterance or that are associated with predictable skills that will be used to process and generate a response to the utterance. In some embodiments, the received utterance is sent to or has metadata associated with a particular portion of the data processing system 400 to determine a particular chatbot or area of ​​interest that indicates a set of skills / intents for resolving the utterance. The metadata may be used to determine the intent of the utterance data. For example, the utterance may have been received from an automated digital assistant that operates as part of a portal for a pizza delivery company's website, thus indicating that a chatbot trained to handle pizza orders and related inquiries should be used to process the utterance. The chatbot is associated with a set of skills and intents, which will be determined once the chatbot is selected. In some embodiments, at step 820, a superset of all possible intents known to the data processing system 400 is selected.

[0160] At 830, one or more intent classification data sets associated with probabilities that the utterance data corresponds to one or more intents are generated using the intent classifier model. Specifically, the set of intents determined at 820 is used to select a particular machine learning model (such as the artificial neural network shown in FIG. 7A) that corresponds to the determined intents. For example, the set of intents determined at 820 would all correspond to pizza delivery utterances and would correspond to a chatbot that utilizes the intent classifier model as part of a multi-factor model for resolving pizza delivery based queries.

[0161] In an embodiment implementing an artificial neural network, a particular intent classifier model includes nodes that receive as input vectorized speech data derived from an utterance that will be mapped to a set of input nodes, such as input nodes 710(A)-710(N), to begin the process of generating one or more output intent classification datasets. The intent classification dataset output from the intent classifier is a set of "scores," such as intent scores 740(A)-740(N), that correspond to a set of probabilities that the utterance corresponds to a particular one of the one or more intents determined in 820. This is similar to the input of utterance data 416 to intent classifier 418 to output intent data 420 as shown in FIG. 4. Thus, the output context dataset derived from the intent classifier model indicates the likelihood that a particular intent most closely corresponds to the utterance to be resolved.

[0162] At 840, one or more transformed data sets corresponding to one or more skills are generated using the mask model. Specifically, the mask model is applied to the output intent classification dataset generated at 830 to generate one or more transformed data sets based on the data in the mask model. This is similar to the input of intent data 420 to transformation model 422 to output skill data as shown in FIG. 4. The mask model may be, for example, a vector that can be mathematically applied to a vectorized version of the intent classification dataset. The mathematical application of such a mask matrix to the vectorized intent classification data will result in a vectorized transformed dataset. The generated transformed dataset may correspond to a relative score or metric associated with one or more skill classes corresponding to the intent determined at 820. For example, the one or more transformed datasets may include an integrated score for each skill based on a transformation of the intent classification dataset that will be used to predict a skill to respond to an utterance using a skill selection instruction, such as skill selection instruction 430. The applied mask may include a map value for mapping a context value corresponding to an intent to an associated transformed value corresponding to a skill, such as mask 500.

[0163] At 850, skills for processing the set of utterance data are determined based on the one or more transformed data sets. The determined skills are used to process and respond to the utterance data received at 810. For example, the transformed data set, similar to the skills data 424, is processed according to a set of skill selection instructions 428 or by the skill classifier 430 to determine predicted skills for processing the utterance data. In some embodiments, selection criteria corresponding to value comparisons are performed according to the skill selection instructions 428. In some embodiments, a machine learning model, such as a model operating as part of the skill classifier 430, may take as input the transformed data set and output a set of skill scores that will be used to select predicted skills for processing and responding to the utterance data. An example of the selection of contextual skill classes is provided below with reference to FIG. 9.

[0164] At 860, the determined skills are used to process the set of utterance data to resolve one or more natural language queries. For example, the skills determined at 850 are used to process and generate responses based on processing of the utterance data and / or perform response actions to resolve the natural language queries.

[0165] In various embodiments, the process 800 is performed as part of a training phase of the chatbot model. For example, a loss function or value is calculated during the training phase according to the methods described herein. In various embodiments, a total loss is calculated based on the calculated skill label loss and intent label loss. In various additional embodiments, the total loss is normalized and input into a conversion model to modify the conversion model. In various additional embodiments, the query text data is processed and transformed by the modified conversion model. In various embodiments, the skill class determinations before and after the conversion model is modified are further compared and / or used to further train the conversion model.

[0166] In various embodiments, the process 800 is performed as part of the prediction stage of the chatbot model. For example, the skill classifier may not be properly trained using the methods described herein by the time the chatbot is made available to customers for prediction operations. In this case, the utterance data received at prediction time may be first processed by an intent classifier to increase the accurate prediction of skills to respond to natural language-based queries, and a transformation mask is applied before the skill is selected. In some embodiments, if a contextual skill class is selected to process the utterance using direct input of the utterance data, and the relative value from the derived transformed data set does not meet or exceed the confidence threshold, the utterance is reprocessed at prediction time using the process 800 including the intent classifier and mask transformation.

[0167] 9 is a simplified block diagram of an example selection of a skill bot by an intent classification technique according to some embodiments. Specifically, FIG. 9 illustrates a flow for selecting a skill to process a natural language-based query based on an output score of an intent subclass, such as one generated according to the embodiment shown in FIG. 7A and according to the process described in FIG. 8.

[0168] The intents 900(A)-900(G) may correspond to a context dataset that represents a derived probability that an intent corresponds to an utterance, such as the output intent scores 740(A)-740(N) shown in FIG. 7A. Each intent corresponds to a different capability or processing context. For example, as shown in FIG. 9, each intent 900(A)-900(G) corresponds to a different intent task, such as ordering a pizza, customer reviews of a pizza place, etc. Each intent 900 corresponds to an intent score, such as shown in the combined set of intent scores 910. The intent scores are part of a context dataset that represents the probability that an utterance corresponds to an intent. For example, the utterance being processed may be, "I ordered a pizza and it never showed up."

[0169] The intent scores 910 are transformed according to the masks 920 to generate a transformed data set that will be used to select a skill to process the utterance data. For example, the intent scores 920 are shown in a matrix format and the masks are shown in a matrix format that are multiplied with the intent scores 910 to generate a corresponding skill score. The manner in which the masks are applied and the manner in which the skills are correspondingly selected are determined by a score selector 930 subsystem that is part of the skill selection instructions 428. For example, the score selector 930 may include instructions that the intent scores 910 should be multiplied by the masks 920 to generate a skill score 940 for each skill. The score selector 930 may further indicate that the highest skill score 940 among the skill scores 940 should be used to select a skill 950 to process the utterance.

[0170] Skills 950(A)-950(C) are skills that are selected according to some score selector 930 criteria based on skill score 940. Each of the skills 950 is associated with a subset of intents 900(A)-900(G). As shown in FIG. 9 , the highest score among the skill scores 940 corresponds to "feedback skill" 950(B), which contains the intent for "customer service" 900(E) and is the appropriate skill to process and respond to the utterance "I ordered a pizza and it never showed up."

[0171] Exemplary System 10 shows a simplified diagram of a distributed system 1000. In the illustrated example, the distributed system 1000 includes one or more client computing devices 1002, 1004, 1006, and 1008 coupled to a server 1012 via one or more communication networks 1010. The client computing devices 1002, 1004, 1006, and 1008 are configured to execute one or more applications.

[0172] In various examples, the server 1012 is adapted to execute one or more services or software applications that enable one or more embodiments described in this disclosure. In some examples, the server 1012 may also provide other services or software applications, which may include non-virtual and virtual environments. In some examples, these services are provided as web-based or cloud services, such as under a Software as a Service (SaaS) model, to users of the client computing devices 1002, 1004, 1006 and / or 1008. Users operating the client computing devices 1002, 1004, 1006 and / or 1008 may utilize the services provided by these components by interacting with the server 1012 utilizing one or more client applications.

[0173] In the configuration shown in Figure 10, the server 1012 may include one or more components 1018, 1020, and 1022 that implement the functions performed by the server 1012. These components may include software components that may be executed by one or more processors, hardware components, or a combination thereof. It should be appreciated that a wide variety of system configurations different from the distributed system 1000 are possible. Thus, the example shown in Figure 10 is one example of a distributed system for implementing the example system and is not intended to be limiting.

[0174] A user may use a client computing device 1002, 1004, 1006, and / or 1008 to execute one or more applications, models, or chatbots, which may generate one or more events or models, which may then be implemented or processed according to the teachings of this disclosure. A client device may provide an interface that allows a user of the client device to interact with the client device. The client device may also output information to the user via this interface. Although FIG. 10 shows only four client computing devices, any number of client computing devices is supported.

[0175] Client devices may include various types of computing systems, such as portable handheld devices, general purpose computers such as personal computers and laptops, workstation computers, wearable devices, gaming systems, thin clients, various messaging devices, sensors or other sensing devices, etc. These computing devices may include various types and versions of software applications and operating systems (e.g., Microsoft Windows®, Apple Macintosh®, UNIX® or UNIX-like operating systems, Linux® or Linux-like operating systems, e.g., Google Chrome® OS, including various mobile operating systems (e.g., Microsoft Windows Mobile®, iOS®, Windows Phone®, Android®, BlackBerry®, Palm OS®). Portable handheld devices may include cellular phones, smartphones (e.g., iPhone®), tablets (e.g., iPad®), personal digital assistants (PDAs), etc. Wearable devices may include Google Glass® head-mounted displays and other devices. The gaming systems may include various handheld gaming devices, Internet-enabled gaming devices (e.g., Microsoft Xbox® gaming consoles with or without Kinect® gesture input devices, Sony PlayStation® systems, various gaming systems offered by Nintendo®, etc.) The client devices may be capable of running a wide variety of applications, such as various Internet-related applications, communication applications (e.g., email applications, short message service (SMS) applications), and may use a variety of communication protocols.

[0176] Network 1010 may be any type of network known to those skilled in the art that is capable of supporting data communications using any of a variety of available protocols, including, but not limited to, TCP / IP (Transmission Control Protocol / Internet Protocol), SNA (Systems Network Architecture), IPX (Internet Packet Exchange), AppleTalk, etc. By way of example only, network 1010 may be a local area network (LAN), an Ethernet-based network, a token ring, a wide area network (WAN), the Internet, a virtual network, a virtual private network (VPN), an intranet, an extranet, a public switched telephone network (PSTN), an infrared network, a wireless network (e.g., a wireless network operating under any of the Institute of Electrical and Electronics Engineers (IEEE) 1002.11 protocol suite, Bluetooth, and / or any other wireless protocol), and / or any combination of these and / or other networks.

[0177] The servers 1012 may be comprised of one or more general purpose computers, dedicated server computers (including, by way of example, PC (personal computer) servers, UNIX servers, mid-range servers, mainframe computers, rack-mounted servers, etc.), server farms, server clusters, or other suitable configurations and / or combinations. The servers 1012 may include one or more virtual machines running a virtual operating system, or other computing architectures involving virtualization, such as one or more flexible pools of logical storage that are virtualized to maintain virtual storage for the servers. In various examples, the servers 1012 are adapted to run one or more services or software applications that provide the functionality described in the disclosure above.

[0178] The computing systems in the servers 1012 may run one or more operating systems, including any of the operating systems listed above, as well as commercially available server operating systems. The servers 1012 may also run any of a variety of other server and / or mid-tier applications, including an HTTP (Hypertext Transfer Protocol) server, an FTP (File Transfer Protocol) server, a CGI (Common Gateway Interface) server, a JAVA® server, a database server, and the like. Exemplary database servers include, but are not limited to, those commercially available from Oracle®, Microsoft®, Sybase®, IBM® (International Business Machines), and the like.

[0179] In some implementations, the server 1012 may include one or more applications for parsing and consolidating data feeds and / or event updates received from users of the client computing devices 1002, 1004, 1006, and 1008. By way of example, the data feeds and / or event updates may include, but are not limited to, Twitter® feeds, Facebook® updates, or real-time updates received from one or more third party sources and continuous data streams that may include real-time events related to sensor data applications, financial stock ticker boards, network performance measurement tools (e.g., network monitoring and traffic management applications), clickstream analysis tools, automobile traffic monitoring, etc. The server 1012 may also include one or more applications for displaying the data feeds and / or real-time events via one or more display devices of the client computing devices 1002, 1004, 1006, and 1008.

[0180] The distributed system 1000 may also include one or more data repositories 1014, 1016. In certain examples, these data repositories are used to store data and other information. For example, one or more of the data repositories 1014, 1016 are used to store information such as information related to chatbot performance or generated models for use by the chatbot that are used by the server 1012 in performing various functions according to various embodiments. The data repositories 1014, 1016 may be in a variety of locations. For example, the data repositories used by the server 1012 are local to the server 1012 or are remote from the server 1012 and communicate with the server 1012 via a network-based or dedicated connection. The data repositories 1014, 1016 are of different types. In certain examples, the data repositories used by the server 1012 are databases, for example, relational databases such as those provided by Oracle Corporation® and other manufacturers. One or more of these databases are adapted to allow data to be stored, updated, and retrieved from the database in response to SQL-formatted commands.

[0181] In particular examples, one or more of the data repositories 1014, 1016 may be used by an application to store application data. The data repositories used by the application may be of various types, such as, for example, a key-value store repository, an object store repository, or a general-purpose storage repository supported by a file system.

[0182] In certain examples, the functionality described in this disclosure may be provided as a service via a cloud environment. FIG. 11 is a simplified block diagram of a cloud-based system environment that provides various services as cloud services, according to certain examples. In the example shown in FIG. 11, a cloud infrastructure system 1102 may provide one or more cloud services that users may request using one or more client computing devices 1104, 1106, and 1108. The cloud infrastructure system 1102 may include one or more computers and / or servers, which may include those previously described with respect to server 612. The computers in the cloud infrastructure system 1102 may be organized as general purpose computers, dedicated server computers, server farms, server clusters, or any other suitable arrangement and / or combination.

[0183] The network 1110 may facilitate communication and exchange of data between the clients 1104, 1106, and 1108 and the cloud infrastructure system 1102. The network 1110 may include one or more networks. The networks may be of the same type or of different types. The network 1110 may support one or more communication protocols, including wired and / or wireless protocols, to facilitate communication.

[0184] The example shown in Figure 11 is merely one example of a cloud infrastructure system and is not intended to be limiting. It should be understood that in other examples, cloud infrastructure system 1102 may have more or fewer components than those shown in Figure 11, may combine two or more components, or may have components in a different configuration or arrangement. For example, while Figure 11 shows three client computing devices, in alternative examples, any number of client computing devices are supported.

[0185] The term cloud services is generally used to refer to services made available to users on demand via a communication network such as the Internet by a service provider's system (e.g., cloud infrastructure system 1102). Typically, in a public cloud environment, the servers and systems that make up the cloud service provider's system are different from the customer's own on-premise servers and systems. The cloud service provider's systems are managed by the cloud service provider. Thus, customers can use cloud services provided by the cloud service provider without having to purchase separate licenses, support, or hardware and software resources for the services. For example, the cloud service provider's system may host applications, and users may order and use the applications on demand via the Internet without having to purchase infrastructure resources to run the applications. Cloud services are designed to provide easy and scalable access to applications, resources, and services. Several providers offer cloud services. For example, several cloud services, such as middleware services, database services, and Java cloud services, are offered by Oracle Corporation of Redwood Shores, California.

[0186] In particular examples, cloud infrastructure system 1102 may provide one or more cloud services using a variety of models, such as a Software as a Service (SaaS) model, a Platform as a Service (PaaS) model, an Infrastructure as a Service (IaaS) model, etc., including a hybrid service model. Cloud infrastructure system 1102 may include a suite of applications, middleware, databases, and other resources that enable the provisioning of various cloud services.

[0187] The SaaS model allows applications or software to be delivered as a service to customers over a communications network such as the Internet without the customer having to purchase the hardware or software for the underlying application. For example, the SaaS model may be used to allow customers to access on-demand applications hosted by the cloud infrastructure system 1102. Examples of SaaS services offered by Oracle Corporation include, but are not limited to, various services for human resource / capital management, customer relationship management (CRM), enterprise resource planning (ERP), supply chain management (SCM), enterprise performance management (EPM), analytics services, social applications, and the like.

[0188] The IaaS model is commonly used to provide flexible computing and storage capabilities by providing infrastructure resources (e.g., servers, storage, hardware and networking resources) to customers as cloud services. Various IaaS services are offered by Oracle Corporation.

[0189] The PaaS model is generally used to provide platform and environment resources as a service that enable customers to develop, run, and manage applications and services without having to procure, build, or manage the environment resources. Examples of PaaS services provided by Oracle Corporation (registered trademark) include, but are not limited to, Oracle Java Cloud Service (JCS), Oracle Database Cloud Service (DBCS), data management cloud services, various application development solution services, etc.

[0190] Cloud services are generally provided on an on-demand self-service basis, on a subscription basis, and in a flexible, scalable, reliable, highly available, and secure manner. For example, a customer may order one or more services provided by cloud infrastructure system 1102 via a subscription order. Cloud infrastructure system 1102 then performs processing to provide the services requested in the customer's subscription order. For example, a user may use an utterance to request the cloud infrastructure system to take a particular action (e.g., intent) as described above and / or provide a service for a chatbot system as described herein. Cloud infrastructure system 1102 is configured to provide one cloud service or even multiple cloud services.

[0191] Cloud infrastructure system 1102 may provide cloud services through a variety of deployment models. In a public cloud model, cloud infrastructure system 1102 may be owned by a third-party cloud service provider and cloud services are provided to general public customers. The customers may be individuals or businesses. In another example, under a private cloud model, cloud infrastructure system 1102 operates within an organization (e.g., within a corporate organization) and services are provided to customers within the organization. For example, the customers may be various departments of a company, such as human resources, payroll, etc., and may be individuals within the company. In another example, under a community cloud model, cloud infrastructure system 1102 and the services provided are shared among various organizations within an associated community. Various other models may be used, including hybrids of the above models.

[0192] Client computing devices 1104, 1106, and 1108 may be of different types (e.g., client computing devices 1002, 1004, 1006, and 1008 shown in FIG. 10) and are capable of operating one or more client applications. Users may use the client devices to interact with cloud infrastructure system 1102, such as to request services provided by cloud infrastructure system 1102. For example, a user may use a client device to request information or actions from a chatbot, as described in this disclosure.

[0193] In some examples, the processing that the cloud infrastructure system 1102 performs to provide the service may include model training and deployment. This analysis may include using, analyzing, and processing a data set to train and deploy one or more models. This analysis is performed by one or more processors, possibly processing the data in parallel, running simulations with the data, etc. For example, big data analysis is performed by the cloud infrastructure system 1102 to generate and train one or more models for a chatbot system. The data used in this analysis may include structured data (e.g., data stored in a database or structured according to a structured model) and / or unstructured data (e.g., data blobs (binary large objects)).

[0194] 11, cloud infrastructure system 1102 may include infrastructure resources 1130 that are utilized to facilitate the provision of various cloud services offered by cloud infrastructure system 1102. Infrastructure resources 1130 may include, for example, processing resources, storage or memory resources, networking resources, etc. In a particular example, storage virtual machines available to handle storage requested by applications are part of cloud infrastructure system 1102. In other examples, the storage virtual machines are part of a different system.

[0195] In certain examples, to facilitate efficient provisioning of these resources to support various cloud services offered by the cloud infrastructure system 1102 for different customers, the resources are organized into resource sets or resource modules (also referred to as "pods"). Each resource module or pod may include a pre-integrated and optimized combination of one or more types of resources. In certain examples, different pods are pre-provisioned for different types of cloud services. For example, a first set of pods is provisioned for database services, a second set of pods is provisioned for Java services, etc., which may include a different combination of resources than the pods in the first set of pods. For some services, the resources allocated to provision these services are shared between the services.

[0196] The cloud infrastructure system 1102 itself may use services 1132 internally that are shared by different components of the cloud infrastructure system 1102 and that facilitate the provisioning of services by the cloud infrastructure system 1102. These internal shared services may include, but are not limited to, security and identity services, integration services, enterprise repository services, enterprise manager services, virus scanning and whitelist services, high availability, backup and recovery services, services enabling cloud support, email services, notification services, file transfer services, etc.

[0197] Cloud infrastructure system 1102 may include multiple subsystems. These subsystems may be implemented in software, or hardware, or a combination thereof. As shown in FIG. 11, the subsystems may include a user interface subsystem 1112 that allows users or customers of cloud infrastructure system 1102 to interact with cloud infrastructure system 1102. User interface subsystem 1112 may include a variety of different interfaces, such as a web interface 1114, an online store interface 1116 through which cloud services offered by cloud infrastructure system 1102 are advertised and available for purchase by consumers, and other interfaces 1118. For example, a customer may use a client device to request (service request 1134) one or more services offered by cloud infrastructure system 1102 using one or more of interfaces 1114, 1116, and 1118. For example, a customer may access an online store, browse cloud services offered by cloud infrastructure system 1102, and place a subscription order for one or more services offered by cloud infrastructure system 1102 and for which the customer wishes to subscribe. The service request may include information identifying a customer and one or more services for which the customer wishes to subscribe. For example, a customer may submit an order to subscribe to services provided by cloud infrastructure system 1102. As part of the order, the customer may provide information identifying the chatbot system for which the service is to be provided, and optionally one or more credentials for the chatbot system.

[0198] 11, cloud infrastructure system 1102 may include an order management subsystem (OMS) 1120 configured to process new orders. As part of this process, OMS 1120 is configured to create an account for the customer if not already created, receive billing and / or account information from the customer that will be used to charge the customer for providing the requested services to the customer, verify the customer information, and once verified, reserve the order for the customer, and prepare the order for provisioning by coordinating various workflows.

[0199] Upon proper validation, the OMS 1120 may invoke an order provisioning subsystem (OPS) 1124 configured to provision resources for the order, including processing, memory, and networking resources. Provisioning may include allocating resources for the order and configuring the resources to facilitate the service requested by the customer order. The manner in which resources are provisioned for the order and the type of resources provisioned may depend on the type of cloud service ordered by the customer. For example, following one workflow, the OPS 1124 may be configured to determine the specific cloud service being requested and identify the number of pods that may be pre-configured for the specific cloud service. The number of pods allocated for an order may depend on the size / amount / level / scope of the service requested. For example, the number of pods to allocate may be determined based on the number of users the service should support, the duration for which the service is requested, etc. The allocated pods may then be customized to the specific requesting customer to provide the requested service.

[0200] In a particular example, the setup phase processing is performed as part of the provisioning process, as described above, by cloud infrastructure system 1102. Cloud infrastructure system 1102 can generate an application ID and select a storage virtual machine for the application from among the storage virtual machines provided by cloud infrastructure system 1102 itself or from storage virtual machines provided by other systems other than cloud infrastructure system 1102.

[0201] Cloud infrastructure system 1102 may send a response or notification 1144 to the requesting customer to indicate when the requested service will be available for use. In some examples, information (e.g., a link) is sent to the customer that enables the customer to begin using and utilizing the benefits of the requested service. In a particular example, for a customer requesting a service, the response may include a chatbot system ID generated by cloud infrastructure system 1102 and information identifying the chatbot system selected by cloud infrastructure system 1102 for the chatbot system corresponding to the chatbot system ID.

[0202] Cloud infrastructure system 1102 may provide services to multiple customers. For each customer, cloud infrastructure system 1102 is responsible for managing information related to one or more subscription orders received from the customer, maintaining customer data related to the orders, and providing the requested services to the customer. Cloud infrastructure system 1102 may also collect usage statistics regarding the customer's use of the subscribed services. For example, statistics may be collected on the amount of storage used, the amount of data transferred, the number of users, and the amount of system uptime and system downtime. This usage information is used to bill the customer. Billing may occur, for example, on a monthly basis.

[0203] Cloud infrastructure system 1102 may provide services to multiple customers in parallel. Cloud infrastructure system 1102 may store information about these customers, possibly including copyright information. In a particular example, cloud infrastructure system 1102 includes an identity management subsystem (IMS) 1128 configured to manage information about customers and separate the managed information so that information about one customer is not accessed from information about another customer. IMS 1128 is configured to provide various security-related services, such as identity services such as information access management, authentication and authorization services, services for managing customer identities and roles and associated capabilities, and the like.

[0204] FIG. 12 illustrates an example of a computer system 1200. In some examples, the computer system 1200 is used to realize any of the digital assistant or chatbot systems in a distributed environment, as well as the various servers and computer systems described above. As shown in FIG. 12, the computer system 1200 includes various subsystems, including a processing subsystem 1204 that communicates with several other subsystems via a bus subsystem 1202. These other subsystems may include a processing acceleration unit 1206, an I / O subsystem 1208, a storage subsystem 1218, and a communication subsystem 1224. The storage subsystem 1218 may include a non-transitory computer-readable storage medium, including a storage medium 1222 and a system memory 1210.

[0205] Bus subsystem 1202 provides a mechanism for allowing the various components and subsystems of computer system 1200 to communicate with each other as intended. Although bus subsystem 1202 is shown diagrammatically as a single bus, alternative embodiments of the bus subsystem may utilize multiple buses. Bus subsystem 1202 may be any of several types of bus structures including a memory bus or memory controller, a peripheral bus, a local bus, etc., using any of a variety of bus architectures. For example, such architectures may include an Industry Standard Architecture (ISA) bus, a Micro Channel Architecture (MCA) bus, an Enhanced ISA (EISA) bus, a Video Electronics Standards Association (VESA) local bus, a Peripheral Component Interconnect (PCI) bus, which may be implemented as a mezzanine bus manufactured in accordance with the IEEE P1386.1 standard, and the like.

[0206] The processing subsystem 1204 controls the operation of the computer system 1200 and may include one or more processors, application specific integrated circuits (ASICs), or field programmable gate arrays (FPGAs). The processors may include single-core or multi-core processors. The processing resources of the computer system 1200 are organized into one or more processing units 1232, 1234, etc. The processing units may include one or more processors, one or more cores from the same or different processors, combinations of cores and processors, or other combinations of cores and processors. In some examples, the processing subsystem 1204 may include one or more dedicated co-processors, such as graphics processors, digital signal processors (DSPs), etc. In some examples, some or all of the processing units of the processing subsystem 1204 are realized using customized circuitry, such as application specific integrated circuits (ASICs) or field programmable gate arrays (FPGAs).

[0207] In some examples, the processing units in the processing subsystem 1204 may execute instructions stored in the system memory 1210 or the computer-readable storage medium 1222. In various examples, the processing units may execute various program or code instructions and maintain multiple programs or processes running simultaneously. At any given time, some or all of the program code to be executed resides in the system memory 1210 and / or the computer-readable storage medium 1222, potentially including one or more storage devices. Through appropriate programming, the processing subsystem 1204 may provide the various functions described above. In examples where the computer system 1200 is running one or more virtual machines, one or more processing units are assigned to each virtual machine.

[0208] In certain examples, a processing acceleration unit 1206 may be optionally provided to accelerate the overall processing performed by the computer system 1200, to perform customized processing, or to offload some of the processing performed by the processing subsystem 1204.

[0209] I / O subsystem 1208 may include devices and mechanisms for inputting information to computer system 1200 and / or outputting information from or through computer system 1200. In general, use of the term "input device" is intended to include all conceivable types of devices and mechanisms for inputting information to computer system 1200. User interface input devices may include, for example, keyboards, pointing devices such as mice or trackballs, touchpads or touchscreens integrated into displays, scroll wheels, click wheels, dials, buttons, switches, keypads, voice input devices with voice command recognition systems, microphones, and other types of input devices. User interface input devices may also include motion sensing and / or gesture recognition devices, such as Microsoft Kinect® motion sensors that allow a user to control and interact with the input device, Microsoft Xbox® 360 game controllers, devices that provide an interface for receiving input using gestures and voice commands. The user interface input devices may also include eye gesture recognition devices, such as a Google Glass® blink detector, that detects eye movements from the user (e.g., "blinking" while taking a picture and / or making a menu selection) and translates the eye gestures as input to the input device (e.g., Google Glass®). The user interface input devices may also include a voice recognition sensing device that allows the user to interact with a voice recognition system (e.g., the Siri® navigator) via voice commands.

[0210] Other examples of user interface input devices may include, but are not limited to, three-dimensional (3D) mice, joysticks or pointing sticks, game pads and graphic tablets, as well as audio / visual devices such as speakers, digital cameras, digital camcorders, portable media players, webcams, image scanners, fingerprint scanners, barcode readers 3D scanners, 3D printers, laser range finders, and eye-tracking devices. User interface input devices may also include medical imaging input devices, such as, for example, computed tomography, magnetic resonance imaging, position emission tomography, and medical ultrasound devices. User interface input devices may also include audio input devices, such as, for example, MIDI keyboards, digital musical instruments, and the like.

[0211] In general, use of the term output device is intended to include all conceivable types of devices and mechanisms for outputting information from computer system 1200 to a user or to another computer. User interface output devices may include display subsystems, indicator lights, or non-visual displays such as audio output devices, etc. Display subsystems are flat panel devices such as those using cathode ray tubes (CRTs), liquid crystal displays (LCDs) or plasma displays, projection devices, touch screens, etc. For example, user interface output devices may include, but are not limited to, various display devices that visually convey text, graphics, and audio / visual information, such as monitors, printers, speakers, headphones, automobile navigation systems, plotters, audio output devices, and modems.

[0212] The storage subsystem 1218 provides a repository or data store for storing information and data used by the computer system 1200. The storage subsystem 1218 provides a tangible, non-transitory computer-readable storage medium for storing basic programming and data constructs that provide some example functionality. Software (e.g., programs, code modules, instructions) that, when executed by the processing subsystem 1204, provide the functionality described above may be stored in the storage subsystem 1218. The software is executed by one or more processing units of the processing subsystem 1204. The storage subsystem 1218 may also provide authentication in accordance with the teachings of the present disclosure.

[0213] The storage subsystem 1218 may include one or more non-transitory memory devices, including volatile and non-volatile memory devices. As shown in FIG. 12, the storage subsystem 1218 includes a system memory 1210 and a computer-readable storage medium 1222. The system memory 1210 may include several memories, including a volatile main random access memory (RAM) for storing instructions and data during program execution, and a non-volatile read-only memory (ROM) or flash memory in which fixed instructions are stored. In some implementations, a basic input / output system (BIOS), containing basic routines that help transfer information between elements within the computer system 1200, such as during start-up, may typically be stored in a ROM. Typically, the RAM contains data and / or program modules currently being operated on and executed by the processing subsystem 1204. In some implementations, the system memory 1210 may include several different types of memories, such as static random access memory (SRAM), dynamic random access memory (DRAM), and the like.

[0214] 12, system memory 1210 may load running application programs 1212, program data 1214, and operating system 1216, which may include various applications such as a web browser, a middle tier application, a relational database management system (RDBMS), etc. By way of example, operating system 1216 may include Microsoft Windows®, Apple Macintosh®, and / or Linux operating systems, various commercially available UNIX® or UNIX-like operating systems (including but not limited to various GNU / Linux operating systems, Google Chrome® OS, etc.), and / or various versions of mobile operating systems such as iOS®, Windows Phone, Android® OS, BlackBerry® OS, Palm® OS operating systems, etc.

[0215] The computer-readable storage medium 1222 can store programming and data configurations that provide some example functionality. The computer-readable storage medium 1222 can provide storage of computer-readable instructions, data structures, program modules, and other data for the computer system 1200. Software (programs, code modules, instructions) that provide the above-mentioned functionality when executed by the processing subsystem 1204 is stored in the storage subsystem 1218. As an example, the computer-readable storage medium 1222 can include non-volatile memory such as a hard disk drive, a magnetic disk drive, a CD ROM, a DVD, an optical disk drive such as a Blu-Ray® disk, or other optical media. The computer-readable storage medium 1222 may include, but is not limited to, a Zip® drive, a flash memory card, a Universal Serial Bus (USB) flash drive, a Secure Digital (SD) card, a DVD disk, a digital video tape, and the like. The computer-readable storage medium 1222 may also include solid-state drives (SSDs) based on non-volatile memory such as flash memory-based SSDs, enterprise flash drives, solid-state ROM, etc., SSDs based on volatile memory such as solid-state RAM, dynamic RAM, static RAM, DRAM-based SSDs, magnetoresistive RAM (MRAM) SSDs, and hybrid SSDs that use a combination of DRAM and flash memory-based SSDs.

[0216] In particular examples, storage subsystem 1218 may also include a computer-readable storage medium reader 1220 that may be further connected to a computer-readable storage medium 1222. Reader 1220 may be configured to receive and read data from a memory device such as a disk, a flash drive, or the like.

[0217] In certain examples, computer system 1200 may support virtualization techniques, including but not limited to virtualization of processing and memory resources. For example, computer system 1200 may provide support for running one or more virtual machines. In certain examples, computer system 1200 may execute a program, such as a hypervisor, that facilitates configuration and management of virtual machines. Each virtual machine is assigned memory, computing (e.g., processors, cores), I / O, and networking resources. Each virtual machine typically runs independently from other virtual machines. A virtual machine typically runs its own operating system, which may be the same as or different from the operating systems run by other virtual machines executed by computer system 1200. Thus, potentially multiple operating systems may be run simultaneously by computer system 1200.

[0218] The communication subsystem 1224 provides an interface to other computer systems and networks. The communication subsystem 1224 serves as an interface for sending and receiving data between other systems and the computer system 1200. For example, the communication subsystem 1224 may enable the computer system 1200 to establish a communication channel to one or more client devices over the Internet to send and receive information to and from the one or more client devices. For example, if the computer system 1200 is used to implement the bot system 120 shown in FIG. 1, the communication subsystem is used to communicate with a chatbot system selected for the application.

[0219] The communications subsystem 1224 may support both wired and / or wireless communication protocols. In certain examples, the communications subsystem 1224 may include a radio frequency (RF) transceiver component for accessing wireless voice and / or data networks (e.g., using cellular telephone technology, advanced data network technologies such as 3G, 4G, or EDGE (Enhanced Data Rates for Global Evolution), WiFi (IEEE 802.XX family of standards, or other mobile communications technologies, or any combination thereof), a global positioning system (GPS) receiver component, and / or other components. In some examples, the communications subsystem 1224 may provide a wired network connection (e.g., Ethernet) in addition to or instead of a wireless interface.

[0220] The communications subsystem 1224 may receive and transmit data in a variety of formats. In some examples, the communications subsystem 1224 may receive incoming communications in the form of structured and / or unstructured data feeds 1226, event streams 1228, event updates 1230, etc., in addition to other formats. For example, the communications subsystem 1224 is configured to receive (or transmit) data feeds 1226 in real time from users of social media networks and / or other communications services, such as web feeds, such as Twitter® feeds, Facebook® updates, Rich Site Summary (RSS) feeds, and / or real-time updates from one or more third party sources, etc.

[0221] In particular examples, the communications subsystem 1224 is configured to receive data in the form of a continuous data stream, which may include an event stream 1228 of real-time events and / or event updates 1230 that are continuous or infinite in nature without a clear end. Examples of applications that generate continuous data may include, for example, sensor data applications, financial stock ticker boards, network performance measurement tools (e.g., network monitoring and traffic management applications), clickstream analysis tools, automobile traffic monitoring, and the like.

[0222] Communications subsystem 1224 may be configured to communicate data from computer system 1200 to other computer systems or networks. This data may be communicated in a variety of different formats, such as structured and / or unstructured data feeds 1226, event streams 1228, event updates 1230, etc., to one or more databases in communication with one or more streaming data source computers coupled to computer system 1200.

[0223] The computer system 1200 may be one of a variety of types, including a handheld portable device (e.g., an iPhone® cellular phone, an iPad® computing tablet, a PDA), a wearable device (e.g., a Google Glass® head-mounted display), a personal computer, a workstation, a mainframe, a kiosk, a server rack, or other data processing system. Due to the ever-changing nature of computers and networks, the description of the computer system 1200 shown in FIG. 12 is intended only as a specific example. Many other configurations are possible having more or fewer components than the system shown in FIG. 12. Based on the disclosure and teachings herein, it should be recognized that there are other aspects and / or methods for implementing the various examples.

[0224] Although specific examples have been described, various variations, modifications, alternative configurations, and equivalents are possible. The examples are not limited to operation in a particular data processing environment, but may freely operate in multiple data processing environments. Moreover, while the examples have been described using a particular sequence of transactions and steps, it should be apparent to one skilled in the art that this is not intended to be limiting. Although some flow charts describe operations as a sequential process, many of these operations are performed in parallel or simultaneously. In addition, the order of operations may be re-specified. A process may have additional steps not included in the figures. Various features and aspects of the above examples may be used individually or together.

[0225] Additionally, while particular examples have been described using particular combinations of hardware and software, it should be understood that other combinations of hardware and software are possible. Particular examples may be implemented solely in hardware, solely in software, or using a combination thereof. The various processes described herein may be implemented on the same processor or any combination of different processors.

[0226] Where a device, system, component or module is described as being configured to perform a particular operation or function, such configuration is achieved, for example, by designing an electronic circuit to perform the operation, by programming a programmable electronic circuit (such as a microprocessor) to perform the operation, by executing, for example, computer instructions or code, or a processor or core programmed to execute code or instructions stored in a non-transitory memory medium or any combination thereof. Processes may communicate using a variety of techniques, including, but not limited to, conventional techniques for inter-process communication, and different pairs of processes may use different techniques, and the same pair of processes may use different techniques at different times.

[0227] In this disclosure, specific details are set forth to ensure that the examples are thoroughly understood. However, the examples may be practiced without these specific details. For example, well-known circuits, processes, algorithms, structures, and techniques are shown without unnecessary detail so as not to obscure the examples. This specification provides only illustrative examples and is not intended to limit the scope, applicability, or configuration of other examples. Rather, the above description of the examples provides one skilled in the art with an enabling description for implementing various examples. Various changes may be made within the functionality and configuration of the elements.

[0228] Accordingly, the specification and drawings are to be regarded in an illustrative rather than restrictive sense. It will be apparent, however, that additions, subtractions, deletions, and other modifications and alterations may be made thereto without departing from the broader spirit and scope as set forth in the following claims. Thus, while specific examples have been described, they are not intended to be limiting. Various modifications and equivalents are within the scope of the appended claims.

[0229] In the above specification, aspects of the disclosure are described with reference to specific examples thereof, but those skilled in the art will recognize that the disclosure is not limited thereto. Various features and aspects of the above disclosure may be used individually or together. Moreover, the examples may be utilized in a variety of environments and applications beyond those described herein without departing from the broader spirit and scope of the specification. Thus, the specification and drawings should be regarded as illustrative rather than restrictive.

[0230] In the above description, the methods are described in a particular order for illustrative purposes. It should be understood that in alternative examples, the methods are performed in an order different from that described. It should also be understood that the above methods are performed by hardware components or embodied in a sequence of machine-executable instructions that, when used, cause a machine, such as a general-purpose or special-purpose processor or logic circuitry programmed with such instructions, to perform the method. These machine-executable instructions can be stored on one or more machine-readable media, such as a CD-ROM or other type of optical disk, a floppy disk, a ROM, a RAM, an EPROM, an EEPROM, a magnetic or optical card, a flash memory, or other type of machine-readable medium suitable for storing electronic instructions. Alternatively, the methods are performed by a combination of hardware and software.

[0231] Where a component is described as being configured to perform particular operations, such configuration may be achieved, for example, by designing electronic circuitry or other hardware to perform the particular operations, by programming a programmable electronic circuitry (e.g., a microprocessor or other suitable electronic circuitry) to perform the particular operations, or any combination thereof.

[0232] Although illustrative examples of the present application have been described in detail herein, it is to be understood that the concepts of the present invention may be variously embodied and employed in other forms, and the claims are intended to be construed to include such modifications except insofar as limited by the prior art.

Claims

1. 1. A method comprising: receiving, at a computing device, a set of utterance data corresponding to a natural language based query interpretable by the chatbot; and determining, by the computing device, one or more intents for the chatbot, each of the one or more intents corresponding to a possible context for the natural language based query and associated with a skill of one or more skills for the chatbot, the method further comprising: and generating, using an intent classifier model, one or more intent classification data sets based on the one or more intents, each intent classification data set being associated with a probability that the natural language based query corresponds to an intent of the one or more intents, the method further comprising: and generating, based on the one or more intent classification datasets using a transformation mask model, one or more transformed datasets, each of the one or more transformed datasets corresponding to a skill of the one or more skills, the method further comprising: determining, by the computing device, a first skill from the one or more skills based on the one or more transformed data sets; and processing the set of utterance data based on the determined first skill to resolve the natural language based query.

2. The method of claim 1 , wherein the set of utterance data is received from a client device interacting with an automated digital assistant in communication with the computing device.

3. The method of claim 1 or 2, wherein the transformation mask model includes one or more transformation values, each transformation value corresponding to a ratio of at least one intent and at least one skill.

4. the set of utterance data includes at least an indication of a ground truth skill of the one or more skills; The method further comprises: the computing device comparing the ground truth skill to the first skill to generate one or more training loss values; and modifying the one or more transformation values ​​based on the training loss value.

5. the intent classifier model is a machine learning model configured to receive as input the set of utterance data and output the one or more intent classification data sets; the set of utterance data includes at least an indication of a ground truth skill of the one or more skills; The method further comprises: the computing device comparing the ground truth skill to the first skill to generate one or more training loss values; The method of any one of claims 1 to 4, further comprising: the computing device training the machine learning model using the one or more training loss values ​​by modifying a structure of the machine learning model based on the training loss values.

6. The method further comprises: The computing device may further include generating second skill data using a skill classifier model, the skill classifier model being a machine learning model configured to receive the set of utterance data as an input and output a predicted skill from the one or more skills, the method further comprising: The method further includes: the computing device comparing the first skill and the second skill to generate one or more training loss values, the one or more training loss values ​​corresponding to one or more differences between the first skill and the second skill, the method further comprising:

6. The method of claim 1, further comprising: the computing device training the machine learning model using the one or more training loss values ​​by modifying a structure of the machine learning model based on the training loss values.

7. The method further comprises: processing the set of utterance data based on the determined first skill to generate a natural language based response to the natural language based query; and transmitting the natural language based response to a client device.

8. 8. The method of claim 1, further comprising: the computing device processing the speech data to generate refined speech data in a format for input to the intent classifier model, the refined speech data being input to the intent classifier model to generate the one or more intent classification data sets.

9. each transformed data set of the one or more transformed data sets includes a probability value corresponding to a likelihood that the utterance data corresponds to a skill of the one or more skills; 9. The method of claim 1, wherein determining the first skill comprises selecting a skill corresponding to the transformed data set having a highest probability value from a set of one or more probability values.

10. 10. The method of claim 1, wherein determining the first skill comprises generating one or more average probability values ​​for each skill based on probability values ​​associated with the intent classification dataset, and selecting a skill associated with a highest average probability value from the one or more average probability values.

11. 1. A system comprising an integrated computing system, the integrated computing system including one or more processors and a non-transitory machine-readable storage medium having instructions stored thereon, the instructions, when executed by the one or more processors, causing the one or more processors to: receiving, at a computing device, a set of utterance data corresponding to a natural language based query interpretable by the chatbot; and determining one or more intents for the chatbot, each of the one or more intents corresponding to a possible context for the natural language based query and associated with a skill of one or more skills for the chatbot. The instructions, when executed by the one or more processors, further cause the one or more processors to: and causing the computing device to generate one or more intent classification datasets based on the one or more intents using an intent classifier model, each intent classification dataset being associated with a probability that the natural language based query corresponds to an intent of the one or more intents. The instructions, when executed by the one or more processors, further cause the one or more processors to: and causing the computing device to generate one or more transformed data sets based on the one or more intent classification data sets using a transformation mask model, each of the one or more transformed data sets corresponding to a skill of the one or more skills. The instructions, when executed by the one or more processors, further cause the one or more processors to: determining, by the computing device, a first skill from the one or more skills based on the one or more transformed data sets; and processing the set of utterance data based on the determined first skill to resolve the natural language based query.

12. The system of claim 11 , wherein the set of utterance data is received from a client device interacting with an automated digital assistant in communication with the computing device.

13. The system of claim 11 or 12, wherein the transformation mask model includes one or more transformation values, each transformation value corresponding to a ratio of at least one intent and at least one skill.

14. the set of utterance data includes at least an indication of a ground truth skill of the one or more skills; The instructions further cause the one or more processors to: the computing device comparing the ground truth skill to the first skill to generate one or more training loss values; and modifying the one or more transformation values ​​based on the training loss value.

15. the intent classifier model is a machine learning model configured to receive as input the set of utterance data and output the one or more intent classification data sets; the set of utterance data includes at least an indication of a ground truth skill of the one or more skills; The instructions further cause the one or more processors to: the computing device comparing the ground truth skill to the first skill to generate one or more training loss values; and training the machine learning model using the one or more training loss values ​​by modifying a structure of the machine learning model based on the training loss values.

16. The computing device further includes generating second skill data using a skill classifier model, the skill classifier model being a machine learning model configured to receive the set of utterance data as an input and output a predicted skill from the one or more skills; and the computing device comparing the first skill and the second skill to generate one or more training loss values, the one or more training loss values ​​corresponding to one or more differences between the first skill and the second skill; and The system of any one of claims 11 to 15, further comprising: the computing device training the machine learning model using the one or more training loss values ​​by modifying a structure of the machine learning model based on the training loss values.

17. A computer program for causing one or more processors to execute the method according to any one of claims 1 to 10.

Images