Automatic out-of-scope transitions in chatbots

Abstract

In one aspect, a method is provided for automatically switching between chatbot skills within a same domain, the method including receiving an utterance from a user in a chatbot session, where a current skill context is a first skill and a current group context is a first group, the method further including inputting the utterance to a candidate skill model for the first group, obtaining a ranking of skills in the first group using the candidate skill model, determining, based on the skill rankings, that a second skill is the highest-ranked skill, changing the current skill context of the chatbot session to the second skill, inputting the utterance to a candidate flow model for the second skill, obtaining, using the candidate flow model, a ranking of intents in the second skill that match the utterance, and determining, based on the intent rankings, that an intent is the highest-ranked intent.

Description

Technical Field

[0001] Priority Claim This application claims the benefit of priority based on U.S. Provisional Application No. 63 / 085,796, filed Sep. 30, 2020, which is hereby incorporated by reference in its entirety for all purposes.

[0002] Field of the Invention The present disclosure generally relates to chatbots, and more specifically to techniques for automatically switching between chatbot skills within the same domain.

Background Art

[0003] Background A large number of users around the world are on instant messaging or chat platforms to get instant responses. It is common for organizations to use these instant messaging or chat platforms to interact with customers (or end users) through live conversations. However, hiring service representatives to interact with customers or end users through live communication can be very costly for an organization. Chatbots or bots have begun to be developed to simulate conversations with end users, especially over the Internet. An end user can interact with a bot through a messaging application that the end user has already installed and is using. Generally, intelligent bots equipped with artificial intelligence (AI) can converse more sophisticatedly and contextually in a live conversation, thus enabling a more natural conversation between the bot and the end user for an advanced conversational experience. Instead of the end user learning a fixed set of keywords or commands that the bot knows how to respond to, an intelligent bot will be able to understand the end user's intent based on the end user's utterance in natural language and respond accordingly.

Summary of the Invention

[0004] Brief Overview A technology is provided for automatically switching between chatbot skills within the same domain (e.g., a method, a system, or a non-transient computer-readable medium containing code or instructions executable by one or more processors). [Means for solving the problem]

[0005] In various embodiments, a computer-based method is provided, which includes the steps of receiving an utterance from a user in a chatbot session, wherein the current skill context of the chatbot session is a first skill, and the current group context of the chatbot session is a first group, and the method further includes the steps of inputting the utterance into a candidate skill model for the first group, using the candidate skill model to obtain a ranking of the skills in the first group that may process the utterance, determining, based on the skill ranking, that a second skill is the highest-ranked skill for processing the utterance, changing the current skill context of the chatbot session to the second skill, inputting the utterance into a candidate flow model for the second skill, using the candidate flow model to obtain a ranking of intents in the second skill that match the utterance, and determining, based on the intent ranking, that intent is the highest-ranked intent for processing the utterance.

[0006] In some embodiments, the step of obtaining skill rankings includes evaluating utterances and generating confidence scores for skills within a first group; identifying any skills having confidence scores exceeding a candidate skill confidence threshold routing parameter value as candidate skills for further evaluation; and ranking the candidate skills, based on their confidence scores, as skills within a first group that may process the utterances.

[0007] In some embodiments, the step of obtaining intent rankings includes evaluating intents and generating confidence scores for intents in a second skill, and identifying any intents with confidence scores exceeding the value of a confidence threshold routing parameter as candidates for further evaluation. Intent The process includes the steps of identifying a candidate intent and ranking it as an intent within a first skill that matches the utterance, based on its confidence score.

[0008] In some embodiments, this method further includes the step of initiating a conversational flow with a user within a chatbot session based on an intent which is the highest-ranking intent for processing an utterance.

[0009] In some embodiments, the method further includes the steps of: receiving an initial utterance from a user in a chatbot session before any utterances are made; inputting the initial utterance into a candidate skill model; using the candidate skill model to obtain a ranking of skills that may be able to handle the initial utterance; determining, based on the skill ranking, that a first skill is the highest-ranked skill for handling the initial utterance; assigning the current skill context of the chatbot session to the first skill; and assigning the current group context of the chatbot session to the first group, wherein the first group is defined for the first skill, and assigning the current group context of the chatbot session to the first group is performed based on the definition of the first group for the first skill.

[0010] In some embodiments, the method further includes the step of receiving a subsequent utterance from a user within a chatbot session, wherein the current skill context of the chatbot session is a second skill, and the current group context of the chatbot session is a first group, and the method includes the steps of inputting the subsequent utterance into a candidate skill model for the first group, using the candidate skill model to obtain a ranking of the skills within the first group that may handle the subsequent utterance, determining, based on the skill ranking, that the unresolved intent skill is the highest-ranked skill for handling the subsequent utterance, inputting the subsequent utterance into another candidate skill model, and using the other candidate skill model The process further includes the steps of obtaining a ranking of skills that may be able to process subsequent utterances, determining, based on the skill rankings, that a third skill is the highest-ranked skill for processing subsequent utterances, determining, based on the skill rankings, that a third skill is the highest-ranked skill for processing subsequent utterances, assigning the current skill context of the chatbot session to the third skill, and assigning the current group context of the chatbot session to the second group, wherein the second group is defined for the third skill, and assigning the current group context of the chatbot session to the second group is performed based on the definition of the second group for the third skill.

[0011] In some embodiments, the method further includes the steps of: determining, based on skill ranking, that a third skill is the second highest-ranked skill for processing an utterance; determining that both the second and third skills are within the win margin parameter; inputting the utterance into a candidate flow model for the second skill and another candidate flow model for the third skill, in response to the determination that both the second and third skills are within the win margin parameter; obtaining intent rankings in the third skill that match the utterance using the other candidate flow model; and determining, based on intent rankings, the intent in the second and third skills that is the highest-ranked intent for processing an utterance.

[0012] Some embodiments of this disclosure include a system comprising one or more data processors. In some embodiments, the system comprises a non-temporary computer-readable storage medium containing instructions, which, when executed on one or more data processors, cause one or more data processors to execute some or all of the methods and / or some or all of the processes described herein.

[0013] Some embodiments of this disclosure include computer program products implemented in tangible form on a non-temporary machine-readable storage medium, which include instructions causing one or more data processors to execute some or all of the methods and / or some or all of the processes disclosed herein.

[0014] The above and below technologies can be implemented in numerous ways and in numerous contexts. Several examples of implementations and contexts are provided below with reference to the diagrams, which are described in more detail later. However, the following implementations and contexts represent only a small fraction of the many available. [Brief explanation of the drawing]

[0015] [Figure 1] A schematic diagram showing the concepts of within domain, outside domain, within range, and outside range according to various embodiments is shown. [Figure 2] A simplified block diagram of a distributed environment incorporating an exemplary embodiment is shown. [Figure 3] A simplified block diagram of a computing system implementing a master bot according to an embodiment is shown. [Figure 4] A simplified block diagram of a computing system implementing a skill bot according to an embodiment is shown. [Figure 5A] A diagram showing a graphical user interface according to various embodiments is shown. [Figure 5B] A diagram showing a graphical user interface according to various embodiments is shown. [Figure 6] A diagram showing an intent call for an exemplary utterance according to various embodiments is shown. [Figure 7] A diagram showing an intent call for another exemplary utterance according to various embodiments is shown. [Figure 8] A diagram showing an intent call for another exemplary utterance according to various embodiments is shown. [Figure 9] A diagram showing an intent call for another exemplary utterance according to various embodiments is shown. [Figure 10A] A diagram showing a skill group context correction for a rule output according to various embodiments is shown. [Figure 10B] A diagram showing a skill group context correction for an intent call according to various embodiments is shown. [Figure 10C] A diagram showing a skill group context correction for a routing summary according to various embodiments is shown. [Figure 11] A diagram showing a process flow of context-aware routing by a skill group according to various embodiments is shown. [Figure 12] A schematic diagram of a distributed system for implementing various embodiments is shown. [Figure 13]A simplified block diagram of one or more components of a system environment in which services provided by one or more components of an embodiment system according to various embodiments can be provided as cloud services. [Figure 14] A diagram showing an example of a computer system that can be used to implement various embodiments.

Best Mode for Carrying Out the Invention

[0016] Detailed Description In the following description, for the purpose of explanation, specific details are set forth in order to provide a thorough understanding of particular embodiments. However, it will be apparent that various embodiments may be practiced without these specific details. The drawings and description are not intended to be limiting. The term "exemplary" is used herein to mean "functioning as an example, instance, or illustration." Any embodiment or design described herein as "exemplary" should not necessarily be construed as preferred or advantageous over other embodiments or designs.

[0017] Overview A digital assistant is an artificial intelligence-driven interface that helps users perform a variety of tasks in natural language conversation. Customers can assemble one or more skills for each digital assistant. Skills (also referred to herein as chatbots, bots, or skillbots) are individual bots focused on specific types of tasks, such as tracking inventory, submitting time cards, and creating expense reports. When an end user interacts with a digital assistant, the digital assistant evaluates the end user's input and routes the conversation to the appropriate chatbot. Digital assistants can be made available to end users through various channels, such as Facebook Messenger, Skype Mobile Messenger, or Short Message Service (SMS). Channels facilitate back-and-forth chats from end users on various messaging platforms to the digital assistant and its various chatbots. Channels can also support user agent escalation, event-driven conversations, and testing.

[0018] An intent helps a chatbot understand what a user wants it to do. An intent consists of a reordering of typical user requests and statements (e.g., get account balance, make a purchase), also called utterances. As used herein, an utterance or message may refer to a set of words (e.g., one or more sentences) exchanged during a conversation with a chatbot. An intent may be created by providing a name that describes a user action (e.g., ordering a pizza) and compiling a set of actual user statements or utterances that commonly associate with triggering this action. Since the chatbot's cognition comes from these intents, each intent may be created from a robust (12-24 utterances) and varied dataset to enable the chatbot to interpret ambiguous user input. A rich set of utterances allows the chatbot to understand what the user wants when it receives messages that mean the same thing but are expressed differently, such as "Forget this order!" or "Cancel the delivery!". Intents and the utterances associated with them form a training corpus for the chatbot. By training a model using this corpus, customers can effectively transform this model into a reference tool for deciphering end-user input into a single intent. Customers can enhance the chatbot's cognitive agility through intent testing and intent training rounds.

[0019] However, the utterances that chatbots receive from real users in real-world environments (e.g., production environments) can be diverse and noisy. Some of these received utterances may differ significantly from those used to train the chatbot and may not fit within the intents for which the chatbot was trained to infer and handle. For example, a banking chatbot may receive utterances unrelated to banking, such as "How do I book a trip to Italy?". As shown in Figure 1, such utterances are called out-of-domain (OOD) utterances because they do not fall within the domain of the trained chatbot's intents. What is important for a chatbot system is the ability to identify such OOD utterances so that it can take an appropriate response action. For example, when a chatbot detects an OOD utterance, it may respond to the user by indicating that it is not an utterance that the bot can process or handle, rather than selecting the nearest matching intent.

[0020] Furthermore, a group of skills or chatbots may be deployed as part of the same domain. Typically, these skills are deployed by different groups or departments within a company that are part of the same domain. In such cases, it is common for a chatbot to receive utterances about intents belonging to different skills within the same domain. For example, a rewards chatbot in the Human Capital Management (HCM) domain might receive an utterance such as, "What is the benefit to me?", which is unrelated to rewards but certainly related to benefits, which are part of the HCM domain. As shown in Figure 1, such an utterance is called an out-of-scope (OOS) utterance because it is not within the scope of the trained chatbot's intents. Because the skills are part of the same domain, a user may be involved with skill A (e.g., the rewards chatbot) because a question about skill B (e.g., the benefits chatbot) within the same domain, while out of the scope of skill A, will pass through skill A's OOD detector and may match skill A's intent with relatively high confidence. A crucial aspect for a chatbot system is its ability to identify such out-of-situation (OOS) utterances so that it can take appropriate response actions. For example, when a context-aware router detects an OOS utterance, it routes the utterance from the current chatbot (e.g., a reward chatbot) to the most relevant chatbot within a defined domain group (e.g., a profit chatbot).

[0021] This disclosure describes various embodiments for addressing the problem of identifying out-of-speaking (OOS) utterances and providing OOS transitions between skills within the same domain. In one embodiment, grouping and routing-based techniques are used for OOS determination and transition. In an embodiment for illustrative purposes, a computer-implemented method is provided, which includes the steps of: receiving an utterance from a user in a chatbot session, where the current skill context of the chatbot session is a first skill, and the current group context of the chatbot session is a first group; further, the method includes: inputting the utterance into a candidate skill model for the first group; using the candidate skill model to obtain a ranking of skills within the first group that may process the utterance; determining, based on the skill ranking, that the second skill is the highest-ranked skill in processing the utterance; changing the current skill context of the chatbot session to the second skill; inputting the utterance into a candidate flow model for the second skill; using the candidate flow model to obtain a ranking of intents within the second skill that match the utterance; and determining, based on the intent ranking, that the highest-ranked intent in processing the utterance.

[0022] Bot and analytics systems A bot (also called a skill, chatbot, chatterbot, or talkbot) is a computer program that can engage in conversations with end users. A bot can typically respond to natural language messages (e.g., questions or comments) through a messaging application that uses natural language messaging. A company can use one or more bot systems to communicate with end users through a messaging application. The messaging application, sometimes called a channel, may be a preferred messaging application that the end user has already installed and is familiar with. This eliminates the need for the end user to download and install a new application to chat with the bot system. Messaging applications may include, for example, over-the-top (OTT) messaging channels (such as Facebook Messenger, Facebook WhatsApp, WeChat, Line®, Kik, Telegram, Talk, Skype®, Slack, or SMS), virtual private assistants (such as Amazon® Dot, Echo®, or Show, Google Home®, or Apple HomePod®), mobile and web app extensions that extend native or hybrid / responsive mobile apps or web applications with chat functionality, or voice-based input (such as Siri®, Cortana®, Google® Voice, or devices or apps with interfaces that use other voice input for interaction).

[0023] In some examples, a bot system can be associated with a Uniform Resource Identifier (URI). A URI can identify a bot system using a string. A URI can be used as a webhook for one or more messaging application systems. A URI may include, for example, a Uniform Resource Locator (URL) or Uniform Resource Name (URN). A bot system can be designed to receive messages (e.g., Hypertext Transfer Protocol (HTTP) post-call messages) from a messaging application system. HTTP post-call messages may be directed to a URI from a messaging application system. In some embodiments, messages may differ from HTTP post-call messages. For example, a bot system may receive messages from the Short Message Service (SMS). While the discussion herein may refer to communications that a bot system receives as messages, it should be understood that messages can be HTTP post-call messages, SMS messages, or any other type of communication between two systems.

[0024] End users may interact with bot systems through conversational interactions (sometimes called conversational user interfaces (UI)), similar to interactions between people. In some cases, this interaction may include the end user saying "Hello" to the bot, the bot responding with "Hi," and asking the end user what they need. In other cases, this interaction may include transactions with banking bots, such as transferring money from one account to another; information exchange with HR bots, such as checking remaining vacation time; or interactions with retail bots, such as discussing the return of purchased goods or requesting technical support.

[0025] In some embodiments, a bot system may intelligently handle interactions with end users without interaction with the bot system's administrator or developer. For example, an end user may send one or more messages to the bot system to achieve a desired goal. Messages may include specific content, such as text, emojis, voice, images, videos, or other means of conveying a message. In some embodiments, the bot system may convert the content into a standardized format (e.g., a Representational State Transfer (REST) ​​call to an enterprise service using appropriate parameters) and generate a natural language response. The bot system may also prompt the end user for additional input parameters or request other additional information. In some embodiments, the bot system may also initiate communication with the end user rather than passively responding to the end user's utterance. This specification describes various techniques for identifying explicit calls to a bot system and determining the input to the bot system being called. In certain embodiments, the analysis of explicit calls is performed by the master bot based on the detection of a call name in the utterance. In response to the detection of a call name, the utterance may be refined for input to the skill bot associated with the call name.

[0026] A conversation with a bot may follow a specific conversational flow that includes multiple states. This flow may determine what happens next based on the input. In some embodiments, a bot system may be implemented using a state machine that includes user-defined states (e.g., end-user intents) and actions to be taken in or between states. The conversation may follow different paths based on end-user input, which may influence the decisions the bot makes regarding this flow. For example, in each state, the bot may determine the end-user intent based on end-user input or utterance to determine the appropriate next action to take. As used herein, the term “intent” in the context of an utterance means the intent of the user who provided the utterance. For example, a user may intend to involve the bot in a conversation to order a pizza, and the user’s intent may be expressed through the utterance “order a pizza.” The user’s intent may be directed to a specific task that the user wants the chatbot to perform on their behalf. Thus, an utterance can be expressed as a question, command, request, etc., reflecting the user’s intent. An intent can include the objective that the end user wants to achieve.

[0027] In the context of chatbot configuration, the term “intent” is used herein to mean configuration information for mapping user utterances to specific tasks / actions or categories of tasks / actions that the chatbot can perform. To distinguish between utterance intents (i.e., user intents) and chatbot intents, the latter may be referred to herein as “bot intents.” A bot intent may include one or more sets of utterances associated with that intent. For example, the intent to order a pizza may have various sorts of utterances expressing the desire to order a pizza. By training the chatbot’s intent classifier with these associated utterances, it can later be determined whether an input utterance from a user matches the intent to order a pizza. A bot intent may be associated with one or more dialogue flows for initiating a conversation with a user in a particular state. For example, the first message for the intent to order a pizza might be the question, “What kind of pizza would you like?” In addition to associated utterances, a bot intent may include specified entities associated with the intent. For example, the intent "Pizza Order" might include variables or parameters used to perform the task of ordering a pizza, such as topping 1, topping 2, pizza type, pizza size, and pizza quantity. The entity's values ​​are typically obtained through conversation with the user.

[0028] Figure 2 is a simplified block diagram of an environment 200 incorporating a chatbot system according to one embodiment. Environment 200 includes a digital assistant builder platform (DABP) 202, which enables users of DABP 202 to create and deploy digital assistant or chatbot systems. One or more digital assistants (or DAs) or chatbot systems can be created using DABP 202. For example, as shown in Figure 2, a user 204 representing a specific company can use DABP 202 to create and deploy a digital assistant 206 for users of that company. For example, DABP 202 may be used by a bank to create one or more digital assistants for use by its customers. Multiple companies may use the same DABP 202 platform to create digital assistants. As another example, the owner of a restaurant (e.g., a pizza shop) may use DABP 202 to create and deploy a digital assistant that enables customers of the restaurant to order food (e.g., order pizza).

[0029] For the purposes of this disclosure, “digital assistant” is an entity that helps users of a digital assistant perform various tasks through natural language conversation. A digital assistant can be implemented using software alone (for example, a digital assistant is a digital entity implemented using a program, code, or instructions executable by one or more processors), using hardware, or using a combination of hardware and software. A digital assistant can be implemented or implemented in a variety of physical systems or devices, such as computers, mobile phones, watches, electronic devices, vehicles, etc. A digital assistant is sometimes also called a chatbot system. Therefore, for the purposes of this disclosure, the terms digital assistant and chatbot system are synonymous.

[0030] By using a digital assistant such as a digital assistant 206 built using DABP202, various tasks can be performed through natural language-based conversations between the digital assistant and its user 208. As part of the conversation, the user can provide one or more user inputs 210 to the digital assistant 206 and receive responses 212 from the digital assistant 206. The conversation may include one or more of the inputs 210 and responses 212. Through these conversations, the user can request the digital assistant to perform one or more tasks, and accordingly, the digital assistant is configured to perform the user-requested tasks and respond to the user with appropriate responses.

[0031] User input 210 is generally in natural language form and is called utterance. User utterance 210 may also be in text form, such as when the user types a sentence, a question, a fragment of text, or even a single word and provides the text to the digital assistant 206 as input. In some embodiments, user utterance 210 may also be in audio input or speech form, such as when the user says or speaks what is provided to the digital assistant 206 as input. The utterance is typically the language spoken by the user 208. For example, the utterance may be English or some other language. If the utterance is in speech form, the speech input is converted into a text-form utterance in that particular language, and the text utterance is then processed by the digital assistant 206. Various speech-to-text processing techniques can be used to convert speech or audio input into text utterances, and the text utterance is then processed by the digital assistant 206. In some embodiments, the speech-to-text conversion may be performed by the digital assistant 206 itself.

[0032] Utterances, which may be text or voice utterances, may be fragments, sentences, multiple sentences, one or more words, one or more questions, or combinations of these types. The digital assistant 206 is configured to apply natural language understanding (NLU) techniques to the utterances to understand the meaning of user input. As part of the NLU processing of the utterances, the digital assistant 206 is configured to perform processing to understand the meaning of the utterances, which involves identifying one or more intents and one or more entities corresponding to the utterances. Once the meaning of the utterances is understood, the digital assistant 206 can perform one or more actions or behaviors depending on the understood meaning or intent. For the purposes of this disclosure, it is assumed that the utterances are either text utterances provided directly by the user 208 of the digital assistant 206, or the result of converting input voice utterances into text format. However, this is not intended to be limiting or restrictive in any case.

[0033] For example, user 208's input may be a request to order a pizza by providing an utterance such as "I want to order a pizza." When the digital assistant 206 receives such an utterance, it is configured to understand the meaning of the utterance and take an appropriate action. An appropriate action may include responding to the user with a question that requests user input such as the type of pizza the user wants to order, the size of the pizza, and any toppings. The responses provided by the digital assistant 206 may also be in natural language form and may typically be in the same language as the input utterance. As part of the generation of these responses, the digital assistant 206 may perform natural language generation (NLG). If the user orders a pizza through a conversation between the user and the digital assistant 206, the digital assistant may guide the user to provide all the information necessary to order the pizza, and then the pizza may be ordered at the end of the conversation. The digital assistant 206 may terminate the conversation by outputting information to the user indicating that the pizza has been ordered.

[0034] At a conceptual level, the digital assistant 206 performs various processes in response to utterances received from the user. In some embodiments, this process involves a series of processing steps or a pipeline of processing steps, including, for example, understanding the meaning of the input utterance (using NLU), determining what action should be taken in response to the utterance, performing the action if appropriate, generating a response to be output to the user in response to the user utterance, and outputting the response to the user. NLU processing may include parsing the received input utterance to understand its structure and meaning, and refining and reconstructing the utterance to construct a more understandable form (e.g., logical form) or structure of the utterance. Generating a response may include using natural language generation (NLG) techniques. Thus, natural language processing (NLP) performed by the digital assistant may include a combination of NLU processing and NLG processing.

[0035] NLU processing performed by a digital assistant such as the digital assistant 206 may include a variety of NLU-related processes, such as parsing (e.g., tokenization, headword creation, identification of part-of-speech tags in a sentence, identification of designated entities in a sentence, generation of a dependency tree representing the sentence structure, splitting the sentence into clauses, analysis of individual clauses, analysis of anaphora, performing chunking, etc.). In one embodiment, the NLU processing or a portion of it is performed by the digital assistant 206 itself. In some other embodiments, the digital assistant 206 may use other resources to perform a portion of the NLU processing. For example, the syntax and structure of an input utterance may be identified by processing the sentence using a parser, a part-of-speech tagger, and / or a designated entity recognizer. In one implementation, for English, a parser, a part-of-speech tagger, and a designated entity recognizer, such as those provided by the Stanford NLP Group, are used to analyze the sentence structure and syntax. These are provided as part of the Stanford CoreNLP Toolkit.

[0036] The various examples provided in this disclosure demonstrate English utterances, but these are intended to be merely examples. In some embodiments, the digital assistant 206 may also process utterances in languages ​​other than English. The digital assistant 206 may provide subsystems (e.g., components that implement NLU functionality) configured to perform processing for different languages. These subsystems may be implemented as pluggable units that can be invoked using service calls from the NLU core server. This makes the NLU processing flexible and extensible for each language, which includes enabling processing in different orders. Language packs may be provided for individual languages, in which case the language packs may register a list of subsystems that can be provided from the NLU core server.

[0037] Digital assistants such as the digital assistant 206 shown in Figure 2 can be made available or accessible to their user 208 through various different channels, for example, but not limited to, through certain applications, through social media platforms, through various messaging services and applications (e.g., instant messaging applications), and through other applications or channels. A single digital assistant may have several channels configured to allow it to run and be accessed simultaneously by different services.

[0038] A digital assistant or chatbot system typically includes or is associated with one or more skills. In some embodiments, these skills are individual chatbots (referred to as skillbots) configured to interact with the user and perform specific types of tasks, such as tracking inventory, submitting time cards, creating expense reports, ordering food, verifying bank accounts, making reservations, and purchasing widgets. For example, in the embodiment shown in Figure 2, the digital assistant or chatbot system 206 includes skills 216-1, 216-2, and so on. For the purposes of this disclosure, the terms “skill” and “multiple skills” are used synonymously with the terms “skillbot” and “multiple skillbots,” respectively.

[0039] Each skill associated with a digital assistant helps the user of the digital assistant complete tasks through a conversation with the user, which may include a combination of text or voice input provided by the user and responses provided by the skill bot. These responses may be in the form of text or voice messages to the user and / or may be provided using simple user interface elements (e.g., a selection list) presented to the user for the user to make a choice.

[0040] There are various ways in which skills or skillbots can be associated with or added to a digital assistant. In some examples, a skillbot may be developed by a company and then added to a digital assistant using DABP202, for example, through a user interface provided by DABP202 to register the skillbot with the digital assistant. In other examples, a skillbot may be developed and created using DABP202 and then added to a digital assistant created using DABP202. In yet another example, DABP202 provides an online digital store (called a "skill store") offering multiple skills directed towards a wide range of tasks. Skills offered through the skill store may also publish various cloud services. To add skills to a digital assistant generated using DABP202, a DABP202 user can access the skill store through DABP202, select the desired skill, and indicate that the selected skill will be added to the digital assistant created using DABP202. Skills from the Skill Store can be added to a digital assistant either as is or in a modified form (for example, a DABP202 user may select and clone a specific skill bot provided by the Skill Store, customize or modify the selected skill bot, and then add the modified skill bot to a digital assistant created using DABP202).

[0041] Various different architectures can be used to implement a digital assistant or chatbot system. For example, in one embodiment, a digital assistant created and deployed using DABP202 may be implemented using a master-bot / child (or sub)bot paradigm or architecture. According to this paradigm, the digital assistant is implemented as a master bot that interacts with one or more child bots, which are skill bots. For example, in the embodiment shown in Figure 2, the digital assistant 206 includes a master bot 214 and skill bots 216-1, 216-2, and others that are child bots of the master bot 214. In one embodiment, the digital assistant 206 itself is considered to act as the master bot.

[0042] A digital assistant implemented according to a master-child bot architecture allows users of the digital assistant to interact with multiple skills through an integrated user interface, i.e., through the master bot. When a user interacts with the digital assistant, user input is received by the master bot. The master bot then performs processing to determine the meaning of the user input utterance. The master bot then determines whether it can handle the task requested by the user in the utterance. If not, the master bot selects an appropriate skill bot to handle the user request and routes the conversation to the selected skill bot. This allows users to converse with the digital assistant through a common, single interface, while still providing the ability to use several skill bots configured to perform specific tasks. For example, in the case of a digital assistant developed for an enterprise, the digital assistant's master bot can interface with skill bots with specific functions, such as a CRM bot to perform functions related to customer relationship management (CRM), an ERP bot to perform functions related to enterprise resource planning (ERP), and an HCM bot to perform functions related to human capital management (HCM). In this way, end-users or consumers of digital assistants only need to know how to access the digital assistant through a common master bot interface, with multiple skill bots provided behind the scenes to handle user requests.

[0043] In one embodiment, in a master-bot / child-bot infrastructure, the master bot is configured to recognize an available list of skill bots. The master bot can access metadata that identifies the various available skill bots, and metadata that identifies the capabilities of each skill bot, including the tasks that this skill bot can perform. When a master hot receives a user request in the form of an utterance, it is configured to identify or predict from among several available skill hots which particular skill hot will best respond to or process the user request. The master bot then routes the utterance (or part of the utterance) to that particular skill bot for further processing. Thus, control flows from the master bot to the skill bots. The master bot may support multiple input and output channels. In one embodiment, routing may be performed with the help of processing performed by one or more available skill bots. For example, a skill bot can be trained to infer the intent of an utterance and determine whether the inferred intent matches the intents that constitute the skill bot, as described below. Therefore, routing performed by the master bot may include the skill bot communicating to the master bot whether the skill bot is configured with an intent suitable for processing the utterance.

[0044] The embodiment shown in Figure 2 illustrates a digital assistant 206 including a master bot 214 and skill bots 216-1, 216-2, and 216-3, but this is not intended to be limiting. The digital assistant may include various other components (e.g., other systems and subsystems) that provide the functionality of the digital assistant. These systems and subsystems may be implemented in software only (e.g., code, instructions stored on a computer-readable medium and executable by one or more processors), in hardware only, or in a combination of software and hardware.

[0045] DABP202 provides the infrastructure, services, and features that enable DABP202 users to create digital assistants that include one or more skillbots associated with the digital assistant. In some cases, skillbots can be created by cloning existing skillbots, for example, by cloning skillbots provided by a skill store. As mentioned earlier, DABP202 can provide a skill store or skill catalog that provides multiple skillbots for performing various tasks. DABP202 users can clone skillbots from the skill store. If necessary, the cloned skillbots may be modified or customized. In some other examples, DABP202 users create skillbots from scratch using the tools and services provided by DABP202.

[0046] In one embodiment, creating or customizing a high-level skill bot involves the following steps: (1) Configure settings for the new skillbot (2) Configure one or more intents for SkillBot (3) Constitute one or more entities for one or more intents (4) Train the skillbot (5) Create a dialog flow for SkillBot (6) Add custom components to the skillbot as needed (7) Test and deploy SkillBot The following is a brief explanation of each step.

[0047] (1) Configure settings for a new skillbot - various settings can be configured for a skillbot. For example, a skillbot designer can specify one or more invocation names for the skillbot being created. These invocation names, which then serve as identifiers for the skillbot, may be used by the digital assistant user to explicitly invoke the skillbot. For example, a user can include an invocation name in their utterance to explicitly invoke the corresponding skillbot.

[0048] (2) Configuring one or more intents for a skillbot - The skillbot designer specifies one or more intents (also called bot intents) for the skillbot being created. The skillbot is then trained based on these specified intents. These intents represent categories or classes in which the skillbot is trained to guess about input utterances. Upon receiving an utterance, the trained skillbot guesses the intent of the utterance, and the guessed intent is selected from a predetermined set of intents used to train the skillbot. The skillbot then takes an appropriate action in response to the utterance based on the intent guessed for the utterance. In some cases, the skillbot's intents represent tasks that the skillbot can perform for the user of the digital assistant. Each intent is given an intent identifier or intent name. For example, for a skillbot trained about banking, intents specified for the skillbot might include "CheckBalance," "TransferMoney," and "Depositcheck."

[0049] For each intent defined for a skillbot, the skillbot designer may provide one or more example utterances that represent and explain the intent. These example utterances represent utterances that a user can input to the skillbot about that intent. For example, for the CheckBalance intent, example utterances might include "What's my savings account balance?", "How much is in my checking account?", and "How much money do I have in my account?". Thus, various reorders of typical user utterances can be specified as example utterances for an intent.

[0050] Intents and their associated, example utterances are used as training data to train a skillbot. Various different training techniques may be used. As a result of this training, a predictive model is produced that takes utterances as input and outputs intents inferred about the utterances by the predictive model. In some examples, the input utterances are provided to an intent analysis engine (e.g., a rule-based or machine learning-based classifier run by the skillbot) which is configured to use the trained model to predict or infer intents for the input utterances. The skillbot can then take one or more actions based on the inferred intents.

[0051] (3) Configuring Entities for One or More Intents in the SkillBot - In some cases, additional context may be needed to enable the SkillBot to respond appropriately to user utterances. For example, there may be situations where user input utterances resolve to the same intent in the SkillBot. For example, in the example above, the utterances "What's my savings account balance?" and "How much is in my checking account?" both resolve to the same "CheckBalance" intent, but these utterances are different requests asking for different things. To clarify such requests, one or more entities can be added to the intent. Using the banking SkillBot example, an entity called AccountType, which defines values ​​called "checking" and "saving," may enable the SkillBot to parse user requests and respond appropriately. In the example above, the utterances resolve to the same intent, but the values ​​associated with the AccountType entity are different for the two utterances. This allows the SkillBot to perform potentially different actions for the two utterances, even though they resolve to the same intent. One or more entities can be specified for a particular intent configured for the skillbot. Therefore, entities are used to add context to the intent itself. Entities help to describe the intent more completely and enable the skillbot to complete the user request.

[0052] In one embodiment, there are two types of entities: (a) built-in entities provided by DABP202, and (2) custom entities that can be specified by the skill bot designer. Built-in entities are general-purpose entities that can be used for a variety of bots. Examples of built-in entities include, but are not limited to, entities related to time, date, address, number, email address, period, recurrence period, currency, phone number, URL, etc. Custom entities are used for more customized applications. For example, in the case of a banking skill, the AccountType entity may be defined by the skill bot designer and enable various banking transactions by checking user input for keywords such as checking, savings, and credit cards.

[0053] (4) Training the SkillBot - The SkillBot is configured to receive user input in the form of utterances, parse or otherwise process the received input, and identify or select intents related to the received user input. As described above, the SkillBot must be trained for this purpose. In one embodiment, the SkillBot is trained based on intents configured for the SkillBot and utterance examples associated with those intents (collectively known as training data) so that the SkillBot can resolve user input utterances for one of its configured intents. In one embodiment, the SkillBot uses a predictive model trained with the training data to enable the SkillBot to identify which user is speaking (or possibly about to speak). DABP202 provides a variety of different training techniques that can be used by SkillBot designers to train SkillBots, including a variety of machine learning-based training techniques, rule-based training techniques, and / or combinations thereof. In one embodiment, a portion of the training data (e.g., 80%) is used to train the SkillBot model, and another portion (e.g., the remaining 20%) is used to test or validate the model. Once trained, the trained model (sometimes called a trained skillbot) can then be used to process and respond to user utterances. In some cases, a user utterance may be a question that requires only a single answer and does not require further conversation. To address such situations, a Q&A (question and answer) intent may be defined for the skillbot. Q&A intents are created in the same way as regular intents. The dialogue flow for a Q&A intent may differ from the dialogue flow for a regular intent. For example, unlike regular intents, the dialogue flow for a Q&A intent may not involve prompting the user for additional information (e.g., the value of a particular entity).

[0054] (5) Create a dialogue flow for the skillbot - A dialogue flow specified for a skillbot describes how the skillbot responds when different intents about the skillbot are resolved in response to incoming user input. The dialogue flow defines the actions or behaviors that the skillbot will perform, such as how the skillbot responds to user utterances, how the skillbot prompts the user for input, and how the skillbot returns data. The dialogue flow is like a flowchart that the skillbot follows. The skillbot designer specifies the dialogue flow using a language such as Markdown. In one embodiment, a version of YAML called OBotML may be used to specify the dialogue flow for the skillbot. Defining the dialogue flow for the skillbot acts as a model of the conversation itself, allowing the skillbot designer to orchestrate the interaction between the skillbot and the user that the skillbot serves.

[0055] In one embodiment, the definition of the dialogue flow for the skill bot is expressed as three sections: (a) Context section (b) Default transition section (c) State section Includes.

[0056] Context Section - Skillbot designers can define the variables used in the conversation flow within the context section. Other variables that can be specified in the context section include, but are not limited to, variables for error handling, variables for built-in or custom entities, and user variables that allow Skillbot to recognize and persist user preferences.

[0057] Default Transition Section - Transitions for a skillbot can be defined in a dialog flow state section or a default transition section. Transitions defined in the default transition section act as fallbacks and are triggered when there are no applicable transitions defined within a state, or when the conditions required to trigger a state transition cannot be met. The default transition section may also be used to define routing that allows the skillbot to handle unexpected user actions with care.

[0058] The State Section—Dialogue flow and its associated behavior are defined as a sequence of transient states that govern the logic within the dialog flow. Each state node in the dialog flow definition specifies a component that provides the functionality required at that point in the dialog. States are built around components in this way. States contain component-specific characteristics and define transitions to other states that are triggered after the execution of a component.

[0059] Special case scenarios may be handled using the state section. For example, it may be desirable to provide the user with the option to temporarily leave the first skill they are involved with in order to do something with a second skill within the digital assistant. For instance, if a user is having a conversation with a shopping skill (e.g., the user has made some choices for a purchase), the user may want to jump to a banking skill (e.g., the user may want to ensure they have enough money for the purchase) and then return to the shopping skill to complete the user's order. To accommodate this, the state section in the first skill's dialog flow definition can be configured to initiate an interaction with a second, different skill within the same digital assistant and then return to the original dialog flow.

[0060] (6) Adding Custom Components to the SkillBot - As previously mentioned, the states specified in the dialog flow for the SkillBot specify components that provide the functionality required for those states. Components enable the SkillBot to perform the functionality. In one embodiment, DABP202 provides a set of pre-configured components for performing a wide range of functions. The SkillBot designer can select one or more of these pre-configured components and associate them with states in the SkillBot's dialog flow. The SkillBot designer can use the tools provided by DABP202 to create custom or new components and associate the custom components with one or more states in the dialog flow for the SkillBot.

[0061] (7) Test and deploy the skillbot - DABP202 provides several features that allow skillbot designers to test the skillbot under development. The skillbot can then be deployed and included in the digital assistant.

[0062] The above description explains how to create a skillbot, but similar techniques may be used to create a digital assistant (or masterbot). At the masterbot or digital assistant level, built-in system intents may be configured for the digital assistant. These built-in system intents are used to identify general tasks that the digital assistant itself (i.e., the masterbot) can handle without calling the skillbot associated with the digital assistant. Examples of system intents defined for a masterbot include: (1) Exit: Applies when the user signals to the digital assistant that they wish to exit the current conversation or context. (2) Help: Applies when the user requests help or guidance. (3) UnresolvedIntent: Applies to user input that does not adequately match the exit and help intents. The digital assistant also stores information about one or more skillbots associated with it. This information allows the masterbot to select a specific skillbot to process the utterance.

[0063] At the master bot or digital assistant level, when a user inputs a phrase or utterance into the digital assistant, the digital assistant is configured to perform a process that determines how to route the utterance and the associated conversation. The digital assistant uses a routing model that can be rule-based, AI-based, or a combination of both to make this determination. Using the routing model, the digital assistant determines whether the conversation corresponding to the user-input utterance should be routed to a specific skill for handling, handled by the digital assistant or master bot itself according to built-in system intents, or handled as a different state in the current conversation flow.

[0064] In one embodiment, as part of this process, the digital assistant determines whether the user input utterance explicitly identifies a skillbot using its invocation name. If an invocation name is present in the user input, it is treated as an explicit invocation of the skillbot corresponding to the invocation name. In such a scenario, the digital assistant may route the user input to the explicitly invocationed skillbot for further processing. If there is no specific or explicit invocation, in one embodiment, the digital assistant evaluates the received user input utterance and calculates confidence scores for the system intents and skillbots associated with the digital assistant. The calculated scores for the skillbots or system intents indicate how well the user input represents the task or system intent that the skillbot is configured to perform. Any system intents or skillbots with relevant calculated confidence scores that exceed a threshold (e.g., a confidence threshold routing parameter) are selected as candidates for further evaluation. The digital assistant then selects a specific system intent or skillbot from the identified candidates for further processing of the user input utterance. In one embodiment, after one or more skill bots are identified as candidates, intents associated with those candidate skills are evaluated (using a model trained for each skill), and a confidence score is determined for each intent. Generally, any intent with a confidence score above a threshold (e.g., 70%) is treated as a candidate intent. Once a specific skill bot is selected, the user utterance is routed to that skill bot for further processing. Once a system intent is selected, one or more actions are performed by the master bot itself according to the selected system intent.

[0065] Figure 3 is a simplified block diagram of a master bot (MB) system 300 according to one embodiment. The MB system 300 may be implemented using software only, hardware only, or a combination of hardware and software. The MB system 300 includes a preprocessing subsystem 310, a multiple intent subsystem (MIS) 320, an explicit invocation subsystem (EIS) 330, a skillbot caller 340, and a data store 350. The MB system 300 shown in Figure 3 is merely one example of the arrangement of components within the master bot. Those skilled in the art will recognize many possible variations, alternatives, and modifications. For example, in some implementations, the MB system 300 may have more or fewer systems or components than those shown in Figure 3, may combine two or more subsystems, or may have different configurations or arrangements of subsystems.

[0066] The preprocessing subsystem 310 receives the utterance "A" 302 from the user and processes the utterance through the language detector 312 and the language parser 314. As described above, the utterance can be provided in various ways, including speech or text. The utterance 302 may be a sentence fragment, a complete sentence, or multiple sentences. The utterance 302 may contain punctuation. For example, if the utterance 302 is provided as speech, the preprocessing subsystem 310 may convert the speech to text using a speech-to-text converter (not shown) that inserts punctuation, such as commas, semicolons, periods, etc., into the resulting text.

[0067] The language detector 312 detects the language of utterance 302 based on the text of utterance 302. Since each language has its own grammar and semantics, the processing of utterance 302 is language-dependent. Differences between languages ​​are taken into consideration when analyzing the syntactic and structural aspects of the utterance.

[0068] The language parser 314 syntactically analyzes the utterance 302 and extracts part-of-speech (POS) tags for individual linguistic units (e.g., words) within the utterance 302. POS tags include, for example, nouns (NN), pronouns (PN), and verbs (VB). The language parser 314 may also tokenize the linguistic units of the utterance 302 (e.g., converting each word into an independent token) and lemmatize the words. A lemma is the principal form of a set of words represented in a dictionary (e.g., "run" is a lemma of run, runs, ran, running, etc.). Other types of preprocessing that the language parser 314 can perform include chunking compound expressions, for example, combining "credit" and "card" into a single expression "credit_card". The language parser 314 may also identify relationships between words within the utterance 302. For example, in some embodiments, the language parser 314 generates a dependency tree that indicates which parts of the utterance (e.g., a specific noun) are direct objects and which parts are prepositions. The results of the processing performed by the language parser 314 form extracted information 305, which is provided to the MIS 320 as input along with the utterance 302 itself.

[0069] As mentioned earlier, utterance 302 may contain multiple sentences. To detect multiple intents and explicit calls, utterance 302 can be treated as a single unit even if it contains multiple sentences. However, in some embodiments, preprocessing may be performed, for example by the preprocessing subsystem 310, to identify a single sentence from among multiple sentences for multiple intent analysis and explicit call analysis. Generally, the results produced by MIS320 and EIS330 are substantially the same regardless of whether utterance 302 is processed at the level of individual sentences or as a single unit containing multiple sentences.

[0070] MIS320 determines whether utterance 302 represents multiple intents. While MIS320 can detect the presence of multiple intents in utterance 302, the processing performed by MIS320 does not involve determining whether the intents in utterance 302 match any intent configured for the bot. Instead, the processing to determine whether the intents in utterance 302 match a bot intent (as shown, for example, in the embodiment of Figure 3) may be performed by the intent classifier 342 of the MB system 300 or by the skill bot's intent classifier. The processing performed by MIS320 assumes the existence of a bot capable of processing utterance 302 (e.g., a specific skill bot or the master bot itself). Therefore, the processing performed by MIS320 does not require knowledge of what bots are in the chatbot system (e.g., the IDs of skill bots registered with the master bot) or what intents are configured for a particular bot.

[0071] To determine that utterance 302 contains multiple intents, MIS320 applies one or more rules from a set of rules 352 in datastore 350. The rules applied to utterance 302 depend on the language of utterance 302 and may include sentence patterns indicating the presence of multiple intents. For example, a sentence pattern may include a coordinating conjunction that connects two parts of a sentence (e.g., coordinating terms), where these two parts correspond to independent intents. If utterance 302 matches this sentence pattern, it can be inferred that utterance 302 represents multiple intents. Note that utterances with multiple intents do not necessarily have different intents (e.g., intents directed to different bots or different intents within the same bot). Instead, an utterance may have separate instances of the same intent, for example, "Place a pizza order using payment account X, then place a pizza order using payment account Y."

[0072] As part of its determination that utterance 302 represents multiple intents, MIS320 also determines which part of utterance 302 is associated with each intent. As shown in Figure 3, for each intent represented in an utterance containing multiple intents, MIS320 constructs a new utterance for separate processing, for example, utterance "B" 306 and utterance "C" 308, instead of the original utterance. Thus, the original utterance 302 can be divided into two or more separate utterances, which are processed one at a time. Using the extracted information 305 and / or from an analysis of utterance 302 itself, MIS320 determines which of the two or more utterances should be processed first. For example, MIS320 may determine that utterance 302 contains a marker word indicating that a particular intent should be processed first. The newly formed utterance corresponding to this particular intent (for example, one of utterances 306 or 308) will be sent first for further processing by EIS330. After the conversation triggered by the first utterance has ended (or been temporarily interrupted), the next highest priority utterance (for example, utterance 306 or utterance 308, the other of the two) may be sent to the EIS330 for processing.

[0073] The EIS330 determines whether an received utterance (e.g., utterance 306 or utterance 308) contains a skillbot invocation name. In one embodiment, each skillbot in the chatbot system is assigned a unique invocation name that distinguishes it from other skillbots in the chatbot system. A list of invocation names can be managed in the data store 350 as part of the skillbot information 354. When an utterance contains a word that matches an invocation name, the utterance is considered an explicit invocation. If the bot is not explicitly invoked, the utterance received by the EIS330 is considered an implicit invocation utterance 334 and is fed into the master bot's intent classifier (e.g., intent classifier 342) to determine which bot should be used to process the utterance. In some examples, the intent classifier 342 determines that the master bot should process the implicit invocation utterance. In other examples, the intent classifier 342 determines which skillbot to route the utterance to for processing.

[0074] The explicit call functionality provided by EIS330 offers several advantages. It can reduce the amount of processing the master bot must perform. For example, with explicit calls, the master bot does not need to perform intent classification analysis (e.g., using intent classifier 342), or it only needs to perform a reduced intent classification analysis to select a skill bot. Thus, explicit call analysis can enable the selection of a specific skill bot without relying on intent classification analysis.

[0075] Furthermore, situations can arise where functionality overlaps among multiple skill bots. This can occur, for example, when the intents handled by two skill bots overlap or are very similar to each other. In such situations, it may be difficult for the master bot to determine which of the multiple skill bots to select based solely on intent classification analysis. In such scenarios, explicit invocation resolves the ambiguity regarding the specific skill bot to be used.

[0076] In addition to determining that an utterance is an explicit invocation, the EIS330 also plays a role in determining whether any part of an utterance should be used as input to a skill bot that is explicitly being called. Specifically, the EIS330 can determine whether any part of an utterance is not associated with an invocation. The EIS330 can make this determination through analysis of the utterance and / or analysis of the extracted information 305. Instead of sending the entire utterance that the EIS330 receives, it can send the part of the utterance that is not associated with an invocation to the called skill bot. In some examples, the input to the called skill bot is formed by removing any part of the utterance that is associated with the invocation. For example, "I want to order pizza using Pizza Bot" can be shortened to "I want to order pizza" because "using Pizza Bot" is related to the invocation of Pizza Bot but irrelevant to the process that Pizza Bot performs. In some examples, the EIS330 may reformat the part to be sent to the called bot, for example, to form a complete sentence. Therefore, the EIS330 not only determines that there is an explicit call, but also determines what should be sent to the skillbot when there is an explicit call. In some examples, there may be no text to input to the called bot. For example, if the utterance is "pizzabot", the EIS330 can determine that pizzabot is being called, but there is no text for pizzabot to process. In such a scenario, the EIS330 can indicate to the skillbot caller 340 that there is nothing to send.

[0077] The skillbot caller 340 invokes skillbots in various ways. For example, the skillbot caller 340 may invoke a bot in response to receiving an indication 335 that a particular skillbot has been selected as a result of an explicit invocation. The indication 335 may be sent by the EIS 330 along with input about the explicitly invoked skillbot. In this scenario, the skillhot caller 340 takes over control of the conversation to the explicitly invoked skillhot. The explicitly invoked skillhot determines an appropriate response to the input from the EIS 330 by treating the input as an independent utterance. For example, the response may be to perform a specific action or to start a new conversation in a specific state, the initial state of the new conversation depending on the input sent from the EIS 330.

[0078] Another way in which the SkillBot caller 340 can invoke a SkillBot is through implicit invocation using the Intent Classifier 342. The Intent Classifier 342 is trained using machine learning and / or rule-based training techniques to determine the likelihood that an utterance represents a task configured to be performed by a particular SkillBot. The Intent Classifier 342 is trained in different classes, one class for each SkillBot. For example, whenever a new SkillBot is registered with the MasterBot, the Intent Classifier 342 can be trained using a list of utterance examples associated with the new SkillBot to determine the likelihood that a particular utterance represents a task that the new SkillBot can perform. The parameters generated as a result of this training (e.g., a set of parameter values ​​for a machine learning model) can be stored as part of the SkillBot information 354.

[0079] In one embodiment, the intent classifier 342 is implemented using a machine learning model, as will be described in more detail herein. Training the machine learning model may involve inputting at least a subset of utterances from a variety of utterance examples associated with different skill bots, in order to generate an inference as output of the machine learning model about which bot is the correct bot to process any particular training utterance. For each training utterance, a representation of the correct bot to use for the training utterance may be provided as ground truth information. The behavior of the machine learning model can then be adapted (e.g., through backpropagation) to minimize the difference between the generated inference and the ground truth information.

[0080] In one embodiment, the intent classifier 342 determines a confidence score for each skill bot registered with the master bot, indicating the likelihood that the skill bot can process an utterance (e.g., an implicit calling utterance 334 received from the EIS 330). The intent classifier 342 may also determine a confidence score for each configured system-level intent (e.g., Help, Exit). If a particular confidence score satisfies one or more conditions, the skill hot caller 340 invokes the bot associated with that particular confidence score. For example, a threshold confidence score value may need to be met. Thus, the output 345 of the intent classifier 342 is either an identification of a system intent or an identification of a particular hot skill. In some embodiments, in addition to meeting a threshold confidence score value, the confidence score must exceed the next highest confidence score by a certain win margin. Imposing such conditions allows routing to a particular skill bot when each of the confidence scores of multiple skill bots exceeds a threshold confidence score value.

[0081] After identifying a bot based on the confidence score evaluation, the skillbot caller 340 hands over processing to the identified bot. In the case of a system intent, the identified bot is the master bot. Otherwise, the identified bot is the skillbot. Furthermore, the skillbot caller 340 determines what should be provided as input 347 to the identified bot. As previously mentioned, in the case of an explicit call, input 347 may be based on a portion of an utterance not associated with the call, or input 347 may be absent (e.g., an empty string). In the case of an implicit call, input 347 may be the entire utterance.

[0082] The data store 350 includes one or more computing devices that store data used by various subsystems of the master bot system 300. As previously mentioned, the data store 350 includes rules 352 and skillbot information 354. Rules 352 include, for example, rules for MIS 320 to determine when an utterance represents multiple intents and how to divide an utterance that represents multiple intents. Rules 352 further include rules for EIS 330 to determine which part of an utterance that explicitly invokes a skillbot should be sent to the skillbot. Skillbot information 354 includes the invocation names of skillbots in the chatbot system, for example, a list of the invocation names of all skillbots registered with a particular master bot. Skillbot information 354 may also include information used by the intent classifier 342 to determine the confidence score of each skillbot in the chatbot system, for example, parameters of a machine learning model.

[0083] Figure 4 is a simplified block diagram of a skillbot system 400 according to one embodiment. The skillbot system 400 is a computing system that can be implemented with software only, hardware only, or a combination of hardware and software. In one embodiment, such as the embodiment shown in Figure 2, the skillbot system 400 can be used to implement one or more skillbots within a digital assistant.

[0084] The skillbot system 400 includes an MIS 410, an intent classifier 420, and a conversation manager 430. The MIS 410 is similar to the MIS 320 in Figure 3 and provides similar functionality, including being able to operate using rules 452 in the data store 450 to determine whether an utterance represents multiple intents, and if so, how to split the utterance into separate utterances for each of the multiple intents. In one embodiment, the rules applied by the MIS 410 to detect multiple intents and split the utterance are identical to the rules applied by the MIS 220. The MIS 410 receives an utterance 402 and extracted information 404. The extracted information 404 is similar to the extracted information 205 in Figure 1 and is a language parser. 3 It can be generated using the local language parser of the 14 or Skillbot system 400.

[0085] The intent classifier 420 can be trained in a similar manner to the intent classifier 342, which was previously described in relation to the embodiment of Figure 3 and is described in more detail herein. For example, in one embodiment, the intent classifier 420 is implemented using a machine learning model. The machine learning model of the intent classifier 420 is trained for a particular skill bot using at least a subset of utterance examples associated with a particular skill bot as training utterances. The ground truth for each training utterance is the specific bot intent associated with the training utterance.

[0086] Utterance 402 can be received directly from the user or provided via a master bot. If utterance 402 is provided via a master bot, for example, as a result of processing through MIS320 and EIS330 in the embodiment shown in Figure 3, MIS410 can be bypassed, avoiding repetition of processing already performed by MIS320. However, if utterance 402 is received directly from the user, for example during a conversation that occurs after routing to a skill bot, MIS410 can process utterance 402 and determine whether utterance 402 represents multiple intents. In that case, MIS410 applies one or more rules to split utterance 402 into separate utterances, for example, utterance "D" 406 and utterance "E" 408, for each intent. If utterance 402 does not represent multiple intents, MIS410 forwards utterance 402 to the intent classifier 420 without splitting it for intent classification.

[0087] The intent classifier 420 is configured to match an received utterance (e.g., utterance 406 or 408) with an intent associated with the skill hot system 400. As previously described, a skill hot can consist of one or more intents, each intent including at least one example utterance associated with that intent and used to train the classifier. In the embodiment shown in Figure 2, the intent classifier 242 of the master bot system 200 is trained to determine the confidence score of individual skill bots and the confidence score of system intents. Similarly, the intent classifier 420 can be trained to determine the confidence score of each intent associated with the skill bot system 400. The classification performed by the intent classifier 342 is at the bot level, while the classification performed by the intent classifier 420 is at the intent level and therefore less granular. The intent classifier 420 has access to intent information 454. The intent information 454 includes, for each intent associated with the skillbot system 400, a list of utterances that represent and explain the meaning of the intent and are typically associated with tasks that can be performed by that intent. The intent information 454 may further include parameters that are generated as a result of training on this list of utterances.

[0088] The conversation manager 430 receives, as output from the intent classifier 420, an indication 422 of a specific intent identified by the intent classifier 420 as the best match to the utterance input to the intent classifier 420. In some cases, the intent classifier 420 may not be able to determine any match. For example, the confidence score calculated by the intent classifier 420 may fall below a threshold confidence score value if the utterance is directed to a system intent or an intent of a different skillbot. When this occurs, the skillbot system 400 directs the utterance to the master bot for processing, for example, to route it to a different skillbot. However, if the intent classifier 420 successfully identifies an intent within the skillbot, the conversation manager 430 initiates a conversation with the user.

[0089] A conversation initiated by the conversation manager 430 is a conversation specific to an intent identified by the intent classifier 420. For example, the conversation manager 430 may be implemented using a state machine configured to execute a dialogue flow for the identified intent. The state machine may include a default initiation state (e.g., when the intent is invoked without any additional input) and one or more additional states, each state associated with an action performed by the skillbot (e.g., executing a purchase transaction) and / or a dialogue (e.g., a question, response) presented to the user. Thus, upon receiving an instruction 422 that identifies an intent, the conversation manager 430 can determine an action / dialogue 435 and may determine additional actions or dialogues in response to subsequent utterances received during the conversation.

[0090] The data store 450 includes one or more computing devices that store data used by various subsystems of the skillbot system 400. As shown in Figure 4, the data store 450 includes rule 452 and intent information 454. In one embodiment, the data store 450 can be integrated with the data store of the master bot or digital assistant, for example, the data store 250 in Figure 2.

[0091] Routing behavior of digital assistants and skill bots When a user inputs a phrase into a digital assistant (or master bot), the digital assistant determines how to route the conversation to a given skill, to a different state of the current flow, or to the digital assistant's built-in intents, as described in relation to Figures 2–4. At the heart of the routing model is a confidence score, which is calculated for each individual skill and intent to measure how well they match the user's input (e.g., providing information about the confidence of the prediction). The confidence score is derived by applying the underlying natural language processing (NLP) algorithms to the skill and digital assistant's input data (e.g., utterances). The confidence score may be derived using any confidence measurement technique or estimator (e.g., Bayesian models, distance-based loss, adversarial training, etc.). For example, in classification, the confidence of individual predictions may already be estimated as part of the classification. Intuitive estimation approaches use inter-class uncertainty, represented by posterior probabilities or distance to a separating hyperplane, to assess the different properties of individual predictions. In contrast, confidence estimators for regression can leverage the properties of the training data or the machine learning model. Routing decisions are then made by measuring confidence scores for various routing parameters, such as candidate skill confidence thresholds and confidence win margins, and applying one or more rules to the routing as desired.

[0092] For example, regarding intent resolution, a machine learning model associated with a skill may evaluate the user message and return confidence scores for the top-ranking label (intent) and what the model judges to be the runner-up. In a conversational context (e.g., conversational AI), the top-ranking label is resolved as the intent to initiate a conversation with the user. Therefore, based on model training and user messages, one example might be that the model has an 80% confidence that intent A is a good match, a 60% confidence for intent B, and a 45% confidence for intent C. In this case, the model is likely a very good fit if the user desires intent A. However, even if the confidence level of the highest-scoring label is only 30%, is this what the user wants? The developer could either risk the model to follow this intent, or rather play it safe and assume the model cannot predict what the user wants (the unresolved intent) and / or display a message to the user to rephrase the request. To make it easier for the intent model to determine which intents match the user utterance, digital assistants use a setting called a confidence threshold in the conversational context. The intent model evaluates the user utterance for all intents and assigns a confidence score to each intent. The confidence threshold is a range of possible confidence scores that indicates the line below which the intent is considered to have no correspondence with the utterance, and above which the intent is considered a candidate intent to start a conversation. While this example is specific to intent resolution, it should be understood that confidence scores can be derived and used to resolve various types of inferences by digital assistants, such as which skill to use to process a message / utterance.

[0093] The routing model consists of three layers: determining candidate system intents (e.g., using a candidate system intent model), determining candidate skills (e.g., using a candidate skill model), and determining candidate flows (e.g., using a candidate flow model). Regarding the determination of candidate system intents, user input is evaluated, and a confidence score is applied to the digital assistant's intents (exit, help, and unresolved intents). Any of these intents with a confidence score exceeding the value of the digital assistant's built-in system intent confidence threshold routing parameter are treated as candidates for further evaluation. Regarding the determination of candidate skills, user input is evaluated, and a confidence score is applied to each skill. Any skill with a confidence score exceeding the value of the digital assistant's candidate skill confidence threshold routing parameter are treated as candidates for further evaluation. Regarding the determination of candidate flows, after candidate skills have been identified, each intent in those skills is evaluated (according to the intent model for each skill), and a confidence score is applied to each intent. Generally, any intent with a confidence score exceeding the value of the skill's confidence threshold routing parameter (not the digital assistant's candidate skill confidence threshold parameter) is treated as a candidate flow.

[0094] This routing behavior can be adjusted by tuning the routing parameters of the digital assistant. Depending on the configuration of the skills (and their intents) in the digital assistant, users may need to adjust the values ​​of the digital assistant's routing parameters to better control how the digital assistant responds to user input. Routing parameters may be configured to take values ​​from 0 (0% confidence) to 1 (100% confidence). The following is an overview of possible digital assistant routing parameters. • Built-in System Intent Confidence Threshold: The minimum confidence score required to match built-in system intents, such as Help and Exit. The default value may be, for example, 0.9 (90% confidence). • Candidate Skill Confidence Threshold: The minimum confidence score required to match a candidate skill. The default value may be, for example, 0.4 (40% confidence). • Confidence Win Margin: The maximum difference between the confidence score of the top-ranked candidate skill and the confidence score of the lower-ranked candidate skill (which also exceeds the confidence threshold), which should be considered for lower-ranked candidate skills. Built-in digital assistant intents (Help, Exit, and unresolvedIntent) may also be considered. The default value may be, for example, 0.1 (10% confidence). There is a separate confidence range parameter for skills that functions similarly, except that it applies to the confidence scores of intents within the skill. • Consider all thresholds: A minimum confidence score is required to consider all matching intents and flows. This value also takes precedence over the win margin. (If such a high confidence score is given, there is no way to be sure which flow the user wants to use.) The default value may be, for example, 0.8 (80% confidence). • Consider only the current context threshold: The minimum confidence score required when considering only the current skill. If user input matches an intent that exceeds this threshold, other intents will not be considered, even if they meet the confidence threshold. This setting helps prevent unambiguation prompts for user input that sufficiently matches intents from multiple skills. For example, the user input "cancel order" may sufficiently match intents from multiple food delivery skills. The default value may be, for example, 0.8 (80% confidence). • Explicit Call Confidence Threshold: The minimum confidence score required to match inputs that include explicit calls to skills. The default value may be, for example, 0.8 (80% confidence). • Exit Prompt Confidence Threshold: The minimum confidence score required for a user to exit without prompting for confirmation. The default value may be, for example, 1.01, which is nominally set outside the confidence threshold range of 0-1 and ensures that a confirmation prompt is always displayed. However, if the skill designer wants users to be able to exit without a confirmation prompt when their confidence score for exiting is high, this threshold may be lowered.

[0095] In addition to the digital assistant routing parameters, routing behavior can be further adjusted by tuning the skill routing parameters. The following is an overview of the possible skill routing parameters: • Confidence Threshold: The minimum confidence score required to match a skill's intent with user input. If they do not match, the transition action is set to an unresolved intent. The default value may be, for example, 0.7 (70% confidence). • Margin Confidence: Only top intents that exceed the confidence threshold are selected if they are the highest-ranked intents that exceed the confidence threshold. Other intents that exceed the confidence threshold are also presented to the user if their scores are within the score of the top intent but less than the win margin. The default value may be, for example, 0.1 (10% confidence).

[0096] In addition, rules may be generated for various cases that affect the routing formula. For example, rules may include explicit invocation, context-aware routing, and context pinning. Regarding explicit invocation, if a user includes a skill invocation name in their input, the digital assistant will route directly to that skill, even if the input is also sufficiently similar to other skills. Regarding context-aware routing, if a user is already engaged with a skill, that skill will be given more weight during intent resolution than intents from other skills. Regarding context pinning, if user input includes an explicit invocation to a skill but does not include intent-related utterances, the router will "pin" the conversation to the skill. This means that the next utterance will be assumed to be related to that skill.

[0097] An explicit invocation occurs when a user enters a skill invocation name as part of their input. Using explicit invocations makes it easier for users to ensure their input is immediately routed to the intended skill, thus reducing the number of interactions with the digital assistant required to accomplish their task. When explicit invocations are used, the corresponding skill is given additional weight when determining routing. If a user is not yet in a skill and enters an explicit invocation, that invocation takes precedence over other flows within the digital assistant context. If a user is in a flow of different skills, the digital assistant will always try to confirm that the user actually wants to switch skills. In each digital assistant, a user, such as a skill developer, can determine the invocation name they want to use for a given skill. Users may also set the invocation name on the skill's digital assistant configuration page. This behavior may be supported by an explicit invocation confidence threshold routing parameter. If the confidence score for an explicit invocation exceeds its threshold, intents from other skills are not considered in the routing decision. In some cases, the default value of this threshold is set to 0.8 (80% confidence).

[0098] Routing in digital assistants is also context-aware, meaning that matching intents from the skill the user is currently engaged with are given more weight than intents from other skills during intent resolution. For example, suppose a digital assistant has both a banking skill and an online retailer skill. If the user types the question "What's my balance?", this could refer to both the user's bank account balance and the balance remaining on a gift card registered with an online retailer. If the user, such as a customer, types this question before entering context for either skill, the digital assistant must give the user a choice of which "balance" flow to enter (either the banking skill or the retailer skill). However, if the user types this question from within the banking skill, the digital assistant should automatically select the "balance" flow corresponding to the banking skill (and discard intents from other skills, even if they meet the standard confidence threshold routing parameter). Context awareness is supported by the "Consider current context threshold only" routing parameter. If the confidence score for an intent in the current context exceeds that threshold, intents from other contexts are not considered in routing decisions. In some cases, users would likely want to be certain that an intent in the current context is the correct intent before excluding the display of other intents outside the current context, so the default value for this threshold is 0.8 (80% confidence).

[0099] In addition to intents from the skill a user is currently engaged with, there are intents common to multiple or all skills, such as help or unresolved intents. For example, within the context of a skill, if user input matches a help system intent, the user is routed to the help flow determined by that skill (not to a flow determined at the digital intent level). More specifically, if a user engages with skill and type help, help is provided for that skill, not help from the digital assistant as a whole. The behavior is different for unresolved intent system intents. If user input resolves an unresolved intent (and there are no other matching intents in the skill), the input is treated as an unresolved intent at the digital assistant level. However, if the unresolved intent is only one of the matching intents within the skill, the skill processes the response. This behavior is supported by the "built-in system intent confidence threshold" routing parameter. If the confidence score for one of these intents exceeds that threshold, that intent is treated as a candidate for further evaluation. In some cases, the default value for this threshold is 0.9 (90% confidence).

[0100] As described herein, giving greater weight to intents from a skill a user is currently involved with than intents from other skills during intent resolution can lead to inappropriate routing, particularly for out-of-situation (OOS) utterances. It is important that the chatbot system can identify such OOS utterances so that it can take appropriate response actions. Therefore, according to aspects of this disclosure, context awareness may also take skill groups into consideration. This means that when a skill is defined as part of a skill group and that skill is in the current context, the current context also includes other skills in that skill group. As described herein, with respect to a skill domain encompassing many functionalities, it is often desirable to divide those functionalities into multiple specialized skills. This is particularly useful from a development perspective. Different teams can work on different aspects of a functionality and publish the skills on the timeline and their updates that are best suited to them. When there are multiple skills within a domain, users such as customers are likely to need to switch between those skills relatively frequently. For example, in a single session with a digital assistant that includes several HR-related skills, a user may be able to make requests regarding skills related to compensation, personal information, and vacation. To optimize routing behavior between related skills, users such as skill developers can define skill groups. Within a digital assistant, all skills within a group are treated as a single logical skill. As a result, all skills within a group are considered part of the current context, and therefore all of their intents are given equal weight during intent resolution.

[0101] When skill groups are defined for a digital assistant, the routing engine tracks both the skill context and the group context. The routing engine switches the skill context within a group if it determines that another skill within the group is better suited to handling the user request. This determination is based on the ranking of the group's skills in the candidate skill model. In some examples, if the confidence score of the group's top candidate skill is less than 5% higher than the confidence score of the current skill, the skill context within the group does not change.

[0102] To define the scope of a skill group, each skill group must be defined as a collection of skills within the same domain that share a linguistic kinship. Skills within a group must be divided by function. For example, it might make sense to group skills related to profit, compensation, absenteeism, personal information, and employment into an HCM skill group. Skills related to opportunities and accounts might belong to a sales skill group. Skills can be associated with a group using a group identifier, such as the group name or label. In some cases, to organize skill groups and prevent naming conflicts, users may use the <company name>.<domain> pattern for skill group names. For example, a user might define the following HCM skills for a hypothetical company, Acme: 〇Profit Rewards Absence Personal information 〇 Employment For this purpose, you may create a group called Acme.hcm. Similarly, hypothetically, Acme has the following skills within the sales domain. 〇 Opportunity 〇 Account If also available, users can also use acme.sales as a skill group.

[0103] When a developer wants to develop a digital assistant with common skills related to features such as handling help or small talk, they may not want to treat the common skills as a separate group of skills, as they may be called upon at any time in a conversation, regardless of the group of skills the user is primarily interacting with. Once called upon, the developer may want to ensure that the user is not dragged into these common skills. To ensure that skills in other groups are given the same weight as the common skills after the interaction with the common skills has ended, the developer may include the common skills in a group of multiple groups. The developer can do this by including an asterisk (*) or a wildcard in the group name of the common skills. For example, if the developer uses acme.* as the skill group name, it will include all skills in the acme.hcm and acme.sales groups, but not any skills in a group called hooli.hcm. Alternatively, if the developer uses * as the skill group name, it will include all groups (but not skills that are not assigned to a group). When a user navigates from a simple group (a group without an asterisk in its name) to a skill whose group name contains an asterisk or wildcard, the group context remains the same as the group context before navigating to that skill. For example, if a user's conversation moves from a skill in the acme.hcm group to a skill in the acme.* group, the group context remains acme.hcm.

[0104] Continuing with the user-created group example above, the following is an example of context awareness and routing within and between skill groups. The user asks, "What benefits can I receive?" The skill context is Benefits Skill and the group context is acme.hcm. Next, the user asks, "How much will my salary be?" The skill context is changed to Compensation, and the group context remains acme.hcm. The user's current context is profit skills, which means the user's current group context is acme.hcm. The user asks, "What sales opportunities are there?" This request is out of domain for all skills within the HCM group, not just the current skill (although "opportunities" may offer a potential match for employment skills). The user is routed to "opportunities," which is the best match within the acme.sales group context. When a user types "What benefits do I get?", this calls the benefits skills section of the acme.hcm group. The user's context is the benefits skill and the acme.hcm group. When a user types "Tell me a joke," this invokes the generic chat skill assigned to the acme.* group. The user is in the chat skill context at this time. The group context is any group that matches acme.*. This includes both acme.hcm (which contains the previously called benefit skill) and acme.sales, which consists of opportunity skills and account skills. The user asks, "What benefit do I gain from this?" followed by, "I have another question." The user is still in the acme.hcm context because of a question about benefits, but is now being routed to a misc.another.question intent in the "Other" skill, which is a member of the acme.* group. When a user navigates to a skill belonging to a group name containing an asterisk (*), the user group context remains in the same location (acme.hcm in this example) before being routed to the skill belonging to the * group. The user is currently in the context of a skill called "Other," which provides common functionality. This belongs to the acme.* group, meaning the user's current group context is all acme groups (acme.sales and acme.hcm). The current skill context is "Other." The user enters "What benefits can I receive?". The current skill context changes to "Benefits," which belongs to the acme.hcm group.

[0105] As shown in Figure 5A, to define a skill group for a skill, a user such as a skill developer can navigate the graphic user interface 500 on the skill configuration page and enter a group name 505 in the group field 510 for skill 515. When skill 515 is added to the digital assistant, any other skills in the digital assistant that have the same group name 505 are considered part of the same skill group. If a skill has already been added to the digital assistant, the user can set a skill group in the digital assistant (or disable the group specified in the skill configuration). As shown in Figure 5B, to set a skill group in the digital assistant, the user can navigate the graphic user interface 520 on the digital assistant configuration page and enter a group name 525 in the group field 530 for skill 535.

[0106] Routing in digital assistants can also be based on context pinning. If user input includes an explicit invocation of a skill but does not include intent-related utterances, the router "pins" the conversation to the skill of the next user input. This means that the next utterance is assumed to be related to that skill, and therefore the router does not consider any intents from different skills. If the user then inputs something unrelated to that skill, the router treats it as an unresolved intent within that skill, even if it matches an intent from a different skill well enough. (Exit intents are an exception; these are always considered.) The pin is then removed. Therefore, if the user repeats that input or inputs something unrelated to the pinned context, the entire flow is considered again.

[0107] We will consider an example of how context-pinning based routing works when users behave as expected. The user enters "Let's go to Pizza Skill," which is an explicit invocation of the Pizza Skill. (Including the skill name in the utterance makes it an explicit invocation.) At this point, the conversation is pinned to pizza skills, which means the digital assistant will only look for matches related to pizza skills. Next, she types "I want to order". The digital assistant detects a match with the OrderPizza intent within the pizza skill and initiates the process to order a pizza. At this point, the pin is removed. The following is an example of how context-pinning based routing works when the user proceeds in a way that deviates slightly from what was expected. The user enters "Let's go to Pizza Skill," which is an explicit call to Pizza Skill. At this point, the conversation is fixed on pizza skills. Next, she enters "Send Money". Since this input does not match anything within the Pizza skill, the router treats it as an unresolved intent within the Pizza skill (and prompts the user for clarification, depending on how the flow for unresolved intents is designed). Intents from other skills (such as the Financial skill) are ignored, even if they provide a suitable match. The pin from Pizza Skill will be removed. She keeps repeating the request for the money transfer. A match is found in the financial skills, and the remittance flow is initiated.

[0108] Figure 6 shows an example 600 of a digital assistant that evaluates user input and routes the conversation to a specific flow. In this example, the user gives the input / utterance "order pizza," and the digital assistant's first response is "How old are you?" The response "How old are you?" from the digital assistant indicates the start of the OrderPizza flow for the pizza skill (which requires the user ordering the pizza to be 18 years of age or older). As shown in Figure 6, the digital assistant found a strong match (100%) for the pizza skill and a weak match (21.56%) for the retail skill. In the first layer 605, when determining candidate system intents, there were no matches for any system intent. In the second layer 610, when determining candidate skills, there was a strong match (100%) for the pizza skill and a weak match (21.56%) for the retail skill. Because the match for the pizza skill exceeded the candidate skill confidence threshold (40%), the digital assistant evaluated the flow for the pizza skill. As described herein, skill designers can adjust the candidate skill confidence threshold value in the configuration settings of the digital assistant. In the third layer 615, candidate flows were determined, and in the pizza skill, the digital assistant found one match for the intent to order a pizza. Since this match exceeded the confidence threshold for the flow in the pizza skill (and there were no other eligible matches to consider), the pizza ordering flow was initiated.

[0109] Figure 7 illustrates example 700, where the user needs to be prompted to clarify their intent. In this example, the user provides the input / utterance, "What is my balance?", and the digital assistant's initial response is, "Gift card balance, financial account balance, or something else?". As can be seen, the digital assistant provides a prompt asking the user to choose from several options because it does not know what the user wants (eliminating ambiguity). As shown in Figure 7, in determining candidate system intents in the first tier 705, there was no match for any of the system intents. In determining candidate skills in the second tier 710, both financial skills and retail skills candidate skills scored highly (100%). Subsequently, for each of these skills, in determining candidate flows in the third tier 615, the digital assistant identified candidate flows that also scored highly (also 100%). The gift card balance and balance candidate flows exceed the confidence threshold, and the difference between their scores is less than the confidence win margin value (10%), so the digital assistant prompts the user to select from among those intents.

[0110] Figure 8 illustrates an example 800 in which the application of rules affects routing behavior. In this example, the user provides the input / utterance "What is my balance?", and the digital assistant's initial response is "Gift card balance, financial account balance, or something else?". As can be understood, the digital assistant does not know what the user wants, so it provides a prompt asking the user to choose from several options (eliminating ambiguity). In response, the user requests "Financial account balance", and again the digital assistant responds with a prompt asking the user to choose from several options, including savings, checking accounts, and credit cards. In response, the user requests "Checking account", and the digital assistant responds with the user's checking account balance. Subsequently, the user requests "Gift card balance check", and the digital assistant responds with "Gift card number?". In this case, the user has started checking their financial account balance in the finance skill using the digital assistant, but then decides to request their gift card balance in the retail skill. Because she uses an explicit invocation (which is also a gift card or retail skill and is called by its invocation name set on the page for the skill within the digital assistant), the digital assistant prioritizes the retail skill when attempting to resolve the intent, even though the user is in the context of a financial skill. The routing rule that applies here is explicit invocation, and if the user includes the skill's invocation name in its input, the digital assistant will route directly to that skill, even if the input also matches well with other skills. As shown in Figure 8, there is a match for the current context 805, but it is ignored. The match for the explicit invocation 810 for the retail skill's gift card balance wins (100%).

[0111] Figure 9 illustrates an example of how context awareness influences routing behavior. In this example, the user provides the input / utterance "What is my balance?", and the digital assistant's initial response is "Gift card balance, financial account balance, or something else?". As you can see, the digital assistant doesn't know what the user wants, so it provides a prompt asking the user to choose from several options (eliminating ambiguity). In response, the user requests "Financial account balance", and again the digital assistant responds with a prompt asking the user to choose from several options, including savings, checking accounts, and credit cards. In response, the user requests "Checking account", and the digital assistant responds with the user's checking account balance. Subsequently, the user requests "My balance", and the digital assistant again responds with several options, including savings, checking accounts, and credit cards. In this case, the user starts with the question "What is my balance and how much?", and through the prompts, eliminates the ambiguity between financial and retail skills, ultimately obtaining their checking account balance. They then enter "What is the balance?" again, but this time they do not need to navigate deambiguation prompts to distinguish between financial and retail skills. The information in the routing table shown in Figure 9 explains how the current context is set by the context-aware router and how several prompts for deambiguation are avoided. The current context rule that applies is "If the current context flow matches with very high confidence, other skill flows are ignored." Therefore, even if there is a matching intent from retail skill 905, it is ignored. The intent call section shows all matching intents, but the entry set by the context-aware router for "current context" 910, which contains only the financial skill balance intent 915, is obvious.

[0112] Figures 10A–10C illustrate an example 1000 in which context awareness considers skill groups and influences routing behavior. Firstly, Figure 10A shows the routing rule section 1005, which outputs routing rule 1010 when the skill context changes due to group stickiness. When the skill context changes due to skill grouping assignment, the original rule "Consider flows from all models" is changed to "Switch current context to the highest confidence level of the skill group when skill context changes" and "Current context flow matches with very high confidence. Ignore other skills." Secondly, Figure 10B shows that the routing intent call section and routing table are updated to display the group context 1015 assigned for the skill intent output, including the current context 1020 and candidate flow models 1025. In some examples, the group context can be different from the skill group (for example, when routing to a common skill belonging to multiple groups), so the group context 1015 is also shown in the "Current Context" column. Finally, Figure 10C shows the routing summary column 1030 in which the group context 1015 displayed after the flow 1035 to be routed is selected by the digital assistant.

[0113] It will be understood that the skills described herein can be equivalently represented as classifiers trained to specialize in processing certain types of natural language input, i.e., input relating to a specific domain or area of ​​the skill. Therefore, the candidate skill model described above functions as a classifier selection mechanism for identifying the classifier best suited to recognizing utterances and generating appropriate outputs, such as text or speech output. Thus, when deployed as described herein, the candidate skill model can provide an improved speech recognition tool by selecting a domain expert classifier (skill) specifically trained to analyze input speech or text within the domain recognized by the candidate skill model. In one embodiment, the utterance is an audio format, such as an audio file containing speech, so that the candidate skill model can act to improve speech recognition by better mapping speech input to text output, and the text output is a text response provided by the skill (classifier) ​​selected by the candidate skill model. Similar advantages can be achieved with text input.

[0114] Exemplary use cases and technologies for context-aware routing of skill groups For the following use case, if Entity_1 in the digital assistant configuration page has skills in the HCM, sales, and common domains, we assume that the skill developer can specify each group to which the skill may belong as follows: • HCM domain skills include the following skills 〇Profit Rewards Absence Personal information 〇 Employment Specify the group label "Entity_1.hcm" for this. • Sales domain skills include the following skills 〇 Opportunity 〇 Account Specify the group label "Entity_1.sales" for this. • Some common skills across all Entity_1 groups are the following skills System 〇ChitChat It has the label (Entity_1.*). If "*" is specified, the skill is part of all groups.

[0115] In the first use case, the user is in a profit skill and provides the input / utterance "How much was my salary last year?" (i.e., a reward skill input), and the digital assistant's context-aware router sets the current skill context to "Profit" and the group context to "Entity_1.hcm". In the first layer, candidate system intent determination, the digital assistant's candidate system intent model does not determine any matches for any system intents. In the second layer, candidate skill determination, the digital assistant's candidate skill model ranks reward skills higher than profit skills. In response, the digital assistant's context-aware router sets the current skill context to "Reward" and maintains the group context as "Entity_1.hcm". Subsequently, for the candidate skill, in the third layer, candidate flow determination, the candidate flow model identifies candidate flows (i.e., intents) in the current skill context and high-scoring group contexts, i.e., flows for obtaining a salary.

[0116] In the second use case, the user is in a profit skill and provides the input / utterance "What sales opportunities are there?" (i.e., opportunity skill input), and the digital assistant's context-aware router sets the current skill context to "Profit" and the group context to "Entity_1.hcm". In the first layer candidate system intent determination, the digital assistant's candidate system intent model does not determine any match for any system intent. In the second layer candidate skill determination, the digital assistant's candidate skill model identifies the opportunity utterance as out of domain (specifically, out of scope) for the profit skill and the group of skills within the "Entity_1.hcm" group. Once the current skill and group context are excluded from competition, the candidate skill model identifies the opportunity skill as the highest-ranking skill. In response, the digital assistant's context-aware router sets the current skill context to "Opportunity" and changes the group context to "Entity_1.sales". Subsequently, with respect to candidate skills, in the third layer of candidate flow determination, the candidate flow model identifies candidate flows (i.e., intents) within the current skill context and group context that score highly, i.e., flows that provide available sales opportunities.

[0117] In the third use case, the user is in a system skill and provides the input / utterance "What benefits can I receive?" (i.e., a benefits skill input), and the digital assistant's context-aware router sets the current skill context to "System" and the group context to "Entity_1.* = "Entity_1.hcm" and "Entity_1.sales". In the first layer candidate system intent determination, the digital assistant's candidate system intent model does not determine any matches for any system intents. In the second layer candidate skill determination, the digital assistant's candidate skill model ranks the benefits skill higher than any of the system skills. In response, the digital assistant's context-aware router sets the current skill context to "Benefit" and changes the group context to "Entity_1.hcm". Subsequently, for the candidate skill, in the third layer candidate flow determination, the candidate flow model identifies the candidate flows (i.e., intents) within the current skill context and group context that have received high scores, i.e., flows that provide eligible benefits.

[0118] In the fourth use case, the user is in the Benefit skill and provides the input / utterance, "Could I have ABC's phone number?" (ABC is an account name, but could match "Personal Information"), and the digital assistant's context-aware router sets the current skill context to "Benefit" and the group context to "Entity_1.hcm". In the first layer candidate system intent determination, the digital assistant's candidate system intent model does not determine any matches for any system intents. In the second layer candidate skill determination, the digital assistant's candidate skill model identifies the top two matches as account skills and personal information skills within the win margin parameter. The candidate skill model ranks the personal information skill higher to provide an opinion on the skills in the current group context, Entity_1.hcm, and the digital assistant's context-aware router maintains the group context as "Entity_1.hcm". Subsequently, in the third layer of candidate flow determination for candidate skills, the candidate flow model identifies and prompts the user for a disambiguation flow (i.e., intent), and distinguishes between account skills and personal information skills, assuming that both account skills and personal information skills are ranked within the win margin parameter.

[0119] In the fifth use case, the user is in the Personal Information skill and provides the input / utterance "Could I have ABC's phone number?" (ABC is an account name, but could match "Personal Information"), and the digital assistant's context-aware router sets the current skill context to "Personal Information" and the group context to "Entity_1.hcm". In the first layer candidate system intent determination, the digital assistant's candidate system intent model makes no matches for any system intents. In the second layer candidate skill determination, the digital assistant's candidate skill model identifies the top two matches as account skill and personal information skill within the win margin parameter. However, in this example, the personal information match is >0.8, and therefore exceeds the threshold parameter for current context only. The candidate skill model is ignored, and the digital assistant's context-aware router maintains the skill context as "Personal Information" and the group context as "Entity_1.hcm". Subsequently, regarding candidate skills, in the third layer of candidate flow determination, the candidate flow model identifies candidate flows (i.e., intents) within the current skill context and group context that have high scores, i.e., flows for providing a phone number under personal information. Note that in this example, the user does not obtain what they intended to obtain. The utterance "Could I have the phone number for your ABC account?" would have worked more appropriately.

[0120] In the sixth use case, the user is in the Personal Information skill and provides the input / utterance, "Could I have ABC's phone number?" (where ABC is a person's name). The digital assistant's context-aware router sets the current skill context to "Personal Information" and the group context to "Entity_1.hcm". In the first layer candidate system intent determination, the digital assistant's candidate system intent model determines no matches for any system intents. In the second layer candidate skill determination, the digital assistant's candidate skill model identifies the top two matches as account skill and personal information skill within the win margin parameter. Again, in this example, the personal information match is >0.8 and therefore exceeds the threshold parameter for the current context only. The candidate skill model is ignored, and the digital assistant's context-aware router maintains the skill context as "Personal Information" and the group context as "Entity_1.hcm". Subsequently, regarding candidate skills, in the third layer of candidate flow determination, the candidate flow model identifies candidate flows (i.e., intents) within the current skill context and group context that have high scores, i.e., flows for providing a phone number under personal information. In this case, it should be noted that the user will receive what they intended to receive.

[0121] Figure 11 shows process flows for context-aware routing using skill groups according to various embodiments. The processes shown in Figure 11 may be implemented by software (e.g., code, instructions, programs), hardware, or a combination thereof, executed by one or more processing units (e.g., processors, cores) of each system. The software may be stored on a non-temporary storage medium (e.g., on a memory device). The methods presented in Figure 11 and described below are illustrative and intended to be non-limiting. Figure 11 shows various processing steps occurring in a particular sequence or order, but this is not intended to be limiting. In some alternative embodiments, the steps may occur in several different orders, or some steps may occur in parallel. In some embodiments, such as those shown in Figures 2 to 4, the processes shown in Figure 11 may be executed by various subsystems, models, or modules to route conversations (e.g., digital assistant 206, master bot 214, and skill bots 216-1, 216-2, and 216-3, etc.). The conversation routing using the actions described in Figure 11 may work more efficiently, or otherwise more effectively, when routing OOS utterances.

[0122] In step 1105, the chatbot receives an utterance from the user within the chatbot session. The current skill context of the chatbot session may be the first skill, and the current group context of the chatbot session is the first group. The first group defines a domain space that includes the first set of skills, which includes the first and second skills. The first set of skills is associated with the first group by identifying or providing the name or label of the first group within the configuration page of each skill in the first set of skills.

[0123] In step 1110, the utterances are input into a candidate skill model for the first group. The candidate skill model may be a machine learning model trained to evaluate the utterances using one or more NLP algorithms and measure how well the skills in the first set of skills match the user's utterances.

[0124] In step 1115, the candidate skill model is used to obtain a ranking of skills within a first group that may process the utterance. The step of obtaining skill rankings includes evaluating the utterance and generating confidence scores for the skills within the first group (i.e., the first set of skills); identifying any skills with confidence scores exceeding the value of the candidate skill confidence threshold routing parameter as candidate skills for further evaluation; and ranking the candidate skills as skills within the first group that may process the utterance based on their confidence scores. In some examples, the candidate skill confidence threshold routing parameter is set to a confidence level of 40%.

[0125] In step 1120, based on the skill ranking, the second skill is determined to be the highest-ranked skill for speech processing and is treated as a candidate skill for further evaluation. It should be understood that additional skills may be treated as candidate skills for further evaluation. For example, if the win-margin parameter is configured for a digital assistant, a third skill (e.g., determined to be the second highest-ranked skill for speech processing) may be within the win-margin parameter and therefore may be treated as a candidate skill for further evaluation.

[0126] In step 1125, change the current skill context of the chatbot session to the second skill. Note that in examples where multiple skills are identified as candidate skills for further evaluation, the current skill context of the chatbot session may be changed to include the second skill and all other skills identified as candidate skills. In other words, the skill context does not have to be limited simply to the highest-ranked skill.

[0127] In step 1130, the utterance is input into a candidate flow model for the second skill. The candidate flow model may be a machine learning model that has been trained to evaluate the utterance using one or more NLP algorithms and measure how well the intent associated with the second skill matches the user's utterance. In some examples, the machine learning model is an intent classifier.

[0128] In step 1135, the candidate flow model is used to obtain a ranking of intents within a second skill that match the utterance. Obtaining skill rankings involves evaluating the utterance and generating confidence scores for skills within the first group, identifying any skills with confidence scores exceeding the value of the candidate skill confidence threshold routing parameter as candidate skills for further evaluation, and ranking the candidate skills as skills within the first group that may process the utterance based on their confidence scores. In some examples, the candidate skill confidence threshold routing parameter is set to a confidence level of 70%.

[0129] In step 1140, based on the intent ranking, the intent that is the highest-ranked intent for utterance processing is determined. It should be understood that further skills and intents may be treated as candidate skills and intents for further evaluation. For example, if the win margin parameter and confidence win margin are configured for the digital assistant and skills, a third skill (e.g., determined to be the second highest-ranked skill for utterance processing) and its associated intent may also be within the win margin parameter and confidence win margin, and therefore may also be treated as a candidate skill and intent for further evaluation.

[0130] In step 1145, the conversation flow with the user within the chatbot session is initiated based on the intent that is the highest-ranked intent for processing the utterance. If multiple intents are identified for further evaluation (e.g., based on confidence win margin), it should be understood that the flow may be a deambiguation flow that prompts the user to respond to clarify the user's utterances across multiple skills and / or intents.

[0131] Example system Figure 12 shows a simplified diagram of a distributed system 1200. In the example shown, the distributed system 1200 includes one or more client computing devices 1202, 1204, 1206, and 1208 connected to a server 1212 via one or more communication networks 1210. The client computing devices 1202, 1204, 1206, and 1208 may be configured to run one or more applications.

[0132] In various examples, server 1212 may be adapted to run one or more services or software applications that enable one or more embodiments described in this disclosure. In some examples, server 1212 may also provide other services or software applications that may include non-virtual and virtual environments. In some examples, these services may be provided to users of client computing devices 1202, 1204, 1206, and / or 1208 as web-based or cloud services, such as a Software as a Service (SaaS) model. Users operating client computing devices 1202, 1204, 1206, and / or 1208 can then interact with server 1212 and utilize the services provided by these components by using one or more client applications.

[0133] In the configuration shown in Figure 12, server 1212 may include one or more components 1218, 1220, and 1222 that implement the functions performed by server 1212. These components may include one or more processors, hardware components, or software components that can be executed by a combination thereof. It should be understood that a wide variety of system configurations different from the distributed system 1200 are possible. Therefore, the example shown in Figure 12 is just one example of a distributed system for realizing an example system and is not intended to be limiting.

[0134] A user may use client computing devices 1202, 1204, 1206, and / or 1208 to run one or more applications, models, or chatbots, which may then generate one or more events or models that can be realized or serviced in accordance with the teachings of this disclosure. A client device may provide an interface that allows a user of the client device to interact with the client device. A client device may also output information to the user through this interface. Although Figure 12 shows only four client computing devices, any number of client computing devices can be supported.

[0135] Client devices may include various types of computing systems, such as portable handheld devices, general-purpose computers like personal computers and laptops, workstation computers, wearable devices, game systems, thin clients, various messaging devices, sensors or other sensing devices. These computing devices may include various types and versions of software applications and operating systems (e.g., Microsoft Windows®, Apple Macintosh®, UNIX® or UNIX-like operating systems, Linux® or Linux-like operating systems, and various mobile operating systems, such as Google Chrome® OS, including Microsoft Windows Mobile®, iOS®, Windows Phone®, Android®, BlackBerry®, and Palm OS®). Portable handheld devices may include cellular phones, smartphones (e.g., iPhone®), tablets (e.g., iPad®), and personal digital assistants (PDAs). Wearable devices may include Google Glass® head-mounted displays and other devices. The game system may include various handheld game devices and internet-connected game devices (for example, Microsoft Xbox® game consoles with Kinect® gesture input devices, Sony PlayStation® systems, and various game systems provided by Nintendo®). Client devices may be capable of running a wide variety of applications, such as various internet-related applications and communication applications (for example, email applications and short message service (SMS) applications), and may use various communication protocols.

[0136] Network 1210 may be any type of network known to those skilled in the art that can support data communication using any of the various available protocols, including but not limited to TCP / IP (Transmission Control Protocol / Internet Protocol), SNA (System Network Architecture), IPX (Internet Packet Switching), AppleTalk®, etc. Just as an example, Network 1210 may include a local area network (LAN), an Ethernet®-based network, Token Ring, a wide area network (WAN), the Internet, a virtual network, a virtual private network (VPN), an intranet, an extranet, a public switched telephone network (PSTN), an infrared network, a wireless network (for example, a wireless network operating under any of the IEEE 1002.11 protocol suites, Bluetooth®, and / or any other wireless protocol), and / or any combination of these and / or other networks.

[0137] Server 1212 may consist of one or more general-purpose computers, dedicated server computers (including, for example, PC (personal computer) servers, UNIX® servers, midrange servers, mainframe computers, and rack-mount servers), server farms, server clusters, or other appropriate configurations and / or combinations. Server 1212 may include one or more virtual machines running a virtual operating system, or other computing architectures involving virtualization. This could be, for example, one or more flexible tools of logical storage that can be virtualized to maintain virtual storage for the server. In various examples, Server 1212 may be configured to run one or more services or software applications that provide the functions described above.

[0138] The computing system within Server 1712 may run one or more operating systems, including any of the above-mentioned operating systems, and any server operating system available on the market. Server 1712 may also run any of a variety of other server applications and / or middle-tier applications, including HTTP (Hypertext Transfer Protocol) servers, FTP (File Transfer Protocol) servers, CGI (Common Gateway Interface) servers, Java® servers, and database servers. Illustrated database servers include, but are not limited to, those available on the market from Oracle®, Microsoft®, Sybase®, IBM® (International Business Machines), and others.

[0139] In some implementation examples, server 1212 may include one or more applications for analyzing and integrating data feeds and / or event updates received from users of client computing devices 1202, 1204, 1206, and 1208. For example, data feeds and / or event updates may include, but are not limited to, Twitter® feeds, Facebook® updates, or real-time updates received from one or more third-party sources and continuous data streams, including real-time events related to sensor data applications, financial stock market boards, network performance measurement tools (e.g., network monitoring and traffic management applications), clickstream analysis tools, and automotive traffic monitoring. Server 1212 may also include one or more applications for displaying data feeds and / or real-time events via one or more display devices on client computing devices 1202, 1204, 1206, and 1208.

[0140] The distributed system 1200 may also include one or more data repositories 1214, 1216. In one example, these data repositories can be used to store data and other information. For example, one or more of the data repositories 1214, 1216 can be used to store information about chatbot performance or generated models for use by a chatbot used by server 1212 when performing various functions according to various embodiments. The data repositories 1214, 1216 can reside in various locations. For example, the data repository used by server 1212 may be located locally with server 1212, or it may be located remotely from server 1212 and communicates with server 1212 via a network-based or dedicated connection. The data repositories 1214, 1216 may be of different types. In one example, the data repository used by server 1212 may be a database, for example, a relational database such as a database provided by Oracle Corporation®. One or more of these databases may be configured to allow data to be stored, updated, and retrieved from the database in response to commands in SQL format. The data repository used by the application may be of various types, such as a key-value store repository, an object store repository, or a general-purpose storage repository supported by the file system.

[0141] In one example, one or more of the data repositories 1214, 1216 may be used by the application to store application data. The data repositories used by the application may be of different types, such as a key-value store repository, an object store repository, or a general-purpose storage repository supported by the file system.

[0142] In one example, the functions described in this disclosure may be provided as services via a cloud environment. Figure 13 is a simplified block diagram of a cloud-based system environment that may provide various services as cloud services in one example. In the example shown in Figure 13, the cloud infrastructure system 1302 may provide one or more cloud services that a user may request using one or more client computing devices 1304, 1306, and 1308. The cloud infrastructure system 1302 may include one or more computers and / or servers, which may include those described above with respect to server 1212. The computers within the cloud infrastructure system 1302 may be organized as general-purpose computers, dedicated server computers, server farms, server clusters, or any other appropriate configuration and / or combination.

[0143] Network 1310 can facilitate data communication and exchange between clients 1304, 1306, and 1308 and the cloud infrastructure system 1302. Network 1310 may include one or more networks. The networks may be of the same type or different types. Network 1310 can support one or more communication protocols, including wired and / or wireless protocols, to facilitate communication.

[0144] The embodiment shown in Figure 13 is merely one example of a cloud infrastructure system and is not intended to be limiting. It should be understood that in several other examples, the cloud infrastructure system 1302 may have more or fewer components than those shown in Figure 13, may combine two or more components, or may have different configurations or arrangements of components. For example, Figure 13 shows three client computing devices, but any number of client computing devices can be supported in alternative examples.

[0145] The term "cloud service" is generally used to refer to services made available to users on demand via communication networks such as the Internet, through a service provider's system (e.g., cloud infrastructure system 1302). Typically, in a public cloud environment, the servers and systems that make up the cloud service provider's system are different from the customer's own on-premises servers and systems. The cloud service provider's system is managed by the cloud service provider. Therefore, customers can use the cloud services provided by the cloud service provider without having to purchase separate licenses, support, or hardware and software resources for the service. For example, the cloud service provider's system can host applications, and users can order and use applications on demand via the Internet without having to purchase infrastructure resources to run the applications. Cloud services are designed to provide easy and scalable access to applications, resources, and services. Several providers offer cloud services. For example, several cloud services, such as middleware services, database services, Java cloud services, and others, are offered by Oracle Corporation® in Redwood Shores, California.

[0146] In one example, the cloud infrastructure system 1302 may provide one or more cloud services using different models, including a hybrid service model, such as a Software as a Service (SaaS) model, a Platform as a Service (PaaS) model, an Infrastructure as a Service (IaaS) model, or others. The cloud infrastructure system 1302 may include a suite of applications, middleware, databases, and other resources that enable the provisioning of various cloud services.

[0147] The SaaS model enables the delivery of applications or software as a service to customers over a communication network such as the internet, without requiring customers to purchase the underlying hardware or software for the application. For example, the SaaS model can be used to allow customers to access on-demand applications hosted on a cloud infrastructure system 1302. Examples of SaaS services offered by Oracle Corporation® include, but are not limited to, various services for human resources / capital management, customer relationship management (CRM), enterprise resource planning (ERP), supply chain management (SCM), enterprise performance management (EPM), analytics services, social applications, and others.

[0148] The IaaS model is generally used to provide flexible computing and storage capabilities by offering infrastructure resources (such as servers, storage, hardware, and networking resources) to customers as cloud services. Various IaaS services are offered by Oracle Corporation®.

[0149] The PaaS model is generally used to provide a platform and environmental resources as a service, enabling customers to develop, run, and manage applications and services without having to procure, build, or manage those resources themselves. Examples of PaaS services offered by Oracle Corporation® include, but are not limited to, Oracle Java Cloud Service (JCS), Oracle Database Cloud Service (DBCS), data management cloud services, various application development solutions, and others.

[0150] Cloud services are generally delivered in an on-demand, self-service-based, subscription-based, flexibly scalable, reliable, highly available, and secure manner. For example, a customer can order one or more services provided by the cloud infrastructure system 1302 through a subscription order. The cloud infrastructure system 1302 then provides the services requested in the customer's subscription order by performing processing. For example, as described above, a user may use utterances to request the cloud infrastructure system to perform a certain action (e.g., an intent) and / or provide services for a chatbot system as described herein. The cloud infrastructure system 1302 may be configured to provide one or more cloud services.

[0151] The cloud infrastructure system 1302 may provide cloud services through different deployment models. In a public cloud model, the cloud infrastructure system 1302 may be owned by a third-party cloud service provider, and the cloud services are provided to a general public customer, which may be an individual or a company. In another example, in a private cloud model, the cloud infrastructure system 1302 may function within an organization (for example, within a corporate organization), and the services are provided to customers within that organization. For example, these customers may be various departments such as human resources, payroll, etc., or individuals within the company. In yet another example, in a community cloud model, the cloud infrastructure system 1302 and the services provided may be shared among several organizations within a relevant community. Other models, such as hybrid models of the above models, can also be used.

[0152] The client computing devices 1304, 1306, and 1308 may be of different types (e.g., client computing devices 1202, 1204, 1206, and 1208 shown in Figure 12) and may be capable of running one or more client applications. Users may use the client devices to interact with the cloud infrastructure system 1302, for example, to request services provided by the cloud infrastructure system 1302. For example, as described in this disclosure, users may use the client devices to request information or actions from a chatbot.

[0153] In some examples, the processing performed by the cloud infrastructure system 1302 to provide a service may include training and deploying models. This analysis may involve using, analyzing, and manipulating datasets to train and deploy one or more models. This analysis may be performed by one or more processors, possibly processing data in parallel and performing simulations using the data. For example, big data analytics may be performed by the cloud infrastructure system 1302 to generate and train one or more models for a chatbot system. The data used in this analysis may include structured data (e.g., data stored in a database or data structured according to a structured model) and / or unstructured data (e.g., data blobs (binary large objects)).

[0154] As shown in the example in Figure 13, the cloud infrastructure system 1302 may include infrastructure resources 1330 that are used to facilitate the provision of various cloud services offered by the cloud infrastructure system 1302. Infrastructure resources 1330 may include, for example, processing resources, storage or memory resources, networking resources, etc. In one example, a storage virtual machine available to service storage requested by an application may be part of the cloud infrastructure system 1302. In another example, the storage virtual machine may be part of a different system.

[0155] In one example, to facilitate the efficient provisioning of these resources to support various cloud services provided by the cloud infrastructure system 1302 to different customers, resources may be grouped into resource sets or resource modules (also called "pods"). Each resource module or pod may contain a pre-integrated and optimized combination of one or more types of resources. In a particular embodiment, different pods may be pre-provisioned for different types of cloud services. For example, a first set of pods may be provisioned for a database service, and a second set of pods, which may contain different resource combinations than the pods in the first set, may be provisioned for a Java service, etc. For several services, resources allocated to provisioning these services may be shared among the services.

[0156] The cloud infrastructure system 1302 itself may internally use services 1332 that are shared by different components of the cloud infrastructure system 1302 and facilitate the provisioning of services by the cloud infrastructure system 1302. These internally shared services may include, but are not limited to, security and identity services, integration services, enterprise repository services, enterprise manager services, virus scanning and whitelisting services, high-availability backup and recovery services, services that enable cloud support, email services, notification services, and file transfer services.

[0157] The cloud infrastructure system 1302 may include multiple subsystems. These subsystems may be implemented in software, hardware, or a combination thereof. As shown in Figure 13, the subsystems may include a user interface subsystem 1312 that enables users or customers of the cloud infrastructure system 1302 to interact with the cloud infrastructure system 1302. The user interface subsystem 1312 may include various different interfaces, such as a web interface 1314, an online store interface 1316 where cloud services offered by the cloud infrastructure system 1302 are advertised and available for purchase by consumers, and other interfaces 1318. For example, a customer can use a client device to request one or more services offered by the cloud infrastructure system 1302 using one or more of interfaces 1314, 1316, and 1318 (service request 1334). For example, a customer can access the online store, browse the cloud services offered by the cloud infrastructure system 1302, and place a subscription order for one or more services offered by the cloud infrastructure system 1302 that the customer wishes to subscribe to. A service request may include information identifying the customer and one or more services the customer wishes to subscribe to. For example, a customer may place a subscription order for services provided by the cloud infrastructure system 1302. As part of the order, the customer may provide information identifying the chatbot system on which the service is provided, and optionally one or more credentials for the chatbot system.

[0158] In an example such as the embodiment shown in Figure 13, the cloud infrastructure system 1302 may include an Order Management Subsystem (OMS) 1302 configured to process new orders. As part of this process, the OMS 1320 may be configured to prepare the order for provisioning by creating a customer account if one does not already exist, receiving billing and / or account information from the customer to be used to charge the customer in order to provide the requested services to the customer, verifying the customer information, reserving the order for the customer after verification, and coordinating various workflows.

[0159] If properly validated, OMS1320 can invoke Order Provisioning Subsystem (OPS)1324, which is configured to provision resources for this order, including processing, memory, and networking resources. Provisioning may include allocating resources for the order and configuring those resources to facilitate the services requested by the customer order. The way resources are provisioned for an order and the types of resources provisioned may depend on the type of cloud service ordered by the customer. For example, following a certain workflow, OPS1324 may be configured to determine the specific cloud service being requested and to identify the number of pods that would have been pre-configured for that particular cloud service. The number of pods allocated for an order may depend on the size / volume / level / scope of the requested service. For example, the number of pods to allocate may be determined based on the number of users the service should support, the duration for which the service is requested, etc. The allocated pods may then be customized to suit the specific customer making the request in order to provide the requested service.

[0160] In some examples, the setup phase processing may be performed by the cloud infrastructure system 1302 as part of the provisioning process, as described above. The cloud infrastructure system 1302 may generate an application ID and select a storage virtual machine for the application from storage virtual machines provided by the cloud infrastructure system 1302 itself or from storage virtual machines provided by other systems other than the cloud infrastructure system 1302.

[0161] The cloud infrastructure system 1302 may send a response or notification 1344 to the requesting customer to indicate when the requested service will be available. In some examples, the customer may be sent information (e.g., a link) that enables the customer to begin using and utilizing the benefits of the requested service. In one example, the response to the customer requesting the service may include a chatbot system ID generated by the cloud infrastructure system 1302 and information identifying a chatbot system selected by the cloud infrastructure system 1302 for the chatbot system corresponding to the chatbot system ID.

[0162] The cloud infrastructure system 1302 may provide services to multiple customers. For each customer, the cloud infrastructure system 1302 is responsible for managing information related to one or more subscription orders received from the customer, maintaining customer data related to the orders, and providing the requested services to the customer. The cloud infrastructure system 1302 may also collect usage statistics regarding the customer's use of the subscribed services. For example, statistics may be collected on the amount of storage used, the amount of data transferred, the number of users, and the amount of system uptime and system downtime. This usage information may be used to charge customers. Billing may be done, for example, on a monthly basis.

[0163] The cloud infrastructure system 1302 may provide services to multiple customers in parallel. The cloud infrastructure system 1302 may store information about these customers, including, in some cases, copyright information. In one example, the cloud infrastructure system 1302 includes an Identity Management Subsystem (IMS) 1328 configured to manage customer information and provide segregation of managed information so that information about one customer cannot be accessed by another customer. The IMS 1328 may be configured to provide various security-related services, such as identity services, for example, information access management, authentication and authorization services, and services for managing customer identity and roles and associated capabilities.

[0164] Figure 14 shows an example computer system 1400. In some examples, computer system 1400 may be used to implement either a digital assistant or a chatbot system in a distributed environment, as well as within the various servers and computer systems described above. As shown in Figure 14, computer system 1400 includes various subsystems, including a processing subsystem 1404 that communicates with several other subsystems via a bus subsystem 1402. These other subsystems may include a processing acceleration unit 1406, an I / O subsystem 1408, a storage subsystem 1418, and a communication subsystem 1424. The storage subsystem 1418 may include a non-temporary computer-readable storage medium, including a storage medium 1422 and system memory 1410.

[0165] The bus subsystem 1402 provides a mechanism for various components and subsystems of the computer system 1400 to communicate with one another as intended. Although the bus subsystem 1402 is schematically shown as a single bus, alternative examples of the bus subsystem may utilize multiple buses. The bus subsystem 1402 may be one of several types of bus structures, including a memory bus or memory controller, peripheral bus, local bus, etc., using one of various bus architectures. For example, such architectures may include the Industry Standard Architecture (ISA) bus, Micro Channel Architecture (MCA) bus, Enhanced ISA (EISA) bus, Video Electronics Standards Association (VESA) local bus, and Peripheral Component Interconnect (PCI) bus, which can be implemented as a mezzanine bus manufactured according to the IEEE P1386.1 standard.

[0166] The processing subsystem 1404 controls the operation of the computer system 1400 and may include one or more processors, application-specific integrated circuits (ASICs), or field-programmable gate arrays (FPGAs). The processors may include single-core or multi-core processors. The processing resources of the computer system 1400 can be organized into one or more processing units 1432, 1434, etc. A processing unit may include one or more processors, one or more cores from the same or different processors, a combination of cores and processors, or other combinations of cores and processors. In some examples, the processing subsystem 1404 may include one or more dedicated coprocessors, such as graphics processors or digital signal processors (DSPs). In some examples, some or all of the processing units of the processing subsystem 1404 can be implemented using customized circuitry, such as application-specific integrated circuits (ASICs) or field-programmable gate arrays (FPGAs).

[0167] In some examples, processing units within the processing subsystem 1404 can execute instructions stored in system memory 1410 or computer-readable storage medium 1422. In various examples, processing units can execute various programs or code instructions and maintain multiple programs or processes running concurrently. At any given time, some or all of the program code to be executed may reside in system memory 1410 and / or computer-readable storage medium 1410, potentially containing one or more storage devices. Through appropriate programming, the processing subsystem 1404 can provide the various functions described above. In an example where the computer system 1400 is running one or more virtual machines, each virtual machine may be assigned to one or more processing units.

[0168] In one example, a processing acceleration unit 1406 may be optionally provided to perform customized processing to accelerate the overall processing performed by the computer system 1400, or to offload a portion of the processing performed by the processing subsystem 1404.

[0169] The I / O subsystem 1408 may include devices and mechanisms for inputting information into and / or outputting information from or through the computer system 1400. Generally, the use of the term “input device” is intended to include all conceivable types of devices and mechanisms for inputting information into the computer system 1400. User interface input devices may include, for example, pointing devices such as keyboards, mice or trackballs, touchpads or touchscreens integrated into displays, scroll wheels, click wheels, dials, buttons, switches, keypads, voice input devices with voice command recognition systems, microphones, and other types of input devices. User interface input devices may also include motion sensing and / or gesture recognition devices, such as Microsoft Kinect® motion sensors, Microsoft Xbox® 360 game controllers, and devices that provide interfaces for receiving input using gestures and voice commands, enabling users to control and interact with input devices. The user interface input device may also include an eye gesture recognition device, such as the Google Glass® blink detector, which detects eye movements from the user (e.g., blinking while taking a picture and / or making a menu selection) and translates the eye gestures into input to an input device (e.g., Google Glass®). The user interface input device may also include a voice recognition sensing device that enables the user to interact with a voice recognition system (e.g., Siri® Navigator) via voice commands.

[0170] Other examples of user interface input devices may include, but are not limited to, three-dimensional (3D) mice, joysticks or pointing sticks, gamepads and graphic tablets, as well as auditory / visual devices such as speakers, digital cameras, digital camcorders, portable media players, webcams, image scanners, fingerprint scanners, barcode readers, 3D scanners, 3D printers, laser rangefinders, and eye-tracking devices. User interface input devices may also include medical imaging input devices such as computed tomography, magnetic resonance imaging, positional emission tomography, and medical ultrasound devices. User interface input devices may also include audio input devices such as MIDI keyboards and digital musical instruments.

[0171] Generally, the use of the term "output device" is intended to include all conceivable types of devices and mechanisms for outputting information from the computer system 1400 to a user or another computer. User interface output devices may include non-visual displays such as display subsystems, indicator lights, or audio output devices. Display subsystems may include flat panel devices such as those using cathode ray tubes (CRTs), liquid crystal displays (LCDs), or plasma displays, projection devices, touchscreens, etc. For example, user interface output devices may include, but are not limited to, a variety of display devices that visually convey text, graphics, and audio / video information, such as monitors, printers, speakers, headphones, car navigation systems, plotters, audio output devices, and modems.

[0172] The storage subsystem 1418 provides a repository or datastore for storing information and data used by the computer system 1400. The storage subsystem 1418 provides a tangible, non-temporary, computer-readable storage medium for storing basic programming and data configurations that provide some example functionality. Software (e.g., programs, code modules, instructions) that, when executed by the processing subsystem 1404, provides the above-described functionality may be stored in the storage subsystem 1418. The software may be executed by one or more processing units of the processing subsystem 1404. The storage subsystem 1418 may also provide authentication in accordance with the teachings of this disclosure.

[0173] The storage subsystem 1418 may include one or more non-temporary memory devices, including volatile and non-volatile memory devices. As shown in Figure 14, the storage subsystem 1418 includes system memory 1410 and computer-readable storage medium 1422. The system memory 1410 may include several memories, including volatile primary random access memory (RAM) for storing instructions and data during program execution, and non-volatile read-only memory (ROM) or flash memory for storing fixed instructions. In some implementations, a basic input / output system (BIOS) containing basic routines that assist in the transfer of information between elements within the computer system 1400, such as during startup, may typically be stored in ROM. Typically, the RAM contains data and / or program modules that are currently being made operational and executed by the processing subsystem 1404. In some implementations, the system memory 1410 may include several different types of memory, such as static random access memory (SRAM) and dynamic random access memory (DRAM).

[0174] As an example, without limitation, as shown in Figure 14, the system memory 1410 may load running application programs 1412, program data 1414, and operating systems 1416, which may include various applications such as web browsers, middle-tier applications, and relational database management systems (RDBMS). As an example, the operating system 1416 may include Microsoft Windows®, Apple Macintosh® and / or Linux operating systems, UNIX® or UNIX-like operating systems available in various markets (including, but not limited to, various GNU / Linux operating systems, Google Chrome® OS, etc.), and / or various versions of mobile operating systems such as iOS, Windows® Phone, Android® OS, BlackBerry® OS, and Palm® OS operating systems.

[0175] The computer-readable storage medium 1422 can store programming and data structures that provide several example functions. The computer-readable storage medium 1422 can provide storage for computer-readable instructions, data structures, program modules, and other data for the computer system 1400. Software (programs, code modules, instructions) that provides the above functions when executed by the processing subsystem 1404 may be stored in the storage subsystem 1418. As an example, the computer-readable storage medium 1422 may include non-volatile memory such as a hard disk drive, magnetic disk drive, optical disk drive such as a CD-ROM, DVD, Blu-ray® disc, or other optical media. The computer-readable storage medium 1422 may also include, but is not limited to, a Zip® drive, flash memory card, Universal Serial Bus (USB) flash drive, Secure Digital (SD) card, DVD disc, digital videotape, etc. Computer-readable storage media 1422 may also include solid-state drives (SSDs) based on non-volatile memory such as flash memory-based SSDs, enterprise flash drives, and solid-state ROMs; SSDs based on volatile memory such as solid-state RAM, dynamic RAM, and static RAM; DRAM-based SSDs; magnetoresistive RAM (MRAM) SSDs; and hybrid SSDs that use a combination of DRAM and flash memory-based SSDs.

[0176] In one example, the storage subsystem 1418 may also include a computer-readable storage medium reader 1420 that can be further connected to the computer-readable storage medium 1422. The reader 1420 may be configured to receive and read data from memory devices such as disks or flash drives.

[0177] In one example, computer system 1400 may support virtualization technologies, including but not limited to virtualization of processing and memory resources. For example, computer system 1400 may provide support for running one or more virtual machines. In one example, computer system 1400 may run programs such as hypervisors to facilitate the configuration and management of virtual machines. Each virtual machine may be allocated memory, computing (e.g., processors, cores), I / O, and networking resources. Each virtual machine typically runs independently of other virtual machines. A virtual machine typically runs its own operating system, which may be the same as or different from the operating system run by other virtual machines run by computer system 1400. Thus, potentially multiple operating systems may run simultaneously by computer system 1400.

[0178] The communication subsystem 1424 provides an interface to other computer systems and networks. The communication subsystem 1424 acts as an interface for sending and receiving data between other systems and the computer system 1400. For example, the communication subsystem 1424 can enable the computer system 1400 to establish a communication channel to one or more client devices via the internet in order to send and receive information with one or more client devices. For example, if the computer system 1400 is used to implement the bot system 120 shown in Figure 1, the communication subsystem may be used to communicate with a chatbot system selected for the application.

[0179] The communication subsystem 1424 can support both wired and / or wireless communication protocols. In some examples, the communication subsystem 1424 may include radio frequency (RF) transceiver components for accessing wireless voice and / or data networks (e.g., cellular telephone technology, 3G, 4G, or EDGE (High Speed ​​Data Rate for Global Evolution)), a Global Positioning System (GPS) receiver component, and / or other components. In some examples, the communication subsystem 1424 may provide a wired network connection (e.g., Ethernet) in addition to or instead of a wireless interface.

[0180] The communication subsystem 1424 can receive and transmit data in various formats. In some examples, the communication subsystem 1424 can receive input communications in the form of structured data feeds and / or unstructured data feeds 1426, event streams 1428, event updates 1430, etc. For example, the communication subsystem 1424 may be configured to receive (or transmit) data feeds 1426 in real time from users of social media networks and / or other communication services such as web feeds like Twitter® feeds, Facebook® updates, Rich Site Summary (RSS) feeds, and / or real-time updates from one or more third-party sources.

[0181] In one example, the communication subsystem 1424 may be configured to receive data in the form of a continuous data stream, which may include an event stream 1428 and / or event update 1430 of real-time events that are inherently continuous or infinite and do not have a clear end. Examples of applications that generate continuous data include, for example, sensor data applications, financial stock market boards, network performance measurement tools (e.g., network monitoring and traffic management applications), clickstream analysis tools, and automotive traffic monitoring.

[0182] The communication subsystem 1424 may be configured to transmit data from computer system 1400 to other computer systems or networks. This data can be transmitted in various different formats, such as structured and / or unstructured data feeds 1426, event streams 1428, and event updates 1430, to one or more databases that can communicate with one or more streaming data source computers connected to computer system 1400.

[0183] Computer system 1400 may be one of many types, including handheld portable devices (e.g., iPhone® cellular phone, iPad® computing tablet, PDA), wearable devices (e.g., Google Glass® head-mounted display), personal computers, workstations, mainframes, kiosks, server racks, or other data processing systems. Due to the ever-changing nature of computers and networks, the description of computer system 1400 shown in Figure 14 is intended merely as a specific example. Many other configurations are possible, having more or fewer components than the system shown in Figure 14. Based on the disclosures and teachings provided herein, other ways and / or methods for realizing various examples should be understood.

[0184] While specific examples have been described, various modifications, changes, alternative configurations, and equivalents are possible. The examples are not limited to operation within a specific data processing environment, but can freely operate within multiple data processing environments. Furthermore, while some examples have been described using a specific set of transactions and steps, it should be apparent to those skilled in the art that this is not intended as limitation. Some flowcharts describe operations as sequential processes, but many of these operations can be performed in parallel or concurrently. In addition, the order of operations may be re-specified. Processes may have additional steps not shown in the diagrams. The various features and aspects of the above examples may be used individually or together.

[0185] Furthermore, while some examples have been described using specific combinations of hardware and software, it should be understood that other combinations of hardware and software are also possible. Some examples may be implemented using hardware alone, software alone, or a combination thereof. The various processes described herein can be implemented on the same processor or on different processors in any combination.

[0186] Where it is stated that a device, system, component, or module is configured to perform a particular operation or function, such configuration can be achieved, for example, by designing an electronic circuit to perform the operation; by programming a programmable electronic circuit (such as a microprocessor); by designing an electronic circuit to perform the operation, such as by executing computer instructions or code programmed to execute code or instructions stored in a non-temporary memory medium or any combination thereof; or by executing a processor or core. Processes can communicate using a variety of techniques, including but not limited to conventional techniques for inter-process communication, and different pairs of processes may use different techniques, or the same pair of processes may use different techniques at different times.

[0187] This disclosure provides certain details to ensure that the examples are fully understood. However, the examples can be implemented without these specific details. For example, well-known circuits, processes, algorithms, configurations, and techniques are shown without unnecessary details to avoid ambiguity in the examples. This specification provides only examples and is not intended to limit the scope, applicability, or configuration of other examples. Rather, the above description of the examples provides a description that enables the implementation of various examples for those skilled in the art. Various modifications can be made within the scope of the function and configuration of the elements.

[0188] Therefore, the specification and drawings should be considered illustrative rather than restrictive. However, it will be clear that additions, reductions, deletions, and other modifications and changes can be made to them without departing from the broader spirit and scope set forth in the claims. Thus, while specific examples have been described, they are not intended to be limiting. Various modifications and equivalents fall within the scope of the following claims.

[0189] While the above-described specification illustrates aspects of the disclosure with reference to specific examples, those skilled in the art will recognize that the disclosure is not limited thereto. The various features and aspects of the above-described disclosure may be used individually or together. Furthermore, the examples may be used in any number of environments and applications other than those described herein without departing from the broader spirit and scope of this specification. Accordingly, the specification and drawings should be considered illustrative rather than restrictive.

[0190] In the preceding description, the method is explained in a specific order for illustrative purposes. It should be understood that in alternative examples, the method may be performed in a different order than described. It should also be understood that the above method may be performed by hardware components, or by a sequence of machine-executable instructions that may be used to cause a machine, such as a general-purpose or dedicated processor or logic circuit programmed with instructions, to execute the method. These machine-executable instructions may be stored on one or more machine-readable media, such as a CD-ROM or other type of optical disk, a floppy diskette, ROM, RAM, EPROM, EEPROM, magnetic or optical card, flash memory, or other types of machine-readable media suitable for storing electronic instructions. Alternatively, the method may be performed by a combination of hardware and software.

[0191] Where components are described as being configured to perform a particular operation, such configuration may be achieved, for example, by designing electronic circuits or other hardware to perform the operation, by programming programmable electronic circuits (e.g., a microprocessor or other suitable electronic circuit) to perform the operation, or by any combination thereof.

[0192] While this specification has described in detail the exemplary embodiments of the present application, it should be understood that the concepts of the present invention may be carried out and adopted in a variety of other ways, and the appended claims are intended to be construed to include such modifications, unless limited by the prior art.

Claims

1. A method performed by a computer, wherein the method is The method includes the step of receiving an utterance from a user within a chatbot session, wherein the current skill context of the chatbot session is a first skill, and the current group context of the chatbot session is a first group, and the method further includes, The steps include inputting the aforementioned utterance into a candidate skill model for the first group, A step of obtaining a ranking of the skills within the first group that may process the utterance using the candidate skill model, Based on the ranking of the aforementioned skills, the second skill is determined to be the highest-ranked skill for processing the aforementioned utterance. The steps include changing the current skill context of the chatbot session to the second skill, The method further includes the step of inputting the utterance into a candidate flow model for the second skill, wherein the candidate flow model measures a plurality of degrees to which a plurality of intents within the second skill match the utterance, and the method further includes A step of obtaining a ranking of intents within the second skill that match the utterance using the aforementioned multiple degrees, Based on the ranking of the intents, the step of determining which intent is the highest-ranked intent for processing the utterance, A step of evaluating the intent and generating a confidence score for the intent within the second skill, The steps include identifying any intent with a confidence score exceeding the value of the confidence threshold routing parameter as a candidate intent for further evaluation, A computer-based method comprising the step of routing the candidate intent to the first skill which includes an intent that matches the utterance.

2. A method performed by a computer, wherein the method is The method includes the step of receiving an utterance from a user within a chatbot session, wherein the current skill context of the chatbot session is a first skill, and the current group context of the chatbot session is a first group, and the method further includes, The steps include inputting the aforementioned utterance into a candidate skill model for the first group, A step of obtaining a ranking of the skills within the first group that may process the utterance using the candidate skill model, Based on the ranking of the aforementioned skills, the second skill is determined to be the highest-ranked skill for processing the aforementioned utterance. The steps include changing the current skill context of the chatbot session to the second skill, The method further includes the step of inputting the utterance into a candidate flow model for the second skill, wherein the candidate flow model measures a plurality of degrees to which a plurality of intents within the second skill match the utterance, and the method further includes A step of obtaining a ranking of intents within the second skill that match the utterance using the aforementioned multiple degrees, Based on the ranking of the intents, the step of determining which intent is the highest-ranked intent for processing the utterance, The method includes the step of receiving a subsequent utterance from the user within the chatbot session, wherein the current skill context of the chatbot session is the second skill, the current group context of the chatbot session is the first group, and the method is The steps include inputting the subsequent utterance into the candidate skill model for the first group, A step of obtaining a ranking of the skills within the first group that may process the subsequent utterance using the candidate skill model, Based on the ranking of the aforementioned skills, the step of determining that the skill containing the intent closest to the unresolved intent is the highest-ranked skill for processing the subsequent utterance, The steps include inputting the subsequent utterance into another candidate skill model, A step of obtaining a ranking of skills that may be able to process the subsequent utterance using another candidate skill model, Based on the ranking of the aforementioned skills, the third skill is determined to be the highest-ranked skill for processing the subsequent utterance, Based on the ranking of the aforementioned skills, the third skill is determined to be the highest-ranked skill for processing the subsequent utterance. A computer-operated method further comprising the steps of assigning the current skill context of the chatbot session to the third skill and assigning the current group context of the chatbot session to the second group, wherein the second group is defined for the third skill, and assigning the current group context of the chatbot session to the second group is performed based on the definition of the second group for the third skill.

3. A method performed by a computer, wherein the method is The method includes the step of receiving an utterance from a user within a chatbot session, wherein the current skill context of the chatbot session is a first skill, and the current group context of the chatbot session is a first group, and the method further includes, The steps include inputting the aforementioned utterance into a candidate skill model for the first group, A step of obtaining a ranking of the skills within the first group that may process the utterance using the candidate skill model, Based on the ranking of the aforementioned skills, the second skill is determined to be the highest-ranked skill for processing the aforementioned utterance. The steps include changing the current skill context of the chatbot session to the second skill, The method further includes the step of inputting the utterance into a candidate flow model for the second skill, wherein the candidate flow model measures a plurality of degrees to which a plurality of intents within the second skill match the utterance, and the method further includes A step of obtaining a ranking of intents within the second skill that match the utterance using the aforementioned multiple degrees, Based on the ranking of the intents, the step of determining which intent is the highest-ranked intent for processing the utterance, Based on the ranking of the aforementioned skills, the third skill is determined to be the second highest-ranked skill for processing the aforementioned utterance. The steps include determining that both the second skill and the third skill are within the win margin parameter, In response to the determination that both the second skill and the third skill are within the win margin parameter, the steps include inputting the utterance into the candidate flow model for the second skill and another candidate flow model for the third skill, A step of obtaining a ranking of intents within the third skill that match the utterance using the aforementioned alternative candidate flow model, A computer-based method comprising the step of determining, based on the ranking of intents within the second and third skills, which intent is the highest-ranked intent for processing the utterance.

4. A computer-based method according to any one of claims 1 to 3, wherein the step of obtaining the skill ranking includes: evaluating the utterance and generating confidence scores for the skills in the first group; identifying any skills having confidence scores exceeding the value of a candidate skill confidence threshold routing parameter as candidate skills for further evaluation; and ranking the candidate skills as skills in the first group that may process the utterance based on the confidence scores.

5. A computer-based method according to any one of claims 1 to 4, further comprising the step of initiating a conversation flow with the user within the chatbot session based on the intent which is the highest-ranked intent for processing the utterance.

6. The chatbot session includes the step of receiving the first utterance from the user that is received before the aforementioned utterance, The steps include inputting the aforementioned first utterance into the candidate skill model, The steps include obtaining a ranking of skills that may be able to process the initial utterance using the candidate skill model, The steps include determining, based on the ranking of the skills, that the first skill is the highest-ranked skill for processing the first utterance, A computer-based method according to any one of claims 1 to 5, further comprising the steps of assigning the current skill context of the chatbot session to a first skill and assigning the current group context of the chatbot session to a first group, wherein the first group is defined for the first skill, and assigning the current group context of the chatbot session to the first group is performed based on the definition of the first group for the first skill.

7. One or more data processors, A system comprising a non-temporary computer-readable storage medium containing instructions, wherein, when the instructions are executed on one or more data processors, the one or more data processors cause the one or more data processors to perform an action, The action includes receiving utterances from the user within a chatbot session, wherein the current skill context of the chatbot session is a first skill, the current group context of the chatbot session is a first group, and the action further includes, Inputting the aforementioned utterance into a candidate skill model for the first group, Using the candidate skill model, obtain a ranking of the skills within the first group that may process the utterance. Based on the ranking of the aforementioned skills, it is determined that the second skill is the highest-ranked skill for processing the aforementioned utterance, The current skill context of the chatbot session is changed to the second skill, The action further includes inputting the utterance into a candidate flow model for the second skill, the candidate flow model measuring the degree to which multiple intents within the second skill match the utterance, and the action further Using the aforementioned multiple degrees, obtain a ranking of intents within the second skill that match the utterance, Based on the ranking of the intents, determine which intent is of the highest rank for processing the utterance. The intent is evaluated and a confidence score is generated for the intent within the second skill, Identifying any intent with a confidence score exceeding the value of the confidence threshold routing parameter as a candidate intent for further evaluation, A system comprising routing the candidate intent to the first skill which includes an intent that matches the utterance.

8. One or more data processors, A system comprising a non-temporary computer-readable storage medium containing instructions, wherein, when the instructions are executed on one or more data processors, the one or more data processors cause the one or more data processors to perform an action, The action includes receiving utterances from the user within a chatbot session, wherein the current skill context of the chatbot session is a first skill, the current group context of the chatbot session is a first group, and the action further includes, Inputting the aforementioned utterance into a candidate skill model for the first group, Using the candidate skill model, obtain a ranking of the skills within the first group that may process the utterance. Based on the ranking of the aforementioned skills, it is determined that the second skill is the highest-ranked skill for processing the aforementioned utterance, The current skill context of the chatbot session is changed to the second skill, The action further includes inputting the utterance into a candidate flow model for the second skill, the candidate flow model measuring the degree to which multiple intents within the second skill match the utterance, and the action further Using the aforementioned multiple degrees, obtain a ranking of intents within the second skill that match the utterance, Based on the ranking of the intents, the intent that is of the highest rank for processing the utterance is determined, The chatbot session further includes receiving subsequent utterances from the user, wherein the current skill context of the chatbot session is the second skill, the current group context of the chatbot session is the first group, and the action is Inputting the subsequent utterance into the candidate skill model for the first group, Using the candidate skill model, obtain a ranking of the skills within the first group that may process the subsequent utterance. Based on the aforementioned skill ranking, the skill containing the intent closest to the unresolved intent is determined to be the highest-ranked skill for processing the subsequent utterance. The subsequent utterance is input into another candidate skill model, Using a different candidate skill model, obtain a ranking of skills that may be able to process the subsequent utterance, Based on the ranking of the aforementioned skills, it is determined that the third skill is the highest-ranked skill for processing the subsequent utterance, Based on the ranking of the aforementioned skills, it is determined that the third skill is the highest-ranking skill for processing the subsequent utterance. A system further comprising assigning the current skill context of the chatbot session to the third skill, and assigning the current group context of the chatbot session to the second group, wherein the second group is defined for the third skill, and assigning the current group context of the chatbot session to the second group is performed based on the definition of the second group for the third skill.

9. One or more data processors, A system comprising a non-temporary computer-readable storage medium containing instructions, wherein, when the instructions are executed on one or more data processors, the one or more data processors cause the one or more data processors to perform an action, The action includes receiving utterances from the user within a chatbot session, wherein the current skill context of the chatbot session is a first skill, the current group context of the chatbot session is a first group, and the action further includes, Inputting the aforementioned utterance into a candidate skill model for the first group, Using the candidate skill model, obtain a ranking of the skills within the first group that may process the utterance. Based on the ranking of the aforementioned skills, it is determined that the second skill is the highest-ranked skill for processing the aforementioned utterance, The current skill context of the chatbot session is changed to the second skill, The action further includes inputting the utterance into a candidate flow model for the second skill, the candidate flow model measuring the degree to which multiple intents within the second skill match the utterance, and the action further Using the aforementioned multiple degrees, obtain a ranking of intents within the second skill that match the utterance, Based on the ranking of the intents, the intent that is of the highest rank for processing the utterance is determined, Based on the ranking of the aforementioned skills, it is determined that the third skill is the second highest-ranked skill for processing the aforementioned utterance, It is determined that both the second skill and the third skill are within the win margin parameter, In response to the determination that both the second skill and the third skill are within the win margin parameter, the utterance is input to the candidate flow model for the second skill and another candidate flow model for the third skill, Using the aforementioned alternative candidate flow model, obtain the ranking of intents within the third skill that match the utterance, A system that includes determining which intent is the highest-ranked intent for processing the utterance, based on the ranking of intents within the second and third skills.

10. The system according to any one of claims 7 to 9, wherein obtaining the ranking of the skills includes evaluating the utterance and generating confidence scores for the skills in the first group; identifying any skills having confidence scores exceeding the value of a candidate skill confidence threshold routing parameter as candidate skills for further evaluation; and ranking the candidate skills as skills in the first group that may process the utterance based on the confidence scores.

11. The system according to any one of claims 7 to 10, wherein the action further comprises initiating a conversational flow with the user within the chatbot session based on the intent which is the highest-ranking intent for processing the utterance.

12. The aforementioned Action is, Within the aforementioned chatbot session, the first utterance received from the user before the aforementioned utterance is received, The initial utterance is input into the candidate skill model, Using the aforementioned candidate skill model, obtain a ranking of the skills that may be able to process the initial utterance. Based on the ranking of the skills, it is determined that the first skill is the highest-ranked skill for processing the first utterance, The system according to any one of claims 7 to 11, further comprising assigning the current skill context of the chatbot session to a first skill and assigning the current group context of the chatbot session to a first group, wherein the first group is defined for the first skill, and assigning the current group context of the chatbot session to the first group is performed based on the definition of the first group for the first skill.

13. A computer program configured to cause one or more data processors to perform the method described in any one of claims 1 to 6.

Images