Out-of-scope automatic transition for chatbots

Abstract

Techniques for automatic switching between chatbot skills in the same domain. In one particular aspect, a method is provided that includes receiving an utterance from a user within a chatbot session, where a current skill context is a first skill and a current group context is a first group; inputting the utterance into a candidate skill model for the first group; obtaining a ranking of skills within the first group using the candidate skill model; determining that a second skill is a highest ranked skill based on the ranking of the skills; changing the current skill context of the chatbot session to the second skill; inputting the utterance into a candidate flow model for the second skill; obtaining a ranking of intents within the second skill that match the utterance using the candidate flow model; and determining an intent as a highest ranked intent based on the ranking of the intents.

Description

Automatic conversion outside the scope of chatbots

[0001] Priority requirements

[0002] This application claims the benefit and priority of U.S. Provisional Application No. 63 / 085,796, filed on September 30, 2020, which is incorporated herein by reference in its entirety for all purposes. Technical Field

[0003] This disclosure relates generally to chatbots, and more specifically to techniques for automatically switching between chatbot skills in the same domain. Background Technology

[0004] To achieve immediate responses, many users around the world utilize instant messaging or chat platforms. Organizations frequently use these platforms to engage in real-time conversations with customers (or end users). However, hiring service personnel to communicate with customers or end users in real time is prohibitively expensive for organizations. Chatbots or bots have begun to be developed to simulate conversations with end users, especially via the internet. End users can interact with bots through messaging applications they already have installed and are using. Intelligent bots (typically powered by artificial intelligence (AI)) can communicate more intelligently and context-sensitively in real-time conversations, thus allowing for more natural conversations between bots and end users to improve the conversational experience. Instead of end users learning a fixed set of keywords or commands to know how to respond, intelligent bots can understand the end user's intent based on natural language utterances and respond accordingly. Summary of the Invention

[0005] It provides a technology for automatically switching between chatbot skills in the same domain (e.g., a method, system, or storage of non-transitory computer-readable medium containing code or instructions executable by one or more processors).

[0006] In various embodiments, a computer-implemented method is provided, the method comprising: receiving a utterance from a user within a chatbot session, wherein the current skill context of the chatbot session is a first skill and the current group context of the chatbot session is a first group; inputting the utterance into a candidate skill model of the first group; using the candidate skill model to obtain a ranking of skills within the first group that may potentially process the utterance; determining, based on the skill ranking, a second skill as the highest-ranking skill for processing the utterance; changing the current skill context of the chatbot session to the second skill; inputting the utterance into a candidate flow model of the second skill; using the candidate flow model to obtain a ranking of intents within the second skill that match the utterance; and determining, based on the intent ranking, an intent as the highest-ranking intent for processing the utterance.

[0007] In some embodiments, obtaining the skill ranking includes: evaluating the utterance and generating a confidence score for the skill within the first group; identifying any skill whose confidence score exceeds the value of a candidate skill confidence threshold routing parameter as a candidate skill for further evaluation; and ranking the candidate skills as skills within the first group that may potentially process the utterance based on the confidence score.

[0008] In some embodiments, obtaining the intent ranking includes: evaluating the intent and generating a confidence score for the intent within the second skill; identifying any intent whose confidence score exceeds the value of a confidence threshold routing parameter as a candidate skill for further evaluation; and ranking the candidate intents as intents within the first skill that match the utterance based on the confidence score.

[0009] In some embodiments, the method further includes initiating a conversation flow in the chatbot session with the user based on an intent that is the highest-ranking intent for processing the utterance.

[0010] In some embodiments, the method further includes: receiving an initial utterance from a user within the chatbot session, wherein the initial utterance is received prior to the utterance; inputting the initial utterance into the candidate skill model; using the candidate skill model to obtain a ranking of skills that may potentially process the initial utterance; determining, based on the skill ranking, that the first skill is the highest-ranking skill for processing the initial utterance; assigning the current skill context of the chatbot session to the first skill and assigning the current group context of the chatbot session to the first group, wherein the first group is defined for the first skill, and assigning the current group context of the chatbot session to the first group is performed based on defining the first group for the first skill.

[0011] In some embodiments, the method further includes: receiving subsequent utterances from the user within the chatbot session, wherein the current skill context of the chatbot session is a second skill and the current group context of the chatbot session is a first group; inputting the subsequent utterances into a candidate skill model of the first group; using the candidate skill model to obtain a ranking of skills within the first group that may potentially process the subsequent utterances; determining, based on the skill rankings, that an unresolved intent skill is the highest-ranked skill for processing the subsequent utterances; inputting the subsequent utterances into another candidate skill model; using the other candidate skill model to obtain a ranking of skills that may potentially process the subsequent utterances; determining, based on the skill rankings, that a third skill is the highest-ranked skill for processing the subsequent utterances; and assigning the current skill context of the chatbot session to the third skill and assigning the current group context of the chatbot session to a second group, wherein the second group is defined for the third skill, and the assignment of the current group context of the chatbot session to the second group is performed based on the definition of the second group for the third skill.

[0012] In some embodiments, the method further includes: determining, based on the skill ranking, that a third skill is the second-highest ranked skill for processing the utterance; determining that both the second skill and the third skill are within a win margin parameter; in response to determining that both the second skill and the third skill are within the win margin parameter, inputting the utterance into a candidate flow model for the second skill and another candidate flow model for the third skill; using the other candidate flow model to obtain an intent ranking within the third skill that matches the utterance; and determining, based on the intent rankings within the second skill and the third skill, an intent as the highest-ranked intent for processing the utterance.

[0013] Some embodiments of this disclosure include a system comprising one or more data processors. In some embodiments, the system includes a non-transitory computer-readable storage medium containing instructions that, when executed on the one or more data processors, cause the one or more data processors to perform part or all of one or more of the methods and / or one or more processes disclosed herein.

[0014] Some embodiments of this disclosure include a computer program product tangibly embodied in a non-transitory machine-readable storage medium, the computer program product including instructions configured to cause one or more data processors to perform part or all of the methods and / or part or all of the processes disclosed herein.

[0015] The techniques described above and below can be implemented in various ways and in various contexts. Several example implementations and contexts are provided below with reference to the accompanying drawings, as described in more detail below. However, the following implementations and contexts are only a few of many implementations and contexts. Attached Figure Description

[0016] Figure 1 depicts a simplified diagram illustrating the concepts of in-domain, out-of-domain, within-scope, and outside-scope according to various embodiments.

[0017] Figure 2 depicts a simplified block diagram of a distributed environment incorporating exemplary embodiments.

[0018] Figure 3 depicts a simplified block diagram of a computing system implementing a master robot according to certain embodiments.

[0019] Figure 4 depicts a simplified block diagram of a computing system for a skill-implementing robot according to some embodiments.

[0020] Figures 5A and 5B depict graphical user interfaces according to various embodiments.

[0021] Figure 6 illustrates the intent invocation of exemplary utterances according to various embodiments.

[0022] Figure 7 depicts an intent invocation of another exemplary utterance according to various embodiments.

[0023] Figure 8 depicts an intent invocation of another exemplary utterance according to various embodiments.

[0024] Figure 9 depicts an intent invocation of another exemplary utterance according to various embodiments.

[0025] Figures 10A to 10C depict skill group context modifications for rule output, intent invocation, and route summarization according to various embodiments.

[0026] Figure 11 illustrates the process flow of context-aware routing with skill groups according to various embodiments.

[0027] Figure 12 depicts a simplified diagram of a distributed system used to implement various embodiments.

[0028] Figure 13 is a simplified block diagram of one or more components of a system environment according to various embodiments, through which services provided by one or more components of the embodiment system can be provided as cloud services.

[0029] Figure 14 illustrates an example computer system that can be used to implement various embodiments. Detailed Implementation

[0030] In the following description, specific details are set forth for purposes of explanation in order to provide a thorough understanding of certain embodiments. However, it will be apparent, however, that various embodiments may be practiced without these specific details. The accompanying drawings and description are not intended to be limiting. The word “exemplary” is used herein to mean “serving as an example, instance, or illustration.” Any embodiment or design described herein as “exemplary” is not necessarily to be construed as being more preferred or advantageous than other embodiments or designs.

[0031] introduction

[0032] A digital assistant is an AI-driven interface that helps users complete various tasks in natural language conversations. For each digital assistant, a client can assemble one or more skills. Skills (also described in this document as chatbots, bots, or skill bots) are individual bots focused on specific types of tasks such as tracking inventory, submitting timecards, and creating expense reports. When an end user interacts with a digital assistant, the assistant evaluates the end user's input and routes the conversation to and from the appropriate chatbot. This can be achieved through methods such as... Messenger, Skype Various channels, such as Messenger or Short Message Service (SMS), make digital assistants available to end users. Channels transmit chat back and forth between end users and digital assistants and their chatbots on various messaging platforms. Channels can also support user agent upgrades, event-initiated sessions, and testing.

[0033] Intents allow chatbots to understand what users want them to do. Intents comprise a sequence of typical user requests and statements, also known as utterances (e.g., getting account balance, making a purchase, etc.). As used in this paper, an utterance or message can refer to a set of words (e.g., one or more sentences) exchanged during a conversation with the chatbot. Intents can be created by providing a name that describes a user action (e.g., ordering pizza) and compiling a set of real-life user statements or utterances typically associated with triggering the action. Because the chatbot's cognition stems from these intents, each intent can be created and varied from a robust dataset (one to two dozen utterances), allowing the chatbot to interpret ambiguous user input. A rich set of utterances enables the chatbot to understand what a user wants when it receives messages such as "Forget this order!" or "Cancel delivery!" (meaning the same thing but expressed differently). Intents and the utterances that belong to intents together constitute the chatbot's training corpus. By training the model with the corpus, the client essentially turns the model into a reference tool for parsing end-user input into individual intents. Customers can improve the cognitive acuity of chatbots through multiple rounds of intent testing and intent training.

[0034] However, the utterances received by chatbots from real-world users in real-world environments (e.g., production environments) can be diverse and noisy. Some of these received utterances may be very different from the utterances used to train the chatbot and may not belong to the intents that the chatbot was trained to infer and process. For example, a banking chatbot might receive utterances unrelated to banking, such as "How do I book a trip to Italy?". Such utterances are called out-of-domain (OOD) utterances because they are not within the intent domain of the trained chatbot, as shown in Figure 1. It is important that the chatbot system be able to recognize such OOD utterances so that appropriate responses can be taken. For example, upon detecting an OOD utterance, the chatbot could respond to the user by indicating that the utterance is not something the bot can process or handle, rather than selecting the closest matching intent.

[0035] Furthermore, skills or chatbot groups can be deployed as part of the same domain. Typically, these skills are developed by different groups or departments within a company within the same domain. In such instances, it is common for the chatbot to receive utterances about intentions related to different skills belonging to the same domain. For example, a payroll chatbot in the human capital management (HCM) domain might receive utterances such as "What are my benefits," which are unrelated to payroll but are indeed related to benefits, which are part of the HCM domain. Such utterances are called out-of-scope (OOS) utterances because they are outside the scope of the trained chatbot's intent, as shown in Figure 1. Because these skills are part of the same domain, a user might fall into skill A (e.g., the payroll chatbot) because a question related to skill B (e.g., the benefits chatbot) in the same domain is outside the scope of skill A, but will pass through the OOD detector for skill A and may also match the intent in skill A with relatively high confidence. It is important that the chatbot system can recognize such OOS utterances so that appropriate response actions can be taken. For example, when an OOS utterance is detected, the context-aware router routes the utterance from the current chatbot (e.g., the payroll chatbot) to the most relevant chatbot within the defined group in that domain (e.g., the benefits chatbot).

[0036] This disclosure describes various embodiments for solving a problem by identifying OOS utterances and providing OOS transitions between skills within the same domain. In some embodiments, grouping-based techniques and routing-based techniques are used for OOS determination and transition. In an illustrative embodiment, a computer-implemented method is provided, comprising: receiving utterances from a user within a chatbot session, wherein the current skill context of the chatbot session is a first skill and the current group context of the chatbot session is a first group; inputting the utterances into a candidate skill model of the first group; using the candidate skill model to obtain a ranking of skills within the first group that may potentially handle the utterances; determining, based on the skill rankings, that a second skill is the highest-ranked skill for handling the utterances; changing the current skill context of the chatbot session to the second skill; inputting the utterances into a candidate flow model of the second skill; using the candidate flow model to obtain a ranking of intents within the second skill that match the utterances; and determining, based on the intent rankings, an intent as the highest-ranked intent for handling the utterances.

[0037] Robots and Analysis Systems

[0038] A bot (also known as a skill, chatbot, conversational bot, or talkative bot) is a computer program capable of engaging in conversations with an end user. Bots typically respond to natural language messages (e.g., questions or comments) via messaging applications that use natural language messaging. Businesses can use one or more bot systems to communicate with end users through messaging applications. The messaging application (which may be referred to as a channel) can be a preferred messaging application that the end user already has installed and is familiar with. Therefore, the end user does not need to download and install a new application to chat with the bot system. Messaging applications can include, for example, over-the-top (OTT) messaging channels (such as Facebook Messenger, Facebook WhatsApp, WeChat, Line, Kik, Telegram, Talk, Skype, Slack, or SMS), virtual personal assistants (such as Amazon Dot, Echo, or Show, Google Home, Apple HomePod, etc.), native or hybrid extended mobile and web applications / responsive mobile or web applications with chat functionality, or voice-based input (such as devices or applications with interfaces using Siri, Cortana, Google Voice, or other voice inputs for interaction).

[0039] In some examples, a robot system may be associated with a Uniform Resource Identifier (URI). A URI can identify a robot system using a string of characters. A URI can be used as a webhook for one or more messaging application systems. A URI may include, for example, a Uniform Resource Locator (URL) or a Uniform Resource Name (URN). The robot system may be designed to receive messages from a messaging application system (e.g., a Hypertext Transfer Protocol (HTTP) POST message). An HTTP POST message may point to a URI from a messaging application system. In some embodiments, the message may be different from an HTTP POST message. For example, the robot system may receive messages from a Short Message Service (SMS). While the discussion herein may refer to communication received by the robot system as a message, it should be understood that a message can be an HTTP POST message, an SMS message, or any other type of communication between two systems.

[0040] End users can interact with robot systems through conversational interactions (sometimes called conversational user interface (UI)), much like human interactions. In some cases, the interaction may include the end user saying "Hello" to the robot and the robot responding with "Hi" and asking how the robot can help. In other cases, the interaction may be transactional with, for example, a banking robot, such as transferring money from one account to another; informational with, for example, an HR robot, such as checking holiday balances; or with, for example, a retail robot, such as discussing returning purchased goods or seeking technical support.

[0041] In some embodiments, a robot system can intelligently handle end-user interactions without interaction with the robot system's administrator or developer. For example, an end-user can send one or more messages to the robot system to achieve a desired goal. Messages may include some form of content, such as text, emojis, audio, images, video, or other methods of conveying the message. In some embodiments, the robot system can convert the content into a standardized form (e.g., using appropriate parameters for a Representational State Transition (REST) ​​call to an enterprise service) and generate a natural language response. The robot system may also prompt the end-user for additional input parameters or request additional information. In some embodiments, the robot system may also initiate communication with the end-user rather than passively responding to end-user utterances. Various techniques are described herein for identifying explicit calls to a robot system and determining the input of the robot system being called. In some embodiments, explicit call analysis is performed by the master robot based on the detected call name in the utterance. In response to the detection of the call name, the utterance can be refined for input to the skill robot associated with the call name.

[0042] A conversation with a chatbot can follow a specific conversational flow that includes multiple states. This flow can be defined based on input to determine what will happen next. In some embodiments, the chatbot system can be implemented using a state machine that includes user-defined states (e.g., end-user intents) and actions to be taken in or between states. The conversation can take different paths based on end-user input, which can affect the decisions the chatbot makes regarding the flow. For example, in each state, based on end-user input or utterances, the chatbot can determine the end-user's intent in order to determine the next appropriate action to take. As used herein and in the context of utterances, the term "intent" refers to the intent of the user providing the utterance. For example, a user might intend to involve the chatbot in a conversation about ordering pizza, so the user's intent can be expressed through the utterance "Order pizza". User intents can point to a specific task that the user wants the chatbot to perform on their behalf. Therefore, utterances can be expressed as questions, commands, requests, etc., that reflect the user's intent. Intents can include goals that the end-user wants to accomplish.

[0043] In the context of chatbot configuration, the term "intent" as used herein refers to configuration information used to map a user's utterances to a specific task / action or a specific type of task / action that the chatbot can perform. To distinguish the intent of the utterances (i.e., the user's intent) from the intent of the chatbot, the latter is sometimes referred to as "bot intent." A bot intent can include one or more utterances associated with the intent. For example, the intent to order pizza can have various permutations of utterances expressing the expectation of placing an order for pizza. These associated utterances can be used to train the chatbot's intent classifier so that the intent classifier can subsequently determine whether an input utterance from the user matches the pizza-ordering intent. A bot intent can be associated with one or more conversational flows used to initiate a conversation with the user and in a certain state. For example, the first message for the pizza-ordering intent could be the question "What kind of pizza would you like?". In addition to associated utterances, bot intents can further include named entities related to the intent. For example, the pizza-ordering intent can include variables or parameters for performing the task of ordering pizza, such as topping 1, topping 2, pizza type, pizza size, number of pizzas, etc. The values ​​of the entities are typically obtained through conversation with the user.

[0044] Figure 2 is a simplified block diagram of an environment 200 incorporating a chatbot system according to some embodiments. Environment 200 includes a Digital Assistant Builder Platform (DABP) 202, which enables users of DABP 202 to create and deploy digital assistant or chatbot systems. DABP 202 can be used to create one or more digital assistants (or DAs) or chatbot systems. For example, as shown in Figure 2, a user 204 representing a particular business can use DABP 202 to create and deploy a digital assistant 206 for users of that particular business. For example, DABP 202 can be used by a bank to create one or more digital assistants for use by the bank's customers. Multiple businesses can use the same DABP 202 platform to create digital assistants. As another example, a restaurant owner (e.g., a pizzeria) can use DABP 202 to create and deploy a digital assistant that enables the restaurant's customers to order food (e.g., order pizza).

[0045] For the purposes of this disclosure, a "digital assistant" is an entity that assists a user in performing various tasks through natural language conversation. A digital assistant can be implemented using software alone (e.g., a digital assistant is a digital entity implemented using programs, code, or instructions executable by one or more processors), hardware, or a combination of hardware and software. Digital assistants can be embodied or implemented in various physical systems or devices such as computers, mobile phones, watches, appliances, and vehicles. Digital assistants are sometimes also referred to as chatbot systems. Therefore, for the purposes of this disclosure, the terms digital assistant and chatbot system are used interchangeably.

[0046] A digital assistant (such as digital assistant 206 built using DABP 202) can be used to perform various tasks via a natural language-based conversation between the digital assistant and its user 208. As part of the conversation, the user can provide one or more user inputs 210 to the digital assistant 206 and receive a response 212 returned by the digital assistant 206. The conversation may include one or more of the inputs 210 and the response 212. Through these conversations, the user can request one or more tasks to be performed by the digital assistant, and in response, the digital assistant is configured to perform the task requested by the user and respond to the user with an appropriate response.

[0047] User input 210 is typically in the form of natural language and is referred to as a speech. User speech 210 can be in text form, such as when a user types a sentence, question, text fragment, or even a single word and provides the text as input to the digital assistant 206. In some embodiments, user speech 210 can be in the form of audio input or speech, such as when a user speaks or utters something provided as input to the digital assistant 206. Speech is typically in the form of the language spoken by the user 208. For example, speech can be English or some other language. When speech is in speech form, the speech input is converted into speech in text form in that particular language, and then the text speech is processed by the digital assistant 206. Various speech-to-text processing techniques can be used to convert speech or audio input into text speech, which is then processed by the digital assistant 206. In some embodiments, the speech-to-text conversion can be performed by the digital assistant 206 itself.

[0048] A discourse (which may be text or speech) can be a fragment, a sentence, multiple sentences, one or more words, one or more questions, a combination of the above types, etc. Digital Assistant 206 is configured to apply Natural Language Understanding (NLU) technology to the discourse to understand the meaning of user input. As part of the NLU processing of the discourse, Digital Assistant 206 is configured to perform processing to understand the meaning of the discourse, which involves identifying one or more intentions and one or more entities corresponding to the discourse. After understanding the meaning of the discourse, Digital Assistant 206 may perform one or more actions or operations in response to the understood meaning or intention. For the purposes of this disclosure, it is assumed that the discourse is a text discourse directly provided by user 208 of Digital Assistant 206, or the result of converting input speech discourse into text form. However, this is not intended to limit or constrain in any way.

[0049] For example, user 208 can request to order a pizza by providing a statement such as "I want to order a pizza." Upon receiving such a statement, digital assistant 206 is configured to understand the meaning of the statement and take an appropriate action. The appropriate action may involve responding to the user's input, for example, by requesting information about the type of pizza the user wishes to order, its size, and any toppings. The responses provided by digital assistant 206 can also be in natural language and are typically in the same language as the input statement. As part of generating these responses, digital assistant 206 can perform natural language generation (NLG). To enable the user to order a pizza via a conversation between the user and digital assistant 206, the digital assistant can guide the user to provide all the necessary information for the pizza order and then allow the pizza to be ordered at the end of the conversation. Digital assistant 206 can end the conversation by outputting information to the user indicating that the pizza has been ordered.

[0050] At the conceptual level, the digital assistant 206 performs various processes in response to utterances received from a user. In some embodiments, this processing involves a series of processing steps or a pipeline of processing steps, including, for example, understanding the meaning of the input utterance (using NLU), determining the action to be performed in response to the utterance, performing the action where appropriate, generating a response to be output to the user in response to the user's utterance, and outputting the response to the user. NLU processing may include performing syntactic analysis on the received input utterance to understand the structure and meaning of the utterance, refining and reformulating the utterance to develop a better comprehensible form (e.g., logical form) or structure of the utterance. Generating a response may include using natural language generation (NLG) techniques. Therefore, natural language processing (NLP) performed by the digital assistant may include a combination of NLU processing and NLG processing.

[0051] NLU processing performed by a digital assistant (such as digital assistant 206) may include various NLU-related processes such as sentence parsing (e.g., tokenization, lemmatizing, identifying part-of-speech tags, identifying named entities in a sentence, generating dependency trees to represent sentence structure, segmenting sentences into clauses, analyzing individual clauses, parsing initial repetitions, performing chunking analysis, etc.). In some embodiments, NLU processing, or portions thereof, is performed by digital assistant 206 itself. In some other embodiments, digital assistant 206 may use other resources to perform portions of NLU processing. For example, the syntax and structure of the input utterance sentence can be identified by processing the sentence using a parser, part-of-speech tagger, and / or named entity recognizer. In one implementation, for the English language, a parser, part-of-speech tagger, and named entity recognizer, such as those provided by the Stanford NLP Group, are used to analyze sentence structure and syntax. These are provided as part of the Stanford CoreNLP toolkit.

[0052] While the various examples provided in this disclosure illustrate utterances in the English language, this is merely illustrative. In some embodiments, the digital assistant 206 is also capable of processing utterances in languages ​​other than English. The digital assistant 206 may provide subsystems (e.g., components implementing NLU functionality) configured to perform processing for different languages. These subsystems may be implemented as pluggable units that can be invoked from the NLU core server using service calls. This makes NLU processing flexible and scalable for each language, including allowing different processing sequences. Language packs may be provided for individual languages, wherein the language packs may register a list of subsystems that can be served from the NLU core server.

[0053] A digital assistant (such as digital assistant 206 as depicted in Figure 2) can be made available or accessible to its user 208 through various channels (e.g., but not limited to, via certain applications, via social media platforms, via various messaging services and applications (e.g., instant messaging applications), and other applications or channels). A single digital assistant can be configured with multiple channels, allowing it to run on different services simultaneously and be accessed through different services.

[0054] Digital assistant or chatbot systems typically include or are associated with one or more skills. In some embodiments, these skills are individual chatbots (referred to as skill bots) configured to interact with a user and perform specific types of tasks, such as tracking inventory, submitting time cards, creating expense reports, ordering food, checking bank accounts, making appointments, purchasing widgets, etc. For example, in the embodiment depicted in Figure 2, digital assistant or chatbot system 206 includes skills 216-1, 216-2, etc. For the purposes of this disclosure, the terms "one skill" and "multiple skills" are used synonymously with the terms "one skill bot" and "multiple skill bots," respectively.

[0055] Each skill associated with the digital assistant helps the user complete tasks through a conversation with the user, where the conversation may include a combination of text or audio input provided by the user and responses provided by the skill bot. These responses may be in the form of text or audio messages to the user and / or provided using simple user interface elements (e.g., a selection list) presented to the user for selection.

[0056] There are various methods to associate or add skills or skill bots to digital assistants. In some instances, skill bots can be developed by businesses and then added to digital assistants using DABP 202, for example, through a user interface provided by DABP 202 for registering skill bots with digital assistants. In other instances, skill bots can be developed and created using DABP 202 and then added to digital assistants created using DABP 202. In still other instances, DABP 202 provides an online digital store (called the "Skill Store") that offers multiple skills geared towards a wide variety of tasks. Skills offered through the Skill Store can also be exposed to various cloud services. To add a skill to a digital assistant created using DABP 202, a DABP 202 user can access the Skill Store via DABP 202, select the desired skill, and instruct the selected skill to be added to the digital assistant created using DABP 202. Skills from the Skills Store can be added to the digital assistant as is or with modifications (for example, a DABP 202 user can select and copy a specific skill bot from the Skills Store, customize or modify the selected skill bot, and then add the modified skill bot to the digital assistant created using the DABP 202).

[0057] Various architectures can be used to implement digital assistant or chatbot systems. For example, in some embodiments, a digital assistant created and deployed using DABP 202 can be implemented using a master robot / secondary (or sub) robot paradigm or architecture. According to this paradigm, the digital assistant is implemented as a master robot that interacts with one or more secondary robots that are skill robots. For example, in the embodiment depicted in Figure 2, digital assistant 206 includes a master robot 214 and skill robots 216-1, 216-2, etc., that are secondary robots of the master robot 214. In some embodiments, digital assistant 206 itself is considered to act as a master robot.

[0058] Digital assistants implemented using a master-slave robot architecture enable users to interact with multiple skills through a unified user interface, i.e., via a master robot. When a user engages with the digital assistant, the master robot receives user input. The master robot then processes the input to determine its meaning. It then determines whether the task requested in the user's utterance can be handled by the master robot itself; otherwise, it selects the appropriate skill robot to handle the request and routes the conversation to the selected skill robot. This allows users to interact with the digital assistant through a common, single interface while still providing the ability to use multiple skill robots configured to perform specific tasks. For example, for a digital assistant developed for an enterprise, the master robot can interface with skill robots with specific functions, such as a CRM robot for performing functions related to Customer Relationship Management (CRM), an ERP robot for performing functions related to Enterprise Resource Planning (ERP), an HCM robot for performing functions related to Human Capital Management (HCM), etc. In this way, the end user or consumer of the digital assistant only needs to know how to access the digital assistant through a common master robot interface and have multiple skill robots available in the background to handle user requests.

[0059] In some embodiments, in a master robot / sub-robot infrastructure, the master robot is configured to know a list of available skill robots. The master robot can access metadata identifying the various available skill robots, and for each skill robot, access the skill robot's capabilities, including tasks that can be performed by the skill robot. Upon receiving a user request in the form of utterances, the master robot is configured to identify or predict from among the multiple available skill robots that a particular skill robot can best serve or process the user request. The master robot then routes the utterance (or a portion of the utterance) to that particular skill robot for further processing. Thus, control flows from the master robot to the skill robot. The master robot can support multiple input and output channels. In some embodiments, routing can be performed by means of processing performed by one or more available skill robots. For example, as discussed below, skill robots can be trained to infer the intent of the utterance and determine whether the inferred intent matches the intent configured for the skill robot. Therefore, routing performed by the master robot can involve the skill robot conveying to the master robot an indication of whether the skill robot has been configured with an intent suitable for processing the utterance.

[0060] While the embodiment in Figure 2 illustrates digital assistant 206 including master robot 214 and skill robots 216-1, 216-2, and 216-3, this is not intended to be limiting. Digital assistants may include various other components (e.g., other systems and subsystems) that provide the functionality of the digital assistant. These systems and subsystems may be implemented solely in software (e.g., code, instructions stored on a computer-readable medium and executable by one or more processors), solely in hardware, or in implementations using a combination of software and hardware.

[0061] DABP 202 provides the infrastructure, services, and features that enable DABP 202 users to create digital assistants (including one or more skill bots associated with the digital assistant). In some instances, skill bots can be created by copying existing skill bots, for example, copying skill bots provided by a skill store. As mentioned earlier, DABP 202 can provide a skill store or skill catalog that offers multiple skill bots for performing various tasks. DABP 202 users can copy skill bots from the skill store. The copied skill bots can be modified or customized as needed. In some other instances, DABP 202 users create skill bots from scratch using the tools and services provided by DABP 202.

[0062] In some embodiments, at a high level, creating or customizing a skill-based robot involves the following steps:

[0063] (1) Configure settings for new skill robots

[0064] (2) Configure one or more intentions for the skill robot

[0065] (3) Configure one or more entities for one or more intents.

[0066] (4) Training Skills Robot

[0067] (5) Create a dialogue flow for the skill robot

[0068] (6) Add custom parts to the skill robot as needed.

[0069] (7) Test and deploy skill robots

[0070] Each of the above steps is briefly described below.

[0071] (1) Configuring settings for new skill robots—Various settings can be configured for skill robots. For example, skill robot designers can specify one or more invocation names for the skill robot being created. These invocation names, which serve as identifiers for the skill robot, can then be used by the digital assistant's user to explicitly invoke the skill robot. For example, the user can include the invocation name in their speech to explicitly invoke the corresponding skill robot.

[0072] (2) Configuring one or more intents and associated example utterances for the skill bot—The skill bot designer assigns one or more intents (also called bot intents) to the skill bot being created. The skill bot is then trained based on these assigned intents. These intents represent categories or classifications that the skill bot is trained to infer in response to input utterances. Upon receiving a utterance, the trained skill bot infers the intent of that utterance, where the inferred intent is selected from a predefined set of intents used to train the skill bot. The skill bot then takes an appropriate action in response to the utterance based on the intents inferred for it. In some instances, the intents of the skill bot represent tasks that the skill bot can perform for a digital assistant user. Each intent is given an intent identifier or intent name. For example, for a skill bot trained for banking, the intents assigned to that skill bot could include “CheckBalance,” “TransferMoney,” “DepositCheck,” etc.

[0073] For each intent defined for a skill bot, the skill bot designer can also provide one or more example utterances to represent and illustrate the intent. These example utterances are designed to represent the utterances a user would input to the skill bot in response to that intent. For example, for the CheckBalance intent, example utterances could include “What's my savings account balance?”, “How much is in my checking account?”, “How much money do I have in my account?”, etc. Therefore, various permutations of typical user utterances can be designated as example utterances for the intent.

[0074] These intentions and their associated example utterances are used as training data for training the skill robot. Various training techniques can be used. As a result of this training, a predictive model is generated, which is configured to take utterances as input and output the intentions inferred by the predictive model for those utterances. In some instances, the input utterances are fed to an intention analysis engine (e.g., a rule-based or machine learning-based classifier executed by the skill robot), which is configured to use the trained model to predict or infer the intentions of the input utterances. The skill robot can then take one or more actions based on the inferred intentions.

[0075] (3) Configuring Entities for One or More Intents of a Skill Robot—In some instances, additional context may be needed for the skill robot to respond appropriately to user utterances. For example, there may be cases where user input utterances are resolved to the same intent in the skill robot. For instance, in the example above, the utterances “What's my savings account balance?” and “How much is in my checking account?” are both resolved to the same CheckBalance intent, but these utterances are different requests for different things. To clarify such requests, one or more entities can be added to the intent. Using the example of a banking skill robot, an entity called AccountType (which defines values ​​called “checking” and “saving”) enables the skill robot to perform syntactic analysis on user requests and respond appropriately. In the example above, although these utterances are resolved to the same intent, the values ​​associated with the AccountType entity are different for the two utterances. This allows the skill robot to perform potentially different actions for the two utterances, even though they are resolved to the same intent. One or more entities can be specified for certain intents configured for the skill robot. Thus, entities are used to add context to the intent itself. Entities help to more fully describe intent and enable skill-based robots to fulfill user requests.

[0076] In some embodiments, there are two types of entities: (a) built-in entities provided by DABP 202; and (2) custom entities that can be specified by the skill bot designer. Built-in entities are general-purpose entities that can be used with various bots. Examples of built-in entities include, but are not limited to, entities relating to time, date, address, number, email address, duration, cycle time, currency, phone number, URL, etc. Custom entities are used for more customized applications. For example, for banking skills, the AccountType entity can be defined by the skill bot designer to enable various banking transactions by examining user input targeting keywords such as current account, savings, credit card, etc.

[0077] (4) Training the Skill Robot—The skill robot is configured to receive user input in utterance form, perform syntactic analysis or otherwise process the received input, and identify or select an intent associated with the received user input. As indicated above, the skill robot must be trained for this purpose. In some embodiments, the skill robot is trained based on intents configured for the skill robot and example utterances associated with those intents (collectively, training data), such that the skill robot can parse user input utterances into one of its configured intents. In some embodiments, the skill robot uses a predictive model trained with the training data and allows the skill robot to discern what the user says (or, in some cases, is attempting to say). DABP 202 provides a variety of different training techniques that skill robot designers can use to train skill robots, including various machine learning-based training techniques, rule-based training techniques, and / or combinations thereof. In some embodiments, a portion of the training data (e.g., 80%) is used to train the skill robot model and another portion (e.g., the remaining 20%) is used to test or validate the model. Once trained, the trained model (sometimes referred to as the trained skill robot) can be used to process and respond to user utterances. In some cases, a user's utterance may be a question that requires only a single answer and no further conversation. To handle this, a Q&A (question and answer) intent can be defined for the skill bot. Q&A intents are created in a similar manner to regular intents. The dialogue flow for a Q&A intent may differ from that for a regular intent. For example, unlike a regular intent, the dialogue flow for a Q&A intent may not involve prompts to solicit additional information from the user (e.g., the value of a specific entity).

[0078] (5) Creating a Dialogue Flow for a Skilled Bot—The dialogue flow specified for a skilled bot describes how the bot reacts as it parses different intentions in response to received user input. The dialogue flow defines the actions or behaviors the bot will take (e.g., how the bot responds to user utterances, how it prompts for user input, and how it returns data). The dialogue flow is like a flowchart followed by the bot. The bot designer specifies the dialogue flow using a language such as Markdown. In some embodiments, a version of YAML called OBotML can be used to specify the bot's dialogue flow. The dialogue flow definition for the bot acts as a model of the session itself, allowing the bot designer to orchestrate interactions between the bot and the users it serves.

[0079] In some embodiments, the dialogue flow definition for a skill-based robot comprises three parts:

[0080] (a) Contextual part

[0081] (b) Default Transformation Section

[0082] (c) State section

[0083] The context section—skill bot designers can define variables used in the session flow in the context section. Other variables that can be named in the context section include, but are not limited to: variables for error handling, variables for built-in or custom entities, user variables that enable the skill bot to recognize and save user preferences, etc.

[0084] The Default Transition section—transitions for the skill bot can be defined either in the dialog flow state section or in the Default Transition section. Transitions defined in the Default Transition section act as fallbacks and are triggered when no applicable transition is defined within the state or when the conditions required to trigger a state transition cannot be met. The Default Transition section can be used to define routes that allow the skill bot to handle unexpected user actions appropriately.

[0085] The state component—the dialogue flow and its associated operations—is defined as a sequence of temporary states that manage the logic within the dialogue flow. Each state node within the dialogue flow definition is named a component, which provides the functionality required at that point in the dialogue. Therefore, states are built around components. States contain component-specific properties and define the transitions to other states that are triggered after a component's execution.

[0086] A state portion can be used to handle special scenarios. For example, sometimes it might be desirable to offer a user the option to temporarily engage with a first skill within the digital assistant while performing actions within a second skill. For instance, if a user is busy with a shopping skill (e.g., the user has already made some purchase choices), the user might want to switch to a banking skill (e.g., the user might want to ensure he / she has enough money for the purchase) and then return to the shopping skill to complete the order. To address this, the state portion of the first skill's dialogue flow definition can be configured to initiate an interaction with a second, different skill within the same digital assistant and then return to the original dialogue flow.

[0087] (6) Adding Custom Components to the Skill Robot—As described above, the states specified in the skill robot's dialogue flow are named components that provide the required functionality corresponding to those states. Components enable the skill robot to perform functions. In some embodiments, DABP 202 provides a set of pre-configured components for performing a wide variety of functions. The skill robot designer can select one or more of these pre-configured components and associate them with states in the skill robot's dialogue flow. The skill robot designer can also use the tools provided by DABP 202 to create custom or new components and associate custom components with one or more states in the skill robot's dialogue flow.

[0088] (7) Testing and Deploying Skilled Robots – DABP 202 provides several features that enable skilled robot designers to test the skilled robots they are developing. The skilled robots can then be deployed and included in digital assistants.

[0089] While the above description outlines how to create skill bots, similar techniques can be used to create digital assistants (or master bots). At the master bot or digital assistant level, built-in system intents can be configured for the digital assistant. These built-in system intents are used to identify general tasks that the digital assistant (i.e., the master bot) can handle without invoking the skill bots associated with it. Examples of system intents defined for the master bot include: (1) Exit: applicable when a user signals that they wish to exit the current session or context within the digital assistant; (2) Help: applicable when a user requests help or direction; and (3) UnresolvedIntent: applicable to user input that doesn't quite match the Exit and Help intents. The digital assistant also stores information about one or more skill bots associated with it. This information enables the master bot to select a specific skill bot for processing utterances.

[0090] At the master robot or digital assistant level, when a user inputs a phrase or utterance into the digital assistant, the assistant is configured to perform processing to determine how to route the utterance and related conversation. The digital assistant uses a routing model to determine this, which can be rule-based, AI-based, or a combination thereof. The routing model determines whether the conversation corresponding to the user's input utterance should be routed to a specific skill for processing, processed by the digital assistant or master robot itself according to built-in system intent, or processed as a different state within the current conversation flow.

[0091] In some embodiments, as part of this process, the digital assistant determines whether the user input utterance explicitly identifies a skill bot using its invocation name. If the invocation name is present in the user input, the invocation name is considered an explicit invocation of the skill bot corresponding to the invocation name. In this scenario, the digital assistant can route the user input to the explicitly invoked skill bot for further processing. In some embodiments, if no specific invocation or explicit invocation is present, the digital assistant evaluates the received user input utterance and calculates a confidence score for the system intent and skill bot associated with the digital assistant. The score calculated for the skill bot or system intent indicates how likely the user input represents a task that the skill bot is configured to perform or represents a system intent. Any system intent or skill bot whose associated calculated confidence score exceeds a threshold (e.g., a confidence threshold routing parameter) is selected as a candidate for further evaluation. The digital assistant then selects a specific system intent or skill bot from the identified candidates for further processing of the user input utterance. In some embodiments, after one or more skill bots are identified as candidates, intents associated with those candidate skills are evaluated (using a model trained for each skill), and a confidence score is determined for each intent. Typically, any intent with a confidence score exceeding a threshold (e.g., 70%) is considered a candidate intent. If a specific skill bot is selected, the user's utterance is routed to that skill bot for further processing. If a system intent is selected, the main bot itself performs one or more actions based on the selected system intent.

[0092] Figure 3 is a simplified block diagram of a master robot (MB) system 300 according to some embodiments. The MB system 300 may be implemented in software only, hardware only, or a combination of hardware and software. The MB system 300 includes a preprocessing subsystem 310, multiple intention subsystems (MIS) 320, an explicit invocation subsystem (EIS) 330, a skill robot invoker 340, and a data storage 350. The MB system 300 depicted in Figure 3 is merely an example of the component arrangement in the master robot. Those skilled in the art will recognize many possible variations, alternatives, and modifications. For example, in some embodiments, the MB system 300 may have more or fewer systems or components than those shown in Figure 3, may combine two or more subsystems, or may have different subsystem configurations or arrangements.

[0093] The preprocessing subsystem 310 receives the utterance “A” 302 from the user and processes it via a language detector 312 and a language parser 314. As indicated above, the utterance can be provided in various ways, including audio or text. The utterance 302 can be a sentence fragment, a complete sentence, multiple sentences, etc. The utterance 302 can include punctuation marks. For example, if the utterance 302 is provided as audio, the preprocessing subsystem 310 can use a speech-to-text converter (not shown) that inserts punctuation marks (e.g., commas, semicolons, periods, etc.) into the resulting text to convert the audio into text.

[0094] The language detector 312 detects the language of discourse 302 based on the text of discourse 302. The method of processing discourse 302 depends on the language, as each language has its own grammar and semantics. Differences between languages ​​are considered when analyzing the syntax and structure of discourse.

[0095] Language parser 314 performs syntactic analysis on utterance 302 to extract part-of-speech (POS) tags for individual linguistic units (e.g., words) within utterance 302. POS tags include, for example, nouns (NN), pronouns (PN), verbs (VB), etc. Language parser 314 can also tokenize the linguistic units of utterance 302 (e.g., convert each word into a separate token) and categorize words by inflectional forms. A lexical is the primary form of a group of words as represented in a dictionary (e.g., "run" is a lexical for run, runs, ran, running, etc.). Other types of preprocessing that language parser 314 can perform include chunking compound expressions, for example, combining "credit" and "card" into a single expression "credit_card". Language parser 314 can also identify relationships between words in utterance 302. For example, in some embodiments, language parser 314 generates a dependency tree that indicates which part of the utterance (e.g., a specific noun) is a direct object, which part of the utterance is a preposition, etc. The result of the processing performed by the language parser 314 forms the extracted information 305, which, together with the utterance 302 itself, is provided as input to the MIS 320.

[0096] As indicated above, utterance 302 may include more than one sentence. For the purpose of detecting multiple intents and explicit calls, utterance 302 may be considered a single unit, even if it comprises multiple sentences. However, in some embodiments, preprocessing may be performed, for example, by preprocessing subsystem 310, to identify individual sentences among multiple sentences for use in multiple intent analysis and explicit call analysis. Generally, whether utterance 302 is processed at the level of a single sentence or as a single unit comprising multiple sentences, the results produced by MIS 320 and EIS 330 are substantially the same.

[0097] MIS 320 determines whether utterance 302 represents multiple intents. Although MIS 320 can detect multiple intents present in utterance 302, the processing performed by MIS 320 does not involve determining whether the intent of utterance 302 matches any intents already configured for the bot. Instead, the processing to determine whether the intent of utterance 302 matches a bot intent can be performed by the intent classifier 342 of MB system 300 or by the intent classifier of a skill bot (e.g., as shown in the embodiment of FIG3). The processing performed by MIS 320 assumes the existence of a bot capable of processing utterance 302 (e.g., a specific skill bot or the master bot itself). Therefore, the processing performed by MIS 320 does not need to know which bots are in the chatbot system (e.g., the identities of skill bots registered with the master bot) or what intents have been configured for a specific bot.

[0098] To determine that utterance 302 includes multiple intentions, MIS 320 applies one or more rules from a set of rules 352 stored in data storage 350. The rules applied to utterance 302 depend on the language of utterance 302 and may include sentence patterns indicating the presence of multiple intentions. For example, a sentence pattern may include co-conjunctions connecting two parts of a sentence (e.g., conjunctions) that correspond to different intentions. If utterance 302 matches a sentence pattern, it can be inferred that utterance 302 represents multiple intentions. It should be noted that utterances with multiple intentions do not necessarily have different intentions (e.g., intentions pointing to different robots or different intentions within the same robot). Instead, utterances can have different instances of the same intention, such as “Place a pizza order using payment account X, then place a pizza order using payment account Y”.

[0099] As part of determining that utterance 302 represents multiple intentions, MIS 320 also determines which part of utterance 302 is associated with each intention. MIS 320 constructs new utterances for each intention represented in the utterance containing multiple intentions, replacing the original utterances (e.g., utterances “B” 306 and utterances “C” 308, as depicted in Figure 3). Thus, the original utterance 302 can be split into two or more individual utterances, processed one at a time. MIS 320 uses extracted information 305 and / or, based on analysis of utterance 302 itself, to determine which of the two or more utterances should be processed first. For example, MIS 320 may determine that utterance 302 contains marker words indicating that a particular intention should be processed first. The newly formed utterance corresponding to that particular intention (e.g., one of utterances 306 or 308) will be sent first for further processing by EIS 330. After the session triggered by the first utterance has ended (or has been temporarily paused), the next highest priority utterance (e.g., another utterance in utterance 306 or utterance 308) can then be sent to EIS 330 for processing.

[0100] EIS 330 determines whether a received utterance (e.g., utterance 306 or utterance 308) contains a call name for a skill bot. In some embodiments, each skill bot in the chatbot system is assigned a unique call name that distinguishes it from other skill bots in the chatbot system. A list of call names may be stored in data storage 350 as part of skill bot information 354. When a utterance contains a word that matches a call name, the utterance is considered an explicit call. If a bot is not explicitly called, the utterance received by EIS 330 is considered a non-explicit call utterance 334 and is fed into the main bot's intent classifier (e.g., intent classifier 342) to determine which bot to use to handle the utterance. In some instances, intent classifier 342 will determine that the main bot should handle the non-explicit call utterance. In other instances, intent classifier 342 will determine the skill bot to which the utterance is routed for processing.

[0101] The explicit invocation functionality provided by EIS 330 offers several advantages. It reduces the amount of processing the master robot must perform. For example, when explicit invocation is present, the master robot may not need to perform any intent classification analysis (e.g., using intent classifier 342), or may have to perform a simplified intent classification analysis to select the skill robot. Therefore, explicit invocation analysis enables the selection of a specific skill robot without resorting to intent classification analysis.

[0102] Furthermore, there may be functional overlap between multiple skill robots. This can occur, for example, if two skill robots handle overlapping or very similar intents. In such cases, the master robot may struggle to identify which of the multiple skill robots to select based solely on intent classification analysis. In this scenario, explicit invocation ensures unambiguity regarding the specific skill robot to be used.

[0103] In addition to determining whether a utterance is an explicit invocation, the EIS 330 is also responsible for determining whether any part of the utterance should be used as input to the explicitly invoked skill robot. Specifically, the EIS 330 can determine whether a portion of the utterance is irrelevant to the invocation. The EIS 330 can perform this determination by analyzing the utterance and / or analyzing the extracted information 305. The EIS 330 can send the invocation-irrelevant portion of the utterance to the invoked skill robot instead of sending the entire utterance received by the EIS 330. In some instances, the input to the invoked skill robot is simply formed by removing any invocation-related parts of the utterance. For example, “I want to order pizza using Pizza Bot” can be shortened to “I want to order pizza” because “using Pizza Bot” is related to invoking the Pizza Bot but not to any processing to be performed by the Pizza Bot. In some instances, the EIS 330 can reformat the portion to be sent to the invoked robot, for example, to form a complete sentence. Therefore, EIS 330 not only determines that an explicit call exists, but also, when an explicit call exists, determines what to send to the skill robot. In some instances, there may be no text to be input to the called robot. For example, if the utterance is "Pizza Bot," EIS 330 can determine that Pizza Bot is being called, but there is no text to be processed by Pizza Bot. In such a scenario, EIS 330 can indicate to the skill robot caller 340 that there is no content to send.

[0104] The skill robot invoker 340 invokes the skill robot in various ways. For example, the skill robot invoker 340 may invoke the robot in response to receiving an instruction 335 indicating that a specific skill robot has been selected as a result of an explicit invocation. The instruction 335 may be sent by the EIS 330 along with the input of the explicitly invoked skill robot. In this scenario, the skill robot invoker 340 relinquishes control of the session to the explicitly invoked skill robot. The explicitly invoked skill robot will determine the appropriate response to the input from the EIS 330 by treating the input as an independent utterance. For example, the response may be to perform a specific action or to start a new session in a specific state, where the initial state of the new session depends on the input sent from the EIS 330.

[0105] Another way the skill robot invoker 340 can invoke a skill robot is through implicit invocation using an intent classifier 342. The intent classifier 342 can be trained using machine learning and / or rule-based training techniques to determine the probability that a particular utterance represents a task configured to be performed by the skill robot. The intent classifier 342 is trained on different categories, one category per skill robot. For example, whenever a new skill robot is registered with the master robot, a list of example utterances associated with the new skill robot can be used to train the intent classifier 342 to determine the probability that a particular utterance represents a task the new skill robot can perform. The parameters produced as a result of this training (e.g., a set of parameter values ​​for a machine learning model) can be stored as part of the skill robot information 354.

[0106] In some embodiments, the intent classifier 342 is implemented using a machine learning model, as described further in detail herein. Training the machine learning model may involve taking input from at least one subset of example utterances associated with various skill robots to generate an inference as the output of the machine learning model about which robot is the correct robot for processing any particular training utterance. For each training utterance, an indication of the correct robot for the training utterance can be provided as ground fact information. The behavior of the machine learning model can then be adapted (e.g., through backpropagation) to minimize the discrepancy between the generated inference and the ground fact information.

[0107] In some embodiments, the intent classifier 342 determines a confidence score for each skill robot registered with the master robot, indicating the likelihood that the skill robot can handle a utterance (e.g., a non-explicit invocation utterance 334 received from EIS 330). The intent classifier 342 may also determine a confidence score for each configured system-level intent (e.g., help, exit). If a particular confidence score meets one or more conditions, the skill robot invoker 340 invokes the robot associated with that particular confidence score. For example, a threshold confidence score value may need to be met. Therefore, the output 345 of the intent classifier 342 is an identification of a system intent or an identification of a particular skill robot. In some embodiments, in addition to meeting a threshold confidence score value, the confidence score must exceed the next highest confidence score by a certain margin. Imposing such a condition when the confidence scores of multiple skill robots each exceed the threshold confidence score value will enable routing to a particular skill robot.

[0108] After identifying the bot based on an assessment of the confidence score, the skill bot invoker 340 hands over processing to the identified bot. In the case of system intent, the identified bot is the master bot. Otherwise, the identified bot is a skill bot. Further, the skill bot invoker 340 determines what to provide as input 347 to the identified bot. As indicated above, in the case of explicit invoking, input 347 may be based on a portion of the utterance not associated with the invoking, or input 347 may be nothing (e.g., an empty string). In the case of implicit invoking, input 347 may be the entire utterance.

[0109] Data storage 350 includes one or more computing devices that store data used by various subsystems of the main robot system 300. As explained above, data storage 350 includes rules 352 and skill robot information 354. Rules 352 include, for example, rules for determining, by MIS 320, when a utterance represents multiple intents and how to break down utterances representing multiple intents. Rules 352 further include rules for determining, by EIS 330, which parts of a utterance that explicitly invokes a skill robot should be sent to the skill robot. Skill robot information 354 includes the invocation names of skill robots in the chatbot system, for example, a list of invocation names of all skill robots registered with a particular main robot. Skill robot information 354 may also include information used by intent classifier 342 to determine the confidence score of each skill robot in the chatbot system, for example, parameters of a machine learning model.

[0110] Figure 4 is a simplified block diagram of a skill robot system 400 according to some embodiments. The skill robot system 400 is a computing system that can be implemented solely in software, solely in hardware, or a combination of hardware and software. In some embodiments of the embodiments depicted in Figure 2, the skill robot system 400 can be used to implement one or more skill robots within a digital assistant.

[0111] The Skill Robot System 400 includes a Mindset Information System (MIS) 410, an Intent Classifier 420, and a Session Manager 430. The MIS 410 is similar to the MIS 320 in Figure 3 and provides similar functionality, including operablely using rules 452 in the data storage 450 to determine: (1) whether the utterance represents multiple intents, and if so, (2) how to split the utterance into separate utterances for each of the multiple intents. In some embodiments, the rules applied by the MIS 410 for detecting multiple intents and for splitting utterances are the same as those applied by the MIS 220. The MIS 410 receives utterances 402 and extracted information 404. The extracted information 404 is similar to the extracted information 205 in Figure 1 and can be generated using either a language parser 214 or a language parser native to the Skill Robot System 400.

[0112] The intent classifier 420 can be trained in a similar manner to the intent classifier 342 discussed above in conjunction with the embodiment of FIG3, and will be described in further detail herein. For example, in some embodiments, the intent classifier 420 is implemented using a machine learning model. For a specific skill robot, the machine learning model of the intent classifier 420 is trained using at least a subset of example utterances associated with that specific skill robot as training utterances. The underlying fact of each training utterance will be the specific robot intent associated with the training utterance.

[0113] Phrases 402 can be received directly from the user or supplied via a master bot. When phrases 402 are supplied via a master bot (e.g., as a result of processing by MIS 320 and EIS 330 in the embodiment depicted in Figure 3), MIS 410 can be bypassed to avoid repeating processing already performed by MIS 320. However, if phrases 402 are received directly from the user (e.g., during a session after routing to a skill bot), MIS 410 can process phrases 402 to determine whether phrases 402 represent multiple intents. If so, MIS 410 applies one or more rules to split phrases 402 into separate phrases for each intent, such as phrases “D” 406 and “E” 408. If phrases 402 do not represent multiple intents, MIS 410 forwards phrases 402 to intent classifier 420 for intent classification without splitting phrases 402.

[0114] Intent classifier 420 is configured to match received utterances (e.g., utterances 406 or 408) with intents associated with the skill robot system 400. As explained above, a skill robot may be configured with one or more intents, each including at least one example utterance associated with that intent and used to train the classifier. In the embodiment of FIG. 2, intent classifier 242 of the main robot system 200 is trained to determine confidence scores for individual skill robots and confidence scores for system intents. Similarly, intent classifier 420 may be trained to determine confidence scores for each intent associated with the skill robot system 400. The classification performed by intent classifier 342 is at the robot level, while the classification performed by intent classifier 420 is at the intent level and is therefore more granular. Intent classifier 420 may access intent information 454. For each intent associated with the skill robot system 400, intent information 454 includes a list of utterances that represent the intent and explain its meaning and are generally associated with a task that can be performed by that intent. Intent information 454 may further include parameters generated as a result of training on the list of utterances.

[0115] The session manager 430 receives an indication 422 for a specific intent as the output of the intent classifier 420, which identifies the indication as the best match for the utterance input to the intent classifier 420. In some instances, the intent classifier 420 may not determine any match. For example, if the utterance points to a system intent or an intent of a different skill robot, the confidence score calculated by the intent classifier 420 may drop below a threshold confidence score value. When this occurs, the skill robot system 400 may transfer the utterance to the main robot for processing, for example, by routing it to a different skill robot. However, if the intent classifier 420 is successful in identifying the intent within a skill robot, the session manager 430 will initiate a session with the user.

[0116] A session initiated by session manager 430 is a session specific to an intent identified by intent classifier 420. For example, session manager 430 may be implemented using a state machine configured to perform a dialogue flow in response to an identified intent. The state machine may include a default starting state (e.g., when an intent is invoked without any additional input) and one or more additional states, each associated with an action to be performed by the skill robot (e.g., performing a purchase transaction) and / or a dialogue to be presented to the user (e.g., a question, a response). Thus, session manager 430 may determine an action / dialogue 435 upon receiving an indication 422 that an intent has been identified, and may determine additional actions or dialogues in response to subsequent utterances received during the session.

[0117] Data storage 450 includes one or more computing devices that store data used by various subsystems of the skill robot system 400. As depicted in FIG4, data storage 450 may include rules 452 and intent information 454. In some embodiments, data storage 450 may be integrated into the data storage of the main robot or digital assistant, such as data storage 250 in FIG2.

[0118] Routing behavior of digital assistants and skill robots

[0119] When a user inputs a phrase into a digital assistant (or master bot), the digital assistant determines how to route the conversation to a given skill, a different state in the current stream, or the digital assistant's built-in intent, as described with respect to Figures 2 through 4. At the heart of the routing model is a confidence score, computed for each skill and intent to measure how well they match the user's input (e.g., providing information about the reliability of the prediction). Confidence scores are obtained by applying underlying Natural Language Processing (NLP) algorithms to the skills and the digital assistant's input data (e.g., utterances). Confidence scores can be obtained using any confidence measurement technique or estimator (e.g., Bayesian models, distance-based loss, adversarial training, etc.). For example, in classification, the confidence of each prediction may already be estimated as part of the classification process. Intuitive estimation methods use the uncertainty between categories to assess the distinct properties of each prediction, expressed as a posterior probability or distance to the separating hyperplane. In contrast, confidence estimators for regression can leverage the properties of the training data or characteristics of the machine learning model. Then, a routing decision is made by measuring the confidence score against the values ​​of various routing parameters such as candidate skill confidence threshold and confidence margin, and optionally by applying one or more rules to the route.

[0120] For example, regarding parsing intent, a skill-associated machine learning model can evaluate user messages and return what the model determines as a top-level label (intent) and a confidence score for the runner-up. In a conversational context (e.g., conversational AI), the top-level label is parsed as the intent to begin a conversation with the user. Therefore, based on model training and user messages, for example, where the model has 80% confidence that intent A is a good match, 60% confidence for intent B, and 45% confidence for intent C, the model would likely be confident that the user wants intent A. But what if the highest-scoring label only has 30% confidence that this is what the user wants? Would the developers risk letting the model follow that intent, or would they cautiously assume the model cannot predict what the user wants (unparsed intent) and / or show the user a message that rephrases the request? To help the intent model make decisions about which intents to consider matching the user's utterance, in a conversational context, digital assistants use a setting called a confidence threshold. The intent model evaluates the user's utterance for all intents and assigns a confidence score to each intent. The confidence threshold is a value that marks this line within the range of possible confidence scores: below this line, the intent is considered not to correspond to the utterance at all; and above this line, the intent is considered a candidate intent to start a conversation. Although this example is specific to parsing intent, it should be understood that confidence scores can be obtained and used to parse various types of inferences about digital assistants, such as which skill to use to process a message / utterance.

[0121] The routing model comprises three layers: identifying candidate system intents (e.g., using a candidate system intent model), identifying candidate skills (e.g., using a candidate skill model), and identifying candidate flows (e.g., using a candidate flow model). Regarding identifying candidate system intents: User input is evaluated and confidence scores are applied to the digital assistant's intents (exit, help, and unresolved intents). Any intent whose confidence score exceeds the value of the digital assistant's built-in system intent confidence threshold routing parameter is considered a candidate for further evaluation. Regarding identifying candidate skills: User input is evaluated and confidence scores are applied to each skill. Any skill whose confidence score exceeds the value of the digital assistant's candidate skill confidence threshold routing parameter is considered a candidate skill for further evaluation. Regarding identifying candidate flows: After identifying candidate skills, each intent within these skills is evaluated (according to the intent model for each skill) and a confidence score is applied to each intent. Generally, any intent whose confidence score exceeds the value of its skill's confidence threshold routing parameter (not the digital assistant's candidate skill confidence threshold parameter) is considered a candidate flow.

[0122] The behavior of this routing can be tuned by adjusting the routing parameters of the digital assistant. Depending on the skills (and their intentions) within the digital assistant, users may need to adjust the values ​​of the digital assistant's routing parameters to better control how the digital assistant responds to user input. Routing parameters can be configured to take values ​​from 0 (0% confidence) to 1 (100% confidence). Below is a summary of possible digital assistant routing parameters:

[0123] • Built-in system intent confidence threshold: The minimum confidence score required to match a built-in system intent (such as help and exit). The default value can be, for example, 0.9 (90% confidence).

[0124] • Candidate skill confidence threshold: The minimum confidence score required to match a candidate skill. The default value can be, for example, 0.4 (40% confidence).

[0125] • Confidence Margin: The maximum difference between the confidence score of the best candidate skill and the confidence score of any of the lower-ranked candidate skills being considered (which also exceed the confidence threshold). Built-in digital assistant intents (help, exit, and unresolvedIntent) can also be considered. The default value can be, for example, 0.1 (10% confidence). The individual confidence margin parameter for a skill works in the same way, except that it applies to the confidence score of intents within the skill.

[0126] • Consider full threshold: The minimum confidence score required to consider all matching intents and streams. This value also takes precedence over win margin. (If such a high confidence level exists, it is considered impossible to determine which stream the user wants to use.) The default value could be, for example, 0.8 (80% confidence).

[0127] • Consider only current context threshold: The minimum confidence score required when considering only the current skill. If user input matches an intent higher than this threshold, other intents are ignored even if they reach the confidence threshold. This setting is useful for preventing disambiguation suggestions for user input that matches intents from multiple skills well. For example, a user input "cancel order" might match intents well from multiple food delivery skills. A default value could be, for example, 0.8 (80% confidence).

[0128] • Explicit Invocation Confidence Threshold: The minimum confidence score required to match an input containing an explicit invocation of the skill. The default value could be, for example, 0.8 (80% confidence).

[0129] • Exit Prompt Confidence Threshold: The minimum confidence score required to exit without prompting the user for confirmation. The default value can be, for example, 1.01, which is nominally set outside the 0-1 confidence threshold range and ensures that a confirmation prompt will always be displayed. However, if the skill designer wants users to exit without confirmation when the exit confidence score is high, this threshold can be lowered.

[0130] In addition to digital assistant routing parameters, routing behavior can be further tuned by adjusting skill routing parameters. Below is a summary of possible skill routing parameters:

[0131] • Confidence Threshold: The minimum confidence score required to match the skill's intent with the user's input. If no match is found, the transition action is set to unresolved intent. The default value could be, for example, 0.7 (70% confidence).

[0132] • Confidence Margin: If the highest-ranked intent exceeds the confidence threshold, only the best intent exceeding the confidence threshold is selected. If the scores of other intents exceeding the confidence threshold differ from the score of the best intent by less than this margin, these intents are also presented to the user. The default value can be, for example, 0.1 (10% confidence).

[0133] Additionally, rules can be generated for various scenarios that influence routing formulas. For example, rules can include: explicit invocation, context-aware routing, and context pinning. Regarding explicit invocation: if a user includes the name of a skill in their input, the digital assistant will route directly to that skill, even if the input matches well with other skills. Regarding context-aware routing: if the user has already engaged with a skill, that skill is given more weight during intent resolution than intents from other skills. Regarding context pinning: if the user input includes an explicit invocation of a skill but no intent-related utterance, the router "pins" the session to the skill. This means assuming that subsequent utterances are related to the skill.

[0134] Explicit invocation occurs when a user enters the name of a skill as part of their input. By using explicit invocation, users help ensure their input is directly routed to the intended skill, thus reducing the number of interactions with the digital assistant necessary to complete their task. When using explicit invocation, additional weight is given to the corresponding skill when determining the route: if the user is not already in a skill and enters an explicit invocation, that invocation takes precedence over other flows in the digital assistant context. If the user is in a flow of a different skill, the digital assistant will always attempt to confirm that the user actually wants to switch skills. Within each digital assistant, users, such as skill developers, can determine the invocation name they want to use for a given skill. Users can set the invocation name on the skill's digital assistant configuration page. This behavior can be supported by an explicit invocation confidence threshold routing parameter. If the confidence score of an explicit invocation exceeds this threshold, intents from other skills are not considered in the routing decision. In some instances, the default value for this threshold is set to 0.8 (80% confidence).

[0135] Routing in digital assistants is also context-aware, meaning that during intent resolution, matching intents from the skill the user is currently engaged with are given more weight than intents from other skills. For example, consider a digital assistant with banking and online retail skills. If a user enters the question "What's my balance?", this might apply to both the user's bank account balance and the balance on a gift card registered with an online retailer. If a user, such as a customer, enters this question before entering the context of either skill, the digital assistant should give them the choice of which "balance" stream (in the banking or retailer skill) to enter. However, if the user enters the question within the banking skill, the digital assistant should automatically select the "balance" stream corresponding to the banking skill (ignoring intents from other skills, even if they meet the standard confidence threshold routing parameter). Context awareness is supported by a threshold routing parameter that only considers the current context. If the confidence score of an intent in the current context exceeds this threshold, intents from other contexts are not considered in the routing decision. In some instances, the default value for this threshold is 0.8 (80% confidence), because users may want to be fairly certain that the intent in the current context is the correct intent before canceling the display of other intents outside the current context.

[0136] In addition to the intents from the skill the user is currently engaging with, there are common intents across multiple or all skills, such as help or unresolved intents. For example, within the context of a skill, if user input matches a help system intent, the user is routed to the help flow determined by that skill (rather than the flow determined at the digital intent level). More specifically, if the user engages with a skill and types help, help will be provided for that skill, not for the entire digital assistant. The behavior is different for unresolved intent system intents. If user input resolves to an unresolved intent (and there are no other matching intents within the skill), the input is considered an unresolved intent at the digital assistant level. However, if an unresolved intent is simply one of the matching intents within a skill, the skill processes the response. This behavior is supported by a built-in system intent confidence threshold routing parameter. If one of these intents scores a confidence score above this threshold, the intent is considered a candidate for further evaluation. In some instances, the default value for this threshold is 0.9 (90% confidence).

[0137] As discussed herein, assigning more weight to matching intents from the skill a user is currently interacting with than to intents from other skills during intent resolution can lead to inappropriate routing, especially for out-of-control (OOS) utterances. It is important that chatbot systems be able to recognize such OOS utterances so that appropriate responses can be taken. Therefore, according to various aspects of this disclosure, context awareness can also consider skill groups. This means that when a skill is defined as part of a skill group and that skill is in the current context, the current context also includes other skills within that skill group. As discussed herein, for skill domains containing a large number of functions, it is often desirable to divide those functions into multiple specialized skills. This is particularly useful from a developmental perspective. Different teams can work on different aspects of a skill and release skills and their updates on the timeline that best suits them. When multiple skills exist in a domain, users such as customers may need to switch between these skills relatively frequently. For example, in a single session within a digital assistant containing multiple HR-related skills, a user might make requests related to skills concerning payroll, personal information, and vacation time. To optimize routing behavior between related skills, users such as skill developers can define skill groups. Within the digital assistant, all skills within a group are treated as a single logical skill. Therefore, all skills in the group are considered part of the current context, and thus all their intentions are weighted equally during intention resolution.

[0138] When defining skill groups for a digital assistant, the routing engine records both the skill context and the group context. If the routing engine determines that another skill in the group is better suited to handle a user request, it switches the skill context within the group. This determination is based on the skill ranking of the group in the candidate skill model. In some instances, if the confidence score of the best candidate skill in the group is less than 5% higher than the confidence score of the current skill, the skill context within the group is not changed.

[0139] To define skill groups, each skill group should be defined as a collection of skills that are linguistically related within the same domain. Skills within a group should be categorized by function. For example, it might make sense to group skills related to benefits, compensation, absences, personal information, and recruitment into an HCM skill group. Skills related to opportunities and accounts might belong to a sales skill group. Skills can be associated with a group using identifiers such as the group name or labels. In some instances, to organize skill groups and prevent naming conflicts, users can use...<company name> . <domain>The pattern is used for the name of the skill group. For example, a user could create a group called acme.hcm for the following HCM skills of a hypothetical company, Acme.

[0140] οwelfare

[0141] Salary

[0142] Absence

[0143] personal information

[0144] ο Recruitment

[0145] Similarly, if the assumed Acme also possesses the following skills in the sales field, then the user can use acme.sales as a skills group:

[0146] ο opportunity

[0147] ο account

[0148] If developers want to create digital assistants with common skills for functions like helping or handling small talk, they may not want to treat common skills as separate skill groups because they could be invoked at any time during a session, regardless of which skill group a user primarily interacts with. And once invoked, developers may want to ensure users don't get bogged down in these common skills. To ensure other skill groups are given the same weight as the common skill after completing their interaction, developers can include the common skill in a set of groups. Developers can do this by including an asterisk (*) or a wildcard in the group name of the common skill. For example, if a developer uses acme.* as the skill group name, any skills in the acme.hcm and acme.sales groups will be included, but no skills in the group called hooli.hcm will be included. Alternatively, if a developer uses * as the skill group name, all groups will be included (although not any skills not assigned to a group). When a user navigates from a skill in a simple group (a group whose name does not contain an asterisk) to a skill whose group name contains an asterisk or a wildcard, the group context remains the same as before navigating to that skill. For example, if a user's session moves from a skill in the acme.hcm group to a skill in the acme.* group, the group context will remain acme.hcm.

[0149] Continuing with the example of user-created groups above, the following are examples of context awareness and routing within and between skill groups:

[0150] When a user asks, "What benefits do I qualify for?", the skill context is "Benefits Skill" and the group context is "acme.hcm". When the user then asks, "What is my salary?", the skill context changes to "Salary" and the group context remains "acme.hcm".

[0151] The user's current context is Welfare Skills, meaning their current group context is acme.hcm. The user asks, "What sales opportunities are there?" This request is outside the domain not only for the current skill but also for all skills in the HCM group (although "opportunities" might provide a potential match for recruiting skills). The user is routed to the best match in the acme.sales group context, namely "opportunities".

[0152] When a user enters "what are my benefits", the benefits skill is invoked as part of the acme.hcm group.

[0153] The user's context is welfare skills and the acme.hcm group.

[0154] When a user enters "Tell me a joke", this invokes the generic chat skill assigned to the acme.* group.

[0155] The user is now in the chat skills context. The group context is now any group that matches acme.*. This includes acme.hcm (which includes previously invoked welfare skills) and acme.sales, which consists of opportunity skills and account skills.

[0156] The user asked, "What are my benefits?" and then added, "I have another question."

[0157] The user is in the acme.hcm context because they were previously in that context due to a question about welfare, but has now been routed to the misc.another.question intent in the Miscellaneous skill as a member of the acme.* group.

[0158] • When a user navigates to a skill that belongs to a group name including an asterisk (*), the user's group context remains unchanged until they are routed to a skill belonging to the * group (e.g., acme.hcm in this example).

[0159] The user is currently in the context of a skill called Miscellaneous, which provides public functionality. This skill belongs to the acme.* group, meaning the user's current group context is all acme groups (acme.sales and acme.hcm). The current skill context is Miscellaneous. When the user enters "What benefits do I qualify for?", the current skill context changes to benefits belonging to the acme.hcm group.

[0160] As shown in Figure 5A, to define a skill group, users such as skill developers can navigate to the graphical user interface 500 of the skill configuration page and enter a group name 505 in the group field 510 for skill 515. Once skill 515 is added to the digital assistant, any other skill in the digital assistant with that group name 505 will be considered part of the same skill group. If a skill has already been added to the digital assistant, the user can set the skill group in the digital assistant (or override the group specified in the skill settings). As shown in Figure 5B, to set a skill group in the digital assistant, the user can navigate to the graphical user interface 520 of the digital assistant configuration page and enter a group name 525 in the group field 530 for skill 535.

[0161] Routing in digital assistants can also be context-fixed. If user input includes an explicit call to a skill but no intent-related utterance, the router "fixes" the session to the skill for the next user input. This means it assumes the following utterance is relevant to that skill, so the router doesn't consider any intents from different skills. If the user then inputs something unrelated to that skill, the router treats it as an unresolved intent within that skill, even if it would match well with intents from different skills. (Exit intents are an exception; they are always considered.) After this, the router removes the fix. Therefore, if the user then repeats the input or inputs something else unrelated to the fixed context, all flows are considered again.

[0162] Consider this example of how context-fixed routing works when the user behaves as expected:

[0163] The user enters "Go to Pizza Skill," which is an explicit call to the Pizza Skill.

[0164] (Including the skill name in the discourse makes it an explicit invocation.)

[0165] At this point, the conversation is pinned to the pizza skill, meaning the digital assistant will only look for matches within the pizza skill category.

[0166] Then, she typed "I want to place an order".

[0167] The digital assistant finds a match with the pizza ordering intent in the pizza skill and initiates the pizza ordering process.

[0168] At this point, remove the fixation.

[0169] And below is an example of how context-fixed routing should work when a user continues in an unexpected way:

[0170] The user enters "Go to Pizza Skill," which is an explicit call to the Pizza Skill.

[0171] At this point, the conversation is fixed to the pizza skill.

[0172] Then, she typed "transfer money".

[0173] The input does not match anything within the Pizza skill, so the router treats it as an unresolved intent within the Pizza skill (and, depending on how the flow of unresolved intents is designed, requires the user to clarify). Intents from other skills (such as the Finance skill) are ignored, even if they would provide a suitable match.

[0174] Remove the fixed attribute from the Pizza skill.

[0175] She repeated her transfer request.

[0176] Find a match in your financial skills and start the money transfer flow.

[0177] Figure 6 illustrates an example 600 where a digital assistant evaluates user input and routes a session to a specific flow. In this example, the user provides the input / utterance "order pizza," and the digital assistant's initial response is "How old are you?" The response "How old are you?" from the digital assistant indicates the start of the pizza ordering flow for the pizza skill (which requires the user to be 18 years of age or older to order pizza). As shown in Figure 6, the digital assistant finds a strong match (100%) for the pizza skill and a weak match (21.56%) for the retail skill. At the first level 605—identifying candidate system intents—no match is found for any system intent. At the second level 610—identifying candidate skills—a strong match (100%) for the pizza skill and a weak match (21.56%) for the retail skill are found. Since the match for the pizza skill exceeds the candidate skill confidence threshold (40%), the digital assistant evaluates the flow in the pizza skill. As explained in this article, skill designers can adjust the confidence threshold value for candidate skills in the digital assistant's configuration settings. In the third layer, 615—identifying candidate flows—in the pizza skill, the digital assistant finds a match for the intent to order pizza. Because this match exceeds the confidence threshold for the flow in the pizza skill (and there are no other qualified matches to consider), the pizza ordering flow begins.

[0178] Figure 7 illustrates example 700 when prompting the user to clarify their intent. In this example, the user provides the input / phrase "what is my balance," and the digital assistant's initial response is "Do you want to go to: Gift Card Balances, Balances in Financial Account, or None of the Above?" As will be understood, the digital assistant is unsure of what the user wants to do, therefore it provides a prompt requiring the user to choose from several options (disambiguation). As shown in Figure 7, at the first level 705—identifying candidate system intents—no match is found for any system intent. At the second level 710—identifying candidate skills—both "financial skills" and "retail skills" receive high scores (100%). And then for each of these skills, at the third level 615—identifying candidate streams—the digital assistant identifies candidate streams that also receive high scores (also 100%). Since the candidate streams for GiftCardBalance and Balances exceed the confidence threshold, and because the difference between their scores is less than the confidence margin value (10%), the digital assistant asks the user to choose between these intents.

[0179] Figure 8 illustrates an example 800 where the application of a rule affects routing behavior. In this example, the user provides the input / utterance "what is my balance?", and the digital assistant's initial response is "Do you want to go to: Gift Card Balances, Balances in Financial Account, or None of the Above?" As will be understood, the digital assistant is unsure of what the user wants to do, so it provides a prompt requiring the user to choose from several options (disambiguation). In response, the user requests "Balances in financial account," and the digital assistant again responds with a prompt requiring the user to choose from several options including savings, current account, and credit card. In response, the user requests "checking," and the digital assistant responds with the balance in the user's current account. Subsequently, the user requests "now check balance for a gift card," and the digital assistant responds with "What is your gift card number?" In this scenario, the user has started using the digital assistant to check her financial account balance in the financial skill, but then decides to inquire about her gift card balance in the retail skill. Because she used an explicit call (invoked by its call name, which is also a gift card or retail skill, and set on the skill-specific page within the digital assistant), the digital assistant gives a preference for the retail skill when attempting to resolve the intent, even though the user is in the context of the financial skill. The routing rule applied here is explicit call—whereby the digital assistant will directly route to that skill if the user includes the skill's call name in her input, even if the input matches well with other skills. As shown in Figure 8, there is a match for the current context 805, but it is ignored. The match (100%) for the explicit call 810 to the GiftCardBalance skill wins.

[0180] Figure 9 illustrates an example 900 where context awareness influences routing behavior. In this example, the user provides the input / utterance "what is my balance?", and the digital assistant's initial response is "Do you want to go to: Gift Card Balances, Balances in Financial Account, or None of the Above?" As will be understood, the digital assistant is unsure of what the user wants to do, so it provides a prompt requiring the user to choose from several options (disambiguation). In response, the user requests "Balances in financial account," and the digital assistant again responds with a prompt requiring the user to choose from several options including savings, checking account, and credit card. In response, the user requests "checking," and the digital assistant responds with the balance in the user's checking account. Subsequently, the user requests "what is my balance?", and the digital assistant again responds with several options including savings, checking account, and credit card. In this scenario, the user begins with the question "what is my balance?", goes through prompts to disambiguate between financial and retail skills, and ultimately obtains their current account balance. They then type "what is my balance?" again, but this time do not need to navigate through any disambiguation prompts that differentiate between financial and retail skills. The information in the routing table shown in Figure 9 explains how the current context can be set by the context-aware router and used to avoid some disambiguation prompts. The applied current context rule is—if the current context flow matches with a very high confidence, other skill flows are ignored. Therefore, even if a matching intent exists from retail skill 905, it is ignored. The intent invocation section shows all matching intents, but the entry for the Balances intent 915, which contains only financial skills and is set by the context-aware router for the "current context" 910, is decisive.

[0181] Figures 10A through 10C illustrate example 1000 where context awareness takes skill groups into account and influences routing behavior. First, Figure 10A shows the routing rule section 1005, where routing rule 1010 is output when the skill context changes due to group stickiness. When the skill context changes due to skill group assignment, the original rule "consider flows from all models" changes to "the current context switches to the highest confidence in the skill group when the skill context changes" and "the current context flow matches with very high confidence. Ignore other skills." Second, Figure 10B shows the routing intent invocation section and updates the routing table to display the group context 1015, which includes the skill intent output assignment for the current context 1020 and candidate flow models 1025. In one instance, the group context 1015 is also shown in the 'current context' row because the group context can differ from the skill group (e.g., when routing to a common skill belonging to multiple groups). Finally, Figure 10C shows route summary line 1030, where group context 1015, displayed after the flow to be routed 1035, is selected by the digital assistant.

[0182] It will be understood that a skill, as described herein, can be equivalently expressed as a classifier trained to specifically handle a particular type of natural language input (i.e., input relating to a specific region or domain of the skill). The candidate skill model discussed above thus serves as a classifier selection mechanism for identifying the classifier best suited to recognize the utterance and generate appropriate output (e.g., text or speech output). Therefore, when deployed as described herein, the candidate skill model can provide improved speech recognition tools because its selection is based on a domain expert classifier (skill) specifically trained to analyze input speech or text in the domain recognized by the candidate skill model. In this embodiment, the utterance is in an audio format (e.g., an audio file containing speech), allowing the candidate skill model to be used to improve speech recognition because it can better map the speech input to the text output—a text response provided by the skill (classifier) ​​selected by the candidate skill model. Similar advantages can be achieved using text input.

[0183] Illustrative use cases and techniques for context-aware routing with skill groups

[0184] For the following use case, assuming that in the digital assistant configuration page, if Entity_1 has skills in HCM, Sales, and Public Domain, the skill developer can specify each group to which the skill can belong as follows:

[0185] • HCM domain skills will assign the 'Entity_1.hcm' group tag to the following skills.

[0186] οwelfare

[0187] Salary

[0188] Absence

[0189] personal information

[0190] ο Recruitment

[0191] • Sales skills will be assigned the 'Entity_1.sales' group tag for the following skills.

[0192] ο opportunity

[0193] ο account

[0194] Public skills that will become part of all Entity_1 groups will have a ('Entity_1.*') tag for the following skills. If '*' is specified, the skill will be part of all groups.

[0195] οsystem

[0196] ο Casual chat

[0197] In the first use case, the user is in the benefits skill and provides the input / utterance "What was my salary last year?" (i.e., salary skill input), and the digital assistant's context-aware router has set the current skill context to 'benefits' and the group context to 'Entity_1.hcm'. At the first layer—identifying candidate system intents—the digital assistant's candidate system intent model determines that there is no match for any system intent. At the second layer—identifying candidate skills—the digital assistant's candidate skill model ranks salary skills higher than benefits skills. In response, the digital assistant's context-aware router sets the current skill context to 'salary' and maintains the group context at 'Entity_1.hcm'. Then, for candidate skills, at the third layer—identifying candidate flows—the candidate flow model identifies candidate flows (i.e., intents) within the current skill context and the high-scoring group context, specifically the flow for obtaining salary.

[0198] In the second use case, the user is in the Welfare skill and provides the input / utterance "What sales opportunities are there?" (i.e., opportunity skill input), and the digital assistant's context-aware router has set the current skill context to 'Welfare' and the group context to 'Entity_1.hcm'. At the first layer—identifying candidate system intents—the digital assistant's candidate system intent model determines that there is no match for any system intent. At the second layer—identifying candidate skills—the digital assistant's candidate skill model identifies the utterance for opportunity as out-of-domain (specifically, out of scope) for both the Welfare skill and the skill group falling into 'Entity_1.hcm'. Once the current skill and group context are excluded from competition, the candidate skill model identifies the opportunity skill as the highest-ranking skill. In response, the digital assistant's context-aware router sets the current skill context to 'Opportunity' and changes the group context to Entity_1.sales. Then, for candidate skills, at the third layer—identifying candidate flows—the candidate flow model identifies candidate flows (i.e., intents) within the current skill context and the high-scoring group context, i.e., flows used to provide available sales opportunities.

[0199] In the third use case, the user is in a system skill and provides the input / utterance "What benefits do I qualify for?" (i.e., benefits skill input), and the digital assistant's context-aware router has set the current skill context to 'system' and the group context to Entity_1.* = 'Entity_1.hcm', 'Entity_1.sales'. At the first layer—identifying candidate system intents—the digital assistant's candidate system intent model determines that there is no match for any system intent. At the second layer—identifying candidate skills—the digital assistant's candidate skill model ranks benefits skills higher than any system skill. In response, the digital assistant's context-aware router sets the current skill context to 'benefits' and changes the group context to 'Entity_1.hcm'. And then, for candidate skills, at the third layer—identifying candidate flows—the candidate flow model identifies candidate flows (i.e., intents) within the current skill context and the high-scoring group context, i.e., flows used to provide qualifying benefits.

[0200] In the fourth use case, the user is in the Welfare skill and provides the input / utterance "Can I get an atelephone number for ABC?" (where ABC is an account name, but this matches 'Personal Information'), and the digital assistant's context-aware router has set the current skill context to 'Welfare' and the group context to 'Entity_1.hcm'. At the first layer—identifying candidate system intents—the digital assistant's candidate system intent model determines that there is no match for any system intent. At the second layer—identifying candidate skills—the digital assistant's candidate skill model identifies the first two matches as the Account skill and Personal Information skill within the Win Margin parameter. The candidate skill model will rank the Personal Information skill higher because it provides compliance with the skill within the current group context 'Entity_1.hcm', and the digital assistant's context-aware router maintains the group context at 'Entity_1.hcm'. Then, for candidate skills, at the third level—determining candidate flows—if both account skills and personal information skills rank within the win margin parameter, the candidate flow model will identify the disambiguation flow (i.e., intent) used to prompt the user and differentiate between account skills and personal information skills.

[0201] In the fifth use case, the user is in the Personal Information skill and provides the input / utterance "Can I get an atelephone number for ABC?" (where ABC is the account name, but this matches 'Personal Information'), and the digital assistant's context-aware router has set the current skill context to 'Personal Information' and the group context to 'Entity_1.hcm'. At the first layer—identifying candidate system intents—the digital assistant's candidate system intent model determines that there is no match for any system intent. At the second layer—identifying candidate skills—the digital assistant's candidate skill model identifies the first two matches as the Account Skill and Personal Information skill within the win margin parameter. However, in this instance, the Personal Information match > 0.8, and therefore exceeds the current context threshold parameter. The candidate skill model is ignored, and the digital assistant's context-aware router maintains the skill context as 'Personal Information' and the group context as 'Entity_1.hcm'. Then, for candidate skills, at the third layer—identifying candidate flows—the candidate flow model identifies candidate flows (i.e., intents) within the current skill context and the high-scoring group context, namely, the flow for providing a phone number under Personal Information. It should be noted that in this example, the user did not get what they intended to get—the utterance "Can I get phone number for ABC account?" would have been more effective.

[0202] In the sixth use case, the user is in the Personal Information skill and provides the input / utterance "Can I get an atelephone number for ABC?" (where ABC is a name), and the digital assistant's context-aware router has set the current skill context to 'Personal Information' and the group context to 'Entity_1.hcm'. At the first layer—identifying candidate system intents—the digital assistant's candidate system intent model determines that there is no match for any system intent. At the second layer—identifying candidate skills—the digital assistant's candidate skill model identifies the first two matches as the Account Skill and Personal Information skill within the win margin parameter. Again, in this instance, the Personal Information match > 0.8, and therefore greater than the current context threshold parameter only. Ignoring the candidate skill model, the digital assistant's context-aware router maintains the skill context as 'Personal Information' and the group context as 'Entity_1.hcm'. And then, for candidate skills, at the third layer—identifying candidate flows—the candidate flow model identifies candidate flows (i.e., intents) within the current skill context and the high-scoring group context, namely, the flow for providing a phone number under Personal Information. It should be noted that in this instance, the user did indeed receive what they intended to receive.

[0203] Figure 11 illustrates a process flow for context-aware routing with skill groups according to various embodiments. The processing depicted in Figure 11 can be implemented as software (e.g., code, instructions, programs), hardware, or a combination thereof, executed by one or more processing units (e.g., processors, cores) of a corresponding system. The software can be stored on a non-transitory storage medium (e.g., a memory device). The methods presented in Figure 11 and described below are intended to be illustrative and not restrictive. Although Figure 11 depicts various processing steps occurring in a particular sequence or order, this is not intended to be restrictive. In some alternative embodiments, the steps can be performed in a different order or some steps can be performed in parallel. In some embodiments, such as those depicted in Figures 2 through 4, the processing depicted in Figure 11 can be performed by various subsystems, models, or modules for routing dialogues (e.g., digital assistant 206, master robot 214, and skill robots 216-1, 216-2, and 216-3, etc.). Using the operational routing dialogue depicted in Figure 11 can be performed more efficiently or otherwise more effectively when routing OOS utterances.

[0204] In step 1105, utterances are received from the user within the chatbot session. The current skill context of the chatbot session can be a first skill, and the current group context of the chatbot session is a first group. The first group defines a domain space that includes a first set of skills, which includes both the first skill and the second skill. The first set of skills is associated with the first group by identifying or providing the name or label of the first group within the configuration page of each skill in the first set of skills.

[0205] In step 1110, the utterance is input into the candidate skill model of the first group. The candidate skill model may be a machine learning model trained to evaluate the utterance using one or more NLP algorithms and measure the degree of matching between the skills within the first group of skills and the user's utterance.

[0206] In step 1115, a ranking of skills potentially capable of processing utterances within the first group is obtained using a candidate skill model. Obtaining the skill rankings includes: evaluating the utterances and generating confidence scores for skills within the first group (i.e., the first set of skills); identifying any skill whose confidence score exceeds the value of a candidate skill confidence threshold routing parameter as a candidate skill for further evaluation; and ranking the candidate skills as potential utterance processors within the first group based on their confidence scores. In some instances, the candidate skill confidence threshold routing parameter is set to 40% confidence.

[0207] In step 1120, based on skill ranking, the second skill is identified as the highest-ranking skill for processing discourse and is considered a candidate skill for further evaluation. It should be understood that additional skills can also be considered candidate skills for further evaluation. For example, if a win margin parameter is configured for the digital assistant, a third skill (e.g., identified as the second-highest-ranking skill for processing discourse) might fall within the win margin parameter and therefore also be considered a candidate skill for further evaluation.

[0208] In step 1125, the current skill context of the chatbot session is changed to the second skill. It should be noted that in instances where multiple skills are identified as candidate skills for further evaluation, the current skill context of the chatbot session can be changed to include the second skill as well as all other skills identified as candidate skills. In other words, the skill context may not be limited to the highest-ranking skill.

[0209] In step 1130, the utterance is fed into a candidate stream model for the second skill. The candidate stream model can be a machine learning model trained to evaluate the utterance using one or more NLP algorithms and measure the degree of match between the intent associated with the second skill and the user's utterance. In some instances, the machine learning model is an intent classifier.

[0210] In step 1135, a ranking of intents matching the utterance within the second skill is obtained using a candidate flow model. Obtaining the skill ranking includes: evaluating the utterance and generating confidence scores for skills within the first group; identifying any skills whose confidence scores exceed the value of a candidate skill confidence threshold routing parameter as candidate skills for further evaluation; and ranking the candidate skills as skills within the first group that can potentially process the utterance based on their confidence scores. In some instances, the candidate skill confidence threshold routing parameter is set to 70% confidence.

[0211] In step 1140, based on the intention ranking, the intention that ranks highest for processing discourse is determined. It should be understood that additional skills and intentions can be considered as candidate skills and intentions for further evaluation. For example, if a margin parameter and confidence margin are configured for digital assistants and skills, a third skill (e.g., identified as the second-highest ranked skill for processing discourse) and associated intentions may also be considered as candidate skills and intentions within the margin parameter and confidence margin, and therefore also for further evaluation.

[0212] In step 1145, a conversational flow is initiated in the chatbot session with the user based on the intent that is ranked highest for processing the utterance. It should be understood that if multiple intents are identified for further evaluation (e.g., based on confidence margin), the flow can be a disambiguation flow that prompts the user for responses to clarify their utterance among multiple skills and / or intents.

[0213] Explanatory System

[0214] Figure 12 depicts a simplified diagram of distributed system 1200. In the illustrated example, distributed system 1200 includes one or more client computing devices 1202, 1204, 1206, and 1208 coupled to server 1212 via one or more communication networks 1210. Client computing devices 1202, 1204, 1206, and 1208 can be configured to execute one or more applications.

[0215] In various examples, server 1212 may be adapted to run one or more services or software applications implementing one or more embodiments described in this disclosure. In some examples, server 1212 may also provide other services or software applications that may include non-virtual and virtual environments. In some examples, these services may be provided as web-based services or cloud services (such as under a Software as a Service (SaaS) model) to users of client computing devices 1202, 1204, 1206, and / or 1208. Users operating client computing devices 1202, 1204, 1206, and / or 1208 may then use one or more client applications to interact with server 1212 to utilize the services provided by these components.

[0216] In the configuration depicted in Figure 12, server 1212 may include one or more components 1218, 1220, and 1222 that implement the functions performed by server 1212. These components may include software components that can be executed by one or more processors, hardware components, or combinations thereof. It should be understood that various different system configurations, different from distributed system 1200, are possible. Therefore, the example shown in Figure 12 is an example of a distributed system for implementing the example system and is not intended to be limiting.

[0217] Users can use client computing devices 1202, 1204, 1206, and / or 1208 to execute one or more applications, models, or chatbots, which can generate one or more events or models that can then be implemented or serviced in accordance with the teachings of this disclosure. The client devices can provide an interface that enables users of the client devices to interact with them. The client devices can also output information to users via this interface. Although Figure 12 depicts only four client computing devices, any number of client computing devices can be supported.

[0218] Client devices can include various types of computing systems, such as portable handheld devices, general-purpose computers such as personal computers and laptops, workstation computers, wearable devices, gaming systems, thin clients, various messaging devices, sensors, or other sensing devices. These computing devices can run various types and versions of software applications and operating systems (e.g., Microsoft). Apple Or a UNIX-like operating system, Linux, or a Linux-like operating system (such as Google Chrome). TM OS), including various mobile operating systems (e.g., Microsoft Windows). Windows Android TM Black Palm Portable handheld devices can include cellular phones, smartphones (e.g., ), tablet computers (e.g., Wearable devices include Google's personal digital assistants (PDAs), personal digital assistants (PDAs), and more. Head-mounted displays and other devices. Gaming systems can include various handheld gaming devices, internet-enabled gaming devices (e.g., with or without internet access). Microsoft gesture input devices Game console, Sony System, by The client device can run a variety of different applications, such as various Internet-related applications, communication applications (e.g., email applications, short message service (SMS) applications), and can use various communication protocols.

[0219] One or more networks 1210 can be any type of network familiar to those skilled in the art that supports data communication using any of the various available protocols, including but not limited to TCP / IP (Transmission Control Protocol / Internet Protocol), SNA (System Network Architecture), and IPX (Internet Packet Switching). Etc. By way of example only, one or more networks 1210 can be a local area network (LAN), an Ethernet-based network, a token ring network, a wide area network (WAN), the Internet, a virtual network, a virtual private network (VPN), an intranet, an extranet, a public switched telephone network (PSTN), an infrared network, a wireless network (e.g., according to the IEEE 1002.11 protocol suite), etc. (and / or any other wireless protocol operating on a network) and / or any combination of these networks and / or other networks.

[0220] Server 1212 may consist of: one or more general-purpose computers, dedicated server computers (including PC (personal computer) servers by way of example), Server 1212 may be a server, mid-range server, mainframe computer, rack server, etc., server group, server cluster, or any other suitable arrangement and / or combination. Server 1212 may include one or more virtual machines running a virtual operating system or other computing architectures involving virtualization, such as logical storage devices that can be virtualized to maintain one or more flexible pools of virtual storage devices for the server. In various examples, server 1212 may be adapted to run one or more services or software applications that provide the functionality described in the foregoing disclosure.

[0221] The computing system in server 1212 can run one or more operating systems, including any of the operating systems discussed above and any commercially available server operating system. Server 1212 can also run any of a variety of additional server applications and / or middleware applications, including HTTP (Hypertext Transfer Protocol) servers, FTP (File Transfer Protocol) servers, CGI (Common Gateway Interface) servers, etc. Servers, database servers, etc. Exemplary database servers include, but are not limited to, those that can be accessed from... Those database servers acquired through purchases by companies such as IBM.

[0222] In some implementations, server 1212 may include one or more applications to analyze and merge data feeds and / or event updates received from users of client computing devices 1202, 1204, 1206, and 1208. As an example, data feeds and / or event updates may include, but are not limited to, those provided in the original text. feed, The server 1212 may receive real-time updates from one or more third-party information sources and continuous data streams. These real-time updates may include real-time events related to sensor data applications, financial reporting machines, network performance measurement tools (e.g., network monitoring and traffic management applications), clickstream analysis tools, vehicle traffic monitoring, and the like. The server 1212 may also include one or more applications to display data feeds and / or real-time events via one or more display devices of client computing devices 1202, 1204, 1206, and 1208.

[0223] The distributed system 1200 may also include one or more data stores 1214, 1216. In some examples, these data stores may be used to store data and other information. For example, one or more of data stores 1214, 1216 may be used to store information (such as information related to chatbot performance or generated models) for use by the chatbot, which is used by server 1212 in performing various functions according to various embodiments. Data stores 1214, 1216 may reside in various locations. For example, the data store used by server 1212 may be local to server 1212 or may be located remotely to server 1212 and communicate with server 1212 via a network-based or dedicated connection. Data stores 1214, 1216 may be of different types. In some examples, the data store used by server 1212 may be a database, such as one constructed by Oracle. Relational databases, such as those provided by other vendors. One or more of these databases can be adapted to respond to SQL-formatted commands to store, update, and retrieve data into and from the database.

[0224] In some examples, one or more of the data stores 1214 and 1216 may also be used by the application to store application data. The data store used by the application may be of different types, such as a key-value store, an object store, or a general-purpose store supported by a file system.

[0225] In some examples, the functionality described herein can be provided as a service via a cloud environment. Figure 13 is a simplified block diagram of a cloud-based system environment in which various services can be provided as cloud services according to some examples. In the example depicted in Figure 13, cloud infrastructure system 1302 can provide one or more cloud services that can be requested by a user using one or more client computing devices 1304, 1306, and 1308. Cloud infrastructure system 1302 may include one or more computers and / or servers, which may include those computers and / or servers described above with respect to server 1212. The computers in cloud infrastructure system 1302 may be organized as general-purpose computers, dedicated server computers, server groups, server clusters, or any other suitable arrangement and / or combination.

[0226] One or more networks 1310 can facilitate data communication and exchange between clients 1304, 1306, and 1308 and cloud infrastructure system 1302. One or more networks 1310 may include one or more networks. The networks may be of the same or different types. One or more networks 1310 may support one or more communication protocols (including wired and / or wireless protocols) to facilitate communication.

[0227] The example depicted in Figure 13 is merely one example of a cloud infrastructure system and is not intended to be limiting. It should be understood that in some other examples, the cloud infrastructure system 1302 may have more or fewer components than those depicted in Figure 13, may combine two or more components, or may have different component configurations or arrangements. For example, although Figure 13 depicts three client computing devices, in alternative examples, any number of client computing devices may be supported.

[0228] The term cloud service is generally used to refer to services that become available to users on demand through a service provider's systems (e.g., cloud infrastructure system 1302) and via communication networks such as the Internet. Typically, in a public cloud environment, the servers and systems that make up the cloud service provider's systems differ from the customer's own on-premises servers and systems. The cloud service provider's systems are managed by the cloud service provider. Therefore, customers can utilize cloud services provided by the cloud service provider without having to purchase separate licenses, support, or hardware and software resources for the services. For example, the cloud service provider's systems can host applications, and users can subscribe to and use the applications on demand via the Internet without having to purchase the infrastructure resources to run the applications. Cloud services are designed to provide easy, scalable access to applications, resources, and services. Multiple providers offer cloud services. For example, Oracle in Redwood Shores, California... It provides a variety of cloud services, such as middleware services, database services, Java cloud services, and other services.

[0229] In some examples, cloud infrastructure system 1302 may provide one or more cloud services using different models, such as Software as a Service (SaaS), Platform as a Service (PaaS), Infrastructure as a Service (IaaS), and other models (including hybrid service models). Cloud infrastructure system 1302 may include a set of applications, middleware, databases, and other resources that enable the provisioning of various cloud services.

[0230] The SaaS model enables applications or software to be delivered as a service to customers via communication networks such as the Internet, without requiring customers to purchase the hardware or software for the underlying application. For example, the SaaS model can be used to provide customers with access to on-demand applications hosted by a cloud infrastructure system. Oracle Examples of the SaaS services offered include, but are not limited to, various services for human resources / capital management, customer relationship management (CRM), enterprise resource planning (ERP), supply chain management (SCM), enterprise performance management (EPM), analytics services, social applications, and more.

[0231] The IaaS model is typically used to provide customers with infrastructure resources (such as servers, storage, hardware, and networking resources) as cloud services to offer elastic computing and storage capabilities. (Produced by Oracle) It provides a variety of IaaS services.

[0232] The PaaS model is typically used to provide a platform and environment resources as a service that enable customers to develop, run, and manage applications and services without requiring them to purchase, build, or maintain such resources. (Provided by Oracle) Examples of PaaS services provided include, but are not limited to, Oracle Java Cloud Service (JCS), Oracle Database Cloud Service (DBCS), Data Management Cloud Service, and various application development solutions services.

[0233] Cloud services are typically delivered in an on-demand, self-service, subscription-based, elastically scalable, reliable, highly available, and secure manner. For example, a customer may subscribe to one or more services provided by cloud infrastructure system 1302 via a subscription order. Cloud infrastructure system 1302 then performs processing to provide the service requested in the customer's subscription order. For instance, a user may use utterances to request the cloud infrastructure system to take an action (e.g., an intent) as described above and / or to provide services for a chatbot system as described herein. Cloud infrastructure system 1302 may be configured to provide one or more cloud services.

[0234] Cloud infrastructure system 1302 can provide cloud services through different deployment models. In a public cloud model, cloud infrastructure system 1302 can be owned by a third-party cloud service provider, and cloud services are provided to any general public customer, which can be an individual or a business. In some other examples, under a private cloud model, cloud infrastructure system 1302 can operate within an organization (e.g., within a business organization), and services are provided to customers within the organization. For example, customers can be various departments within the organization, such as human resources or payroll departments, or even individuals within the organization. In some other examples, under a community cloud model, cloud infrastructure system 1302 and the services provided can be shared by multiple organizations in the relevant community. Various other models can also be used, such as a combination of the models mentioned above.

[0235] Client computing devices 1304, 1306, and 1308 can be of different types (as depicted in Figure 12, client computing devices 1202, 1204, 1206, and 1208) and can be capable of operating one or more client applications. Users can use the client devices to interact with cloud infrastructure system 1302, such as requesting services provided by cloud infrastructure system 1302. For example, users can use the client devices to request information or actions from a chatbot as described in this disclosure.

[0236] In some examples, the processing performed by cloud infrastructure system 1302 to provide services may involve model training and deployment. This analysis may involve using, analyzing, and manipulating datasets to train and deploy one or more models. This analysis may be performed by one or more processors, potentially processing data in parallel, performing simulations using the data, etc. For example, big data analysis may be performed by cloud infrastructure system 1302 to generate and train one or more models for a chatbot system. The data used for this analysis may include structured data (e.g., data stored in a database or structured according to a structured model) and / or unstructured data (e.g., data blocks (binary large objects)).

[0237] As illustrated in the example in Figure 13, cloud infrastructure system 1302 may include infrastructure resources 1330 used to facilitate the provisioning of various cloud services offered by cloud infrastructure system 1302. Infrastructure resources 1330 may include, for example, processing resources, storage or memory resources, networking resources, etc. In some examples, a storage virtual machine that can be used to service storage requested from an application may be part of cloud infrastructure system 1302. In other examples, the storage virtual machine may be part of a different system.

[0238] In some examples, to facilitate the efficient provisioning of these resources to support various cloud services provided by cloud infrastructure system 1302 to different customers, resources can be bound to resource groups or resource modules (also known as "pods"). Each resource module or pod may include a pre-integrated and optimized combination of one or more types of resources. In some examples, different pods may be pre-provisioned for different types of cloud services. For example, a first pod may be provisioned for a database service, a second pod may be provisioned for a Java service (which may include a different combination of resources than the pods in the first pod), and so on. For some services, resources allocated for provisioning services can be shared between services.

[0239] The cloud infrastructure system 1302 itself can internally use services 1332 shared by different components of the cloud infrastructure system 1302 and facilitate services provided by the cloud infrastructure system 1302. These internal shared services may include, but are not limited to, security and identity services, integration services, enterprise repository services, enterprise manager services, virus scanning and whitelisting services, high availability, backup and recovery services, services for enabling cloud support, email services, notification services, file transfer services, etc.

[0240] Cloud infrastructure system 1302 may include multiple subsystems. These subsystems may be implemented in software or hardware, or a combination thereof. As depicted in Figure 13, a subsystem may include a user interface subsystem 1312 that enables users or customers of cloud infrastructure system 1302 to interact with cloud infrastructure system 1302. User interface subsystem 1312 may include various interfaces, such as web interface 1314, online store interface 1316 (where advertising and customers can purchase cloud services provided by cloud infrastructure system 1302), and other interfaces 1318. For example, a customer may use a client device to request (service request 1334) one or more services provided by cloud infrastructure system 1302 using one or more of interfaces 1314, 1316, and 1318. For example, a customer may access an online store, browse cloud services provided by cloud infrastructure system 1302, and place a subscription order for one or more services provided by cloud infrastructure system 1302 that the customer wishes to subscribe to. Service requests may include information identifying the customer and the one or more services the customer wishes to subscribe to. For example, a customer may place a subscription order for services provided by cloud infrastructure system 1302. As part of the order, the customer may provide information identifying the chatbot system for which services are to be provided and may optionally provide one or more credentials for the chatbot system.

[0241] In some examples (such as the one depicted in Figure 13), cloud infrastructure system 1302 may include an order management subsystem (OMS) 1320 configured to process new orders. As part of this process, OMS 1320 may be configured to: create accounts for customers (if not already created); receive invoicing and / or billing information from customers to be used to bill them for the requested services; verify customer information; place orders for customers after verification; and schedule various workflows to prepare orders for delivery.

[0242] Once correctly verified, OMS 1320 can invoke the Order Provisioning Subsystem (OPS) 1324, which is configured as order provisioning resources (including processing resources, storage resources, and networking resources). Provisioning may include allocating resources for an order and configuring resources to facilitate the service requested by the customer's order. The manner in which resources are provisioned to an order and the type of resources provided may depend on the type of cloud service the customer has subscribed to. For example, according to a workflow, OPS 1324 may be configured to determine the specific cloud service being requested and identify the number of groups that may have been pre-configured for that specific cloud service. The number of groups allocated to an order may depend on the size / volume / tier / scope of the requested service. For example, the number of groups to be allocated may be determined based on the number of users supported by the service, the duration of the requested service, etc. The allocated groups can then be customized for a specific requesting customer to provide the requested service.

[0243] In some examples, the setup phase processing described above can be performed by cloud infrastructure system 1302 as part of the provisioning process. Cloud infrastructure system 1302 can generate application IDs and select storage virtual machines for applications from storage virtual machines provided by cloud infrastructure system 1302 itself or from storage virtual machines provided by other systems besides cloud infrastructure system 1302.

[0244] Cloud infrastructure system 1302 may send a response or notification 1344 to the requesting client to indicate when the requested service is now ready for use. In some instances, information (e.g., a link) enabling the client to begin using and taking advantage of the benefits of the requested service may be sent. In some examples, for the client requesting the service, the response may include a chatbot system ID generated by cloud infrastructure system 1302 and information identifying the chatbot system selected by cloud infrastructure system 1302 for the chatbot system corresponding to the chatbot system ID.

[0245] Cloud infrastructure system 1302 can provide services to multiple customers. For each customer, cloud infrastructure system 1302 is responsible for managing information related to one or more subscription orders received from the customer, maintaining customer data related to the orders, and providing the requested services to the customer. Cloud infrastructure system 1302 can also collect usage statistics about customers' use of subscribed services. For example, it can collect statistics on storage usage, data transfer, number of users, system uptime, and system downtime. This usage information can be used to issue bills to customers. Billing can be done, for example, on a monthly basis.

[0246] Cloud infrastructure system 1302 can provide services to multiple customers in parallel. Cloud infrastructure system 1302 can store information (potentially including proprietary information) for these customers. In some examples, cloud infrastructure system 1302 includes an Identity Management Subsystem (IMS) 1328 configured to manage customer information and provide separation of the managed information so that information related to one customer cannot be accessed by another customer. IMS 1328 can be configured to provide various security-related services, such as information access management, authentication and authorization services, services for managing customer identities and roles, and related functions.

[0247] Figure 14 illustrates an example of computer system 1400. In some examples, computer system 1400 can be used to implement any digital assistant or chatbot system within a distributed environment, as well as the various server and computer systems described above. As shown in Figure 14, computer system 1400 includes various subsystems, including a processing subsystem 1404 that communicates with multiple other subsystems via a bus subsystem 1402. These other subsystems may include a processing acceleration unit 1406, an I / O subsystem 1408, a storage subsystem 1418, and a communication subsystem 1424. Storage subsystem 1418 may include a non-transitory computer-readable storage medium, including storage medium 1422 and system memory 1410.

[0248] Bus subsystem 1402 provides mechanisms for enabling the various components and subsystems of computer system 1400 to communicate with each other as intended. While bus subsystem 1402 is schematically shown as a single bus, alternative examples of bus subsystems may utilize multiple buses. Bus subsystem 1402 can be any of a variety of bus structures, including a memory bus or memory controller, a peripheral bus, a local bus using any of a variety of bus architectures, etc. For example, such architectures may include an Industry Standard Architecture (ISA) bus, a Micro Channel Architecture (MCA) bus, an Enhanced ISA (EISA) bus, a Video Electronics Standards Association (VESA) local bus, and a Peripheral Component Interconnect (PCI) bus (which may be implemented as a mezzanine bus manufactured to the IEEE P1386.1 standard), etc.

[0249] Processing subsystem 1404 controls the operation of computer system 1400 and may include one or more processors, application-specific integrated circuits (ASICs), or field-programmable gate arrays (FPGAs). Processors may include single-core or multi-core processors. The processing resources of computer system 1400 may be organized into one or more processing units 1432, 1434, etc. A processing unit may include one or more processors, one or more cores from the same or different processors, a combination of cores and processors, or other combinations of cores and processors. In some examples, processing subsystem 1404 may include one or more dedicated coprocessors such as graphics processors or digital signal processors (DSPs). In some examples, some or all of the processing units of processing subsystem 1404 may be implemented using custom circuitry such as ASICs or FPGAs.

[0250] In some examples, the processing units in processing subsystem 1404 can execute instructions stored in system memory 1410 or on computer-readable storage medium 1422. In various examples, the processing units can execute various program or code instructions and can maintain multiple concurrently executing programs or processes. At any given time, some or all of the program code to be executed can reside in system memory 1410 and / or on computer-readable storage medium 1422 (potentially including residing on one or more storage devices). With suitable programming, processing subsystem 1404 can provide the various functions described above. In an instance where computer system 1400 is executing one or more virtual machines, one or more processing units can be assigned to each virtual machine.

[0251] In some examples, a processing acceleration unit 1406 may optionally be provided for performing custom processing or for offloading some of the processing performed by the processing subsystem 1404, thereby accelerating the overall processing performed by the computer system 1400.

[0252] I / O subsystem 1408 may include devices and mechanisms for inputting information to and / or outputting information from or via computer system 1400. Generally, the term input device is intended to include all possible types of devices and mechanisms for inputting information to computer system 1400. User interface input devices may include, for example, keyboards, pointing devices such as mice or trackballs, touchpads or touchscreens integrated into displays, scroll wheels, click wheels, dial pads, buttons, switches, keypads, audio input devices with voice command recognition systems, microphones, and other types of input devices. User interface input devices may also include motion sensing and / or gesture recognition devices, such as Microsoft's [unclear - possibly a specific technology or feature], which enables users to control and interact with input devices. Motion sensor, Microsoft The 360 ​​Game Controller provides an interface for receiving input using gestures and spoken commands. The user interface input device may also include eye gesture recognition devices, such as detecting eye movements from the user (e.g., "blinking" when taking a photo and / or making menu selections) and translating the eye gestures into the input device (such as Google). Google input Blink detector. Additionally, the user interface input device may include features that enable the user to interact with a voice recognition system (e.g., via voice commands). Voice recognition sensing devices for navigators.

[0253] Other examples of user interface input devices include, but are not limited to, 3D mice, joysticks or pointing sticks, game controllers and graphics tablets, and audio / visual devices such as speakers, digital cameras, digital camcorders, portable media players, webcams, image scanners, fingerprint scanners, barcode readers, 3D scanners, 3D printers, laser rangefinders, and eye-tracking devices. Additionally, user interface input devices may include, for example, medical imaging input devices such as computed tomography (CT), magnetic resonance imaging (MRI), positron emission tomography (PET), and medical ultrasound examination equipment. User interface input devices may also include, for example, audio input devices such as MIDI keyboards, digital musical instruments, etc.

[0254] Generally, the term "output device" is intended to encompass all possible types of devices and mechanisms for outputting information from computer system 1400 to a user or other computer. User interface output devices may include display subsystems, indicator lights, or non-visual displays such as audio output devices. Display subsystems may be cathode ray tube (CRT), flat panel devices (such as those using liquid crystal displays (LCDs) or plasma displays), projection devices, touchscreens, etc. For example, user interface output devices may include, but are not limited to, various display devices that visually convey text, graphics, and audio / video information, such as monitors, printers, speakers, headsets, car navigation systems, plotters, voice output devices, and modems.

[0255] Storage subsystem 1418 provides a repository or data store for storing information and data used by computer system 1400. Storage subsystem 1418 provides a tangible, non-transitory, computer-readable storage medium for storing basic programming and data constructs that provide some example functionality. Storage subsystem 1418 may store software (e.g., programs, code modules, instructions) that provides the functionality described above when executed by processing subsystem 1404. The software may be executed by one or more processing units of processing subsystem 1404. Storage subsystem 1418 may also be certified in accordance with the teachings of this disclosure.

[0256] Storage subsystem 1418 may include one or more non-transitory memory devices, including volatile memory devices and non-volatile memory devices. As shown in FIG14, storage subsystem 1418 includes system memory 1410 and computer-readable storage medium 1422. System memory 1410 may include multiple memories, including volatile main random access memory (RAM) for storing instructions and data during program execution and non-volatile read-only memory (ROM) or flash memory where fixed instructions are stored. In some embodiments, a basic input / output system (BIOS) containing basic routines such as those that help transfer information between elements within computer system 1400 during startup may typically be stored in ROM. RAM typically contains data and / or program modules currently being operated and executed by processing subsystem 1404. In some embodiments, system memory 1410 may include various types of memory such as static random access memory (SRAM), dynamic random access memory (DRAM), etc.

[0257] By way of example, and not limitation, as depicted in Figure 14, system memory 1410 may load an executing application 1412 (which may include various applications such as a web browser, a middleware application, a relational database management system (RDBMS), etc.), program data 1414, and operating system 1416. By way of example, operating system 1416 may include various versions of Microsoft... Apple and / or Linux operating system, and various commercially available... Or a UNIX-like operating system (including but not limited to various GNU / Linux operating systems, Google...) OS, etc.) and / or such as iOS, Telephone, OS OS OS operating systems and other mobile operating systems, etc.

[0258] Computer-readable storage medium 1422 can store programming and data structures that provide some example functionality. Computer-readable storage medium 1422 can provide computer system 1400 with storage for computer-readable instructions, data structures, program modules, and other data. Software (programs, code modules, instructions) that provides the functionality described above, when executed by processing subsystem 1404, can be stored in storage subsystem 1418. By way of example, computer-readable storage medium 1422 may include, for example, hard disk drives, disk drives, optical disc drives (such as CD ROMs, DVDs, Blu-ray discs, etc.). Non-volatile memory such as disks or other optical media. Computer-readable storage medium 1422 may include, but is not limited to, [other types of storage media]. Drives, flash memory cards, Universal Serial Bus (USB) flash memory drives, Secure Digital (SD) cards, DVD discs, digital videotapes, etc. Computer-readable storage media 1422 may also include solid-state drives (SSDs) based on non-volatile memory (e.g., flash memory-based SSDs, enterprise-class flash memory drives, solid-state ROMs, etc.), SSDs based on volatile memory such as solid-state RAM, dynamic RAM, static RAM, DRAM-based SSDs, magnetoresistive RAM (MRAM) SSDs, and hybrid SSDs using a combination of DRAM and flash memory-based SSDs.

[0259] In some examples, storage subsystem 1418 may also include a computer-readable storage medium reader 1420 that can be further connected to computer-readable storage medium 1422. Reader 1420 may receive data from and be configured to read data from a storage device such as a disk or flash memory drive.

[0260] In some examples, computer system 1400 may support virtualization technologies, including but not limited to virtualization of processing and memory resources. For example, computer system 1400 may provide support for executing one or more virtual machines. In some examples, computer system 1400 may execute programs such as hypervisors that facilitate the configuration and management of virtual machines. Each virtual machine may be allocated memory, computing (e.g., processor, cores), I / O, and networking resources. Each virtual machine typically runs independently of other virtual machines. Virtual machines typically run their own operating system, which may be the same as or different from the operating systems executed by other virtual machines executed by computer system 1400. Therefore, multiple operating systems may potentially run simultaneously by computer system 1400.

[0261] The communication subsystem 1424 provides an interface to other computer systems and networks. The communication subsystem 1424 serves as an interface for receiving data from other systems and transmitting data from computer system 1400 to other systems. For example, the communication subsystem 1424 can enable computer system 1400 to establish a communication channel via the Internet to one or more client devices for receiving and sending information to client devices. For example, when computer system 1400 is used to implement the robot system 120 depicted in FIG1, the communication subsystem can be used to communicate with a chatbot system selected for the application.

[0262] The communication subsystem 1424 may support both wired and / or wireless communication protocols. In some examples, the communication subsystem 1424 may include radio frequency (RF) transceiver components (e.g., using cellular telephone technology, advanced data network technologies such as 3G, 4G, or EDGE (Global Evolution Enhanced Data Rate), WiFi (IEEE 802.XX Home Standard), or other mobile communication technologies, or any combination thereof), a global positioning system (GPS) receiver component, and / or other components for accessing wireless voice and / or data networks. In some examples, in addition to or as an alternative to a wireless interface, the communication subsystem 1424 may provide wired network connectivity (e.g., Ethernet).

[0263] The communication subsystem 1424 can receive and transmit data in various forms. In some examples, among other forms, the communication subsystem 1424 can receive input communications in the form of structured and / or unstructured data feeds 1426, event streams 1428, event updates 1430, etc. For example, the communication subsystem 1424 can be configured to receive (or send) data feeds 1426 in real time from users of social media networks and / or other communication services, such as... feed, Updates, web feeds (such as rich site summary (RSS) feeds) and / or real-time updates from one or more third-party information sources.

[0264] In some examples, the communication subsystem 1424 may be configured to receive data in the form of a continuous data stream, which may include an event stream 1428 and / or event updates 1430 of real-time events (which may be inherently continuous or unbounded and may not have an explicit end). Examples of applications that generate continuous data may include, for example, sensor data applications, financial reporting machines, network performance measurement tools (e.g., network monitoring and traffic management applications), clickstream analysis tools, vehicle traffic monitoring, etc.

[0265] The communication subsystem 1424 can also be configured to transmit data from computer system 1400 to other computer systems or networks. Data can be transmitted in various forms, such as structured and / or unstructured data feeds 1426, event streams 1428, event updates 1430, etc., to one or more databases that can communicate with one or more streaming data source computers coupled to computer system 1400.

[0266] Computer system 1400 can be one of various types, including handheld portable devices (e.g., Cellular phone Computing tablets, PDAs), wearable devices (e.g., Google...) This includes head-mounted displays, personal computers, workstations, mainframes, self-service terminals, server racks, or any other data processing systems. Due to the constantly evolving nature of computers and networks, the description of the computer system 1400 depicted in Figure 14 is intended only as a particular example. Many other configurations with more or fewer components than the system depicted in Figure 14 are possible. Based on this disclosure and the teachings provided herein, it should be understood that other ways and / or methods are available to implement the various examples.

[0267] While specific examples have been described, various modifications, alterations, alternative constructions, and equivalents are possible. The examples are not limited to operations within certain specific data processing environments but are free to operate within multiple data processing environments. Furthermore, although certain examples have been described using specific series of transactions and steps, it will be apparent to those skilled in the art that this is not intended to be restrictive. While some flowcharts describe operations as sequential processes, many operations can be performed in parallel or simultaneously. Additionally, the order of operations can be rearranged. Processes may have additional steps not included in the diagrams. Various features and aspects of the examples described above can be used individually or in combination.

[0268] Furthermore, while certain examples have been described using specific combinations of hardware and software, it should be recognized that other combinations of hardware and software are also possible. Some examples may be implemented using only hardware, only software, or a combination thereof. The various processes described herein can be implemented on the same or different processors in any combination.

[0269] When a device, system, component, or module is described as being configured to perform certain operations or functions, this configuration can be accomplished, for example, by designing electronic circuitry to perform operations, by programming programmable electronic circuitry (such as a microprocessor) to perform operations (e.g., by executing computer instructions or code), or by a processor or core programmed to execute code or instructions stored on a non-transitory memory medium, or any combination thereof. Processes can communicate using a variety of techniques, including but not limited to conventional techniques for inter-process communication, and different pairs of processes may use different techniques, or the same pair of processes may use different techniques at different times.

[0270] Specific details are set forth in this disclosure to provide a thorough understanding of the examples. However, the examples may be practiced without these specific details. For example, well-known circuits, processes, algorithms, structures, and techniques have been shown without unnecessary details to avoid obscuring the examples. This description provides exemplary examples only and is not intended to limit the scope, applicability, or configuration of other examples. Rather, the foregoing description of the examples will provide those skilled in the art with enabling descriptions for implementing the various examples. Various changes can be made to the function and arrangement of the elements.

[0271] Therefore, this specification and drawings should be viewed in an illustrative rather than restrictive sense. However, it will be apparent that additions, omissions, deletions, and other modifications and changes may be made without departing from the broader spirit and scope set forth in the claims. Thus, while specific examples have been described, they are not intended to be restrictive. Various modifications and equivalents are within the scope of the appended claims.

[0272] In the foregoing description, various aspects of this disclosure have been described with reference to specific examples; however, those skilled in the art will recognize that this disclosure is not limited thereto. Various features and aspects of the disclosure described above may be used individually or in combination. Furthermore, the examples may be utilized in any number of environments and applications beyond those described herein without departing from the broader spirit and scope of the specification. Therefore, the specification and drawings are to be regarded as illustrative rather than restrictive.

[0273] In the foregoing description, the methods have been described in a specific order for illustrative purposes. It should be understood that, in alternative examples, the methods may be performed in a different order than described. It should also be understood that the methods described above may be executed by hardware components or may be embodied in a sequence of machine-executable instructions that can be used to cause a machine (e.g., a general-purpose or special-purpose processor or logic circuit programmed with instructions) to execute the methods. These machine-executable instructions may be stored on one or more machine-readable media (e.g., CD-ROM or other types of optical discs, floppy disks, ROM, RAM, EPROM, EEPROM, magnetic cards or optical cards, flash memory, or other types of machine-readable media suitable for storing electronic instructions). Alternatively, the methods may be executed by a combination of hardware and software.

[0274] When a component is described as being configured to perform certain operations, such configuration can be accomplished, for example, by designing electronic circuitry or other hardware for performing those operations, by programming programmable electronic circuitry (e.g., a microprocessor or other suitable electronic circuitry) for performing those operations, or any combination thereof.

[0275] Although illustrative examples of this application have been described in detail herein, it should be understood that the inventive concepts may be embodied and employed in other ways, and the appended claims are intended to be construed as including such variations, except where limited by the prior art.< / domain>

Claims

1. A computer-implemented method, comprising: The method involves receiving utterances from a user within a chatbot session, wherein the current skill context of the chatbot session is a first skill and the current group context of the chatbot session is a first group; inputting the utterances into a candidate skill model for the first group; using the candidate skill model to obtain a ranking of skills within the first group that could potentially handle the utterances; determining a second skill as the highest-ranking skill for handling the utterances based on the skill rankings; changing the current skill context of the chatbot session to the second skill; inputting the utterances into a candidate flow model for the second skill; using the candidate flow model to obtain a ranking of intents within the second skill that match the utterances; and determining an intent as the highest-ranking intent for handling the utterances based on the ranking of the intents, wherein the method further includes: receiving subsequent utterances from the user within the chatbot session, wherein the current skill context of the chatbot session is the second skill and the current group context of the chatbot session is a first group; inputting the utterances into a candidate skill model for the first group; using the candidate skill model to obtain a ranking of skills within the first group that could potentially handle the utterances; determining a second skill as the highest-ranking skill for handling the utterances based on the ranking of the intents; and inputting the utterances into a candidate flow model for the second skill; using the candidate flow model to obtain a ranking of intents within the second skill that could potentially handle the utterances; and inputting the utterances into a candidate flow model for the second skill; using the candidate flow model to obtain a ranking of intents within the second skill that match the utterances; and inputting the utterances into a candidate flow model for the second skill; using the candidate flow model to obtain a ranking of intents within the second skill; and inputting the utterances into a candidate flow model for ... The current group context of the chatbot session is a first group; the subsequent utterance is input into the candidate skill model for the first group; the candidate skill model is used to obtain a ranking of skills within the first group that can potentially handle the subsequent utterance; based on the skill ranking, an unresolved intent skill is determined to be the highest-ranked skill for handling the subsequent utterance; the subsequent utterance is input into another candidate skill model; the other candidate skill model is used to obtain a ranking of skills that can potentially handle the subsequent utterance; based on the skill ranking, a third skill is determined to be the highest-ranked skill for handling the subsequent utterance; and the current skill context of the chatbot session is assigned to the third skill and the current group context of the chatbot session is assigned to a second group, wherein the second group is defined for the third skill, and the assignment of the current group context of the chatbot session to the second group is performed based on the definition of the second group for the third skill.

2. The computer-implemented method as described in claim 1, wherein, Obtaining the ranking of the skills includes: evaluating the utterance and generating a confidence score for the skills within the first group; identifying any skills whose confidence scores exceed the value of a candidate skill confidence threshold routing parameter as candidate skills for further evaluation; and ranking the candidate skills as skills within the first group that are potentially capable of processing the utterance based on the confidence scores.

3. The computer-implemented method as described in claim 1 or claim 2, wherein, The ranking of the intent includes: evaluating the intent and generating a confidence score for the intent within the second skill; identifying any intent whose confidence score exceeds the value of a confidence threshold routing parameter as a candidate intent for further evaluation; and ranking the candidate intents as intents within the first skill that match the utterance based on the confidence score.

4. The computer-implemented method as described in claim 1 or claim 2, further comprising: The conversation flow is initiated in the chatbot session with the user based on the highest-ranking intent used to process the utterance.

5. The computer-implemented method as described in claim 1 or claim 2, further comprising: Within the chatbot session, an initial utterance from a user is received, wherein the initial utterance is received prior to the utterance; the initial utterance is input into the candidate skill model; the candidate skill model is used to obtain a ranking of skills that can potentially handle the initial utterance; based on the skill ranking, a first skill is determined to be the highest-ranked skill for handling the initial utterance; the current skill context of the chatbot session is assigned to the first skill, and the current group context of the chatbot session is assigned to the first group, wherein the first group is defined for the first skill, and the assignment of the current group context of the chatbot session to the first group is performed based on the definition of the first group for the first skill.

6. An electronic system comprising: One or more data processors; And a non-transitory computer-readable storage medium containing instructions that, when executed on the one or more data processors, cause the one or more data processors to perform actions including: receiving a utterance from a user within a chatbot session, wherein the current skill context of the chatbot session is a first skill and the current group context of the chatbot session is a first group; inputting the utterance into a candidate skill model for the first group; using the candidate skill model to obtain a ranking of skills within the first group that are potentially capable of processing the utterance; determining, based on the ranking of the skills, a second skill as the highest-ranking skill for processing the utterance; changing the current skill context of the chatbot session to the second skill; inputting the utterance into a candidate stream model for the second skill; using the candidate stream model to obtain a ranking of intents within the second skill that match the utterance; and determining, based on the ranking of the intents, an intent as the highest-ranking intent for processing the utterance, wherein the actions further include: receiving subsequent utterances from the user within the chatbot session. The process involves: inputting the subsequent utterance into a candidate skill model for the first group; using the candidate skill model to obtain a ranking of skills within the first group that can potentially handle the subsequent utterance; determining, based on the skill ranking, that an unresolved intent skill is the highest-ranked skill for handling the subsequent utterance; inputting the subsequent utterance into another candidate skill model; using the other candidate skill model to obtain a ranking of skills that can potentially handle the subsequent utterance; determining, based on the skill ranking, that a third skill is the highest-ranked skill for handling the subsequent utterance; and assigning the current skill context of the chatbot session to the third skill and assigning the current group context of the chatbot session to a second group, wherein the second group is defined for the third skill, and the assignment of the current group context of the chatbot session to the second group is performed based on the definition of the second group for the third skill.

7. The electronic system of claim 6, wherein, Obtaining the ranking of the skills includes: evaluating the utterance and generating a confidence score for the skills within the first group; identifying any skills whose confidence scores exceed the value of a candidate skill confidence threshold routing parameter as candidate skills for further evaluation; and ranking the candidate skills as skills within the first group that are potentially capable of processing the utterance based on the confidence scores.

8. The electronic system as claimed in claim 6 or claim 7, wherein, The ranking of the intent includes: evaluating the intent and generating a confidence score for the intent within the second skill; identifying any intent whose confidence score exceeds the value of a confidence threshold routing parameter as a candidate intent for further evaluation; and ranking the candidate intents as intents within the first skill that match the utterance based on the confidence score.

9. The electronic system as claimed in claim 6 or claim 7, wherein, The action further includes initiating a conversation flow in the chatbot session with the user based on the highest-ranking intent used to process the utterance.

10. The electronic system as claimed in claim 6 or claim 7, wherein, The action further includes: receiving an initial utterance from a user within the chatbot session, wherein the initial utterance is received prior to the utterance; inputting the initial utterance into the candidate skill model; using the candidate skill model to obtain a ranking of skills that can potentially process the initial utterance; determining, based on the skill ranking, that the first skill is the highest-ranked skill for processing the initial utterance; assigning the current skill context of the chatbot session to the first skill and assigning the current group context of the chatbot session to the first group, wherein the first group is defined for the first skill, and assigning the current group context of the chatbot session to the first group is performed based on defining the first group for the first skill.

11. A computer program product tangibly embodied in a non-transitory machine-readable storage medium, the computer program product comprising instructions configured to cause one or more data processors to perform actions including: receiving utterances from a user within a chatbot session, wherein, The current skill context of the chatbot session is a first skill and the current group context of the chatbot session is a first group; the utterance is input into a candidate skill model for the first group; the candidate skill model is used to obtain a ranking of skills within the first group that can potentially process the utterance; Based on the ranking of the skills, the second skill is determined to be the highest-ranking skill used to process the utterance; the current skill context of the chatbot session is changed to the second skill; The utterance is input into a candidate stream model for the second skill; The candidate flow model is used to obtain a ranking of intentions within the second skill that match the utterance; The action further includes: determining, based on the ranking of the intents, the highest-ranking intent for processing the utterance, wherein the action further includes: receiving subsequent utterances from the user within the chatbot session, wherein the current skill context of the chatbot session is the second skill and the current group context of the chatbot session is the first group; inputting the subsequent utterances into the candidate skill model for the first group; using the candidate skill model to obtain a ranking of skills within the first group that can potentially process the subsequent utterances; and determining, based on the skill rankings, an unresolved intent skill is used to process the subsequent utterances. The highest-ranked skill; inputting the subsequent utterance into another candidate skill model; using the other candidate skill model to obtain a ranking of skills that can potentially handle the subsequent utterance; determining a third skill as the highest-ranked skill for handling the subsequent utterance based on the skill rankings; and assigning the current skill context of the chatbot session to the third skill and assigning the current group context of the chatbot session to a second group, wherein the second group is defined for the third skill, and the assignment of the current group context of the chatbot session to the second group is performed based on the definition of the second group for the third skill.

12. The computer program product of claim 11, wherein, Obtaining the ranking of the skills includes: evaluating the utterance and generating a confidence score for the skills within the first group; identifying any skills whose confidence scores exceed the value of a candidate skill confidence threshold routing parameter as candidate skills for further evaluation; and ranking the candidate skills as skills within the first group that are potentially capable of processing the utterance based on the confidence scores.

13. The computer program product as claimed in claim 11 or claim 12, wherein, The ranking of the intent includes: evaluating the intent and generating a confidence score for the intent within the second skill; identifying any intent whose confidence score exceeds the value of a confidence threshold routing parameter as a candidate intent for further evaluation; and ranking the candidate intents as intents within the first skill that match the utterance based on the confidence score.

14. The computer program product as claimed in claim 11 or claim 12, wherein, The action further includes initiating a conversation flow in the chatbot session with the user based on the highest-ranking intent used to process the utterance.

15. The computer program product as claimed in claim 11 or claim 12, wherein, The action further includes: receiving an initial utterance from a user within the chatbot session, wherein the initial utterance is received prior to the utterance; inputting the initial utterance into the candidate skill model; using the candidate skill model to obtain a ranking of skills that can potentially process the initial utterance; determining, based on the skill ranking, that the first skill is the highest-ranked skill for processing the initial utterance; assigning the current skill context of the chatbot session to the first skill and assigning the current group context of the chatbot session to the first group, wherein the first group is defined for the first skill, and assigning the current group context of the chatbot session to the first group is performed based on defining the first group for the first skill.

Images