Template-based intent classification for chatbots

Abstract

This disclosure relates to template-based intent classification for chatbots. This disclosure relates to chatbot systems, and more specifically to techniques for determining that an input utterance represents a task that a particular chatbot can perform based on matching the input utterance to templates. Techniques for generating templates based on example utterances that have been provided to a chatbot are also described. In certain embodiments, an initial set of templates is generated based on example utterances. This initial set of templates is then refined using template generalization techniques that can be performed at the word level or at the sentence level to generate a final set of templates for use at runtime when matching these templates to user utterances. The final set of templates can include one or more generalized templates derived from the initial set of templates, and can also include the initial set of templates.

Description

[0001] Cross-reference to related applications

[0002] This application claims the benefit and priority of U.S. Application No. 62 / 899,691, filed September 12, 2019, entitled “TEMPLATE-BASED INTENT CLASSIFICATION FOR CHATBOTS,” and U.S. Application No. 17 / 011,296, filed September 3, 2020, entitled “TEMPLATE-BASED INTENT CLASSIFICATION FOR CHATBOTS,” which are incorporated herein by reference in their entirety. Technical Field

[0003] This disclosure relates to chatbot systems, and more specifically to techniques for determining that an input utterance represents a task that a particular chatbot can perform by matching the input utterance with a template. Background Technology

[0004] Chatbots provide an interface for engaging in conversations with human users. Chatbots can be configured to perform various tasks in response to user input provided during a conversation. User input can be provided in various forms, including, for example, audio input and text input. Therefore, Natural Language Understanding (NLU), speech-to-text, and other language processing techniques can be used as part of the processing performed by the chatbot. In some computing environments, multiple chatbots can be used to engage in conversations with users, with each chatbot handling a different set of tasks.

[0005] One of the challenges of implementing a chatbot system is determining whether user input (e.g., utterances expressing requests or queries) should be handled by a particular chatbot. To determine whether a chatbot is capable of handling user input (e.g., fulfilling a request or answering a question), the meaning of the user input should be determined based on analysis of its content. Summary of the Invention

[0006] This disclosure relates to chatbot systems, and more specifically to techniques for determining that input utterances represent tasks that a particular chatbot can perform based on matching input utterances with templates. Various embodiments are described herein, including methods, systems, non-transitory computer-readable storage media storing programs, code, or instructions executable by one or more processors.

[0007] In certain computing environments, multiple chatbots can be available to engage in conversations with users, with each chatbot handling a different set of tasks. In such environments, it can be convenient to handle a user request or query in a user input if it is early determined which chatbot, if any, is capable of handling the request or query. Various techniques are described herein for classifying the intent of a user input based on matching an input utterance to a template.

[0008] In some embodiments described herein, a rule-based classifier is configured to identify a chatbot by matching an utterance to a template. In certain embodiments, the classification involves comparing an input utterance to a set of templates, where each template is a regular expression (regex) and is associated with a particular chatbot. Techniques are also described for generating one or more templates for a chatbot based on example utterances that have been provided to the chatbot. In certain embodiments, an initial set of templates is generated based on example utterances. The initial set of templates is then refined using template generalization techniques that can be performed at the word level or the sentence level or both, to generate a final set of templates for use at runtime when matching the templates to user utterances. The final set of templates can include one or more generalized templates derived from the initial set of templates, and can also include the initial set of templates.

[0009] In certain embodiments, a method includes receiving, at a computer-implemented chatbot system, an input utterance from a user. The chatbot system determines, based on a set of templates, that the input utterance is representative of a task that a first chatbot in the chatbot system is configured to perform. For example, to make this determination, the chatbot system can compare the input utterance to each template in the set of templates. A first template in the set of templates is a regular expression that includes a string pattern that corresponds to a generalized form of one or more example utterances that have been provided to the first chatbot, and the one or more example utterances are representative of a task that the first chatbot is configured to perform. Making the determination further includes, for the chatbot system, computing, for each template in the set of templates, a corresponding score based on a degree of matching between the input utterance and the template. Based on the computed scores, the chatbot system determines that the first template in the set of templates best matches the input utterance, and thus, the first chatbot is suitable to handle the input utterance. In response to determining that the input utterance is representative of a task that the first chatbot is configured to perform, the chatbot system communicates at least a portion of the input utterance as input to the first chatbot.

[0010] The foregoing summary, as well as other features and embodiments, will be more fully understood from the following description, claims, and drawings. BRIEF DESCRIPTION OF DRAWINGS

[0011] Figure 1 is a simplified block diagram of an environment incorporating a chatbot system according to certain embodiments.

[0012] Figure 2 is a simplified block diagram of a master bot system according to certain embodiments.

[0013] Figure 3 is a simplified block diagram of a skill bot system according to certain embodiments.

[0014] Figure 4 illustrates an example template set according to certain embodiments.

[0015] Figure 5 illustrates example utterances and generalization templates according to certain embodiments.

[0016] Figure 6 illustrates the results of a series of generalization steps based on example utterances in Figure 5 according to certain embodiments.

[0017] Figure 7 illustrates the results of a series of generalization steps according to certain embodiments that can follow the generalization shown in Figure 6 and ultimately result in the generalization template shown in Figure 5 according to certain embodiments.

[0018] Figure 8 illustrates a simplified flow diagram that depicts a process for generating a generalization template as part of training a classifier according to certain embodiments.

[0019] Figure 9 illustrates a simplified flow diagram that depicts a process for classifying the intent of a user utterance based on a generalization template according to certain embodiments.

[0020] Figure 10 depicts a simplified diagram of a distributed system for implementing embodiments.

[0021] Figure 11 is a simplified block diagram of a cloud-based system environment providing cloud services according to certain embodiments.

[0022] Figure 12 illustrates an example computer system that can be used to implement certain embodiments. DETAILED DESCRIPTION

[0023] In the following description, for purposes of explanation, specific details are set forth in order to provide a thorough understanding of certain embodiments. However, it will be apparent to one skilled in the art that various embodiments can be practiced without these specific details. The drawings and description are not intended to be restrictive. The word "exemplary" is used herein to mean "serving as an example, instance, or illustration." Any implementation described herein as "exemplary" is not necessarily to be construed as preferred or advantageous over other implementations.

[0024] In some computing environments, multiple chatbots can be available to engage in a conversation with a user, where each chatbot handles a different set of tasks. In such environments, it can be more convenient to handle a user request or query if it is early determined which chatbot, if any, is capable of handling the request or query. Various techniques are described herein for classifying the intent of user input based on matching the input utterance to a template. Certain embodiments described herein improve the art of digital assistants and chatbots by providing a technique for directing user input to the appropriate chatbot in situations where multiple chatbots with different skill sets are available, thereby ensuring or at least increasing the chances that the user input is accurately and efficiently handled.

[0025] One challenging function of a chatbot is determining whether there is (i.e., available) the capability of a chatbot that is capable of handling a given user input or determining whether a particular chatbot is capable of handling a given user input. If there is such a chatbot, the user input can be transmitted to the chatbot for processing. In systems where there are multiple chatbots, it can sometimes be difficult to determine whether there is a particular chatbot among the multiple chatbots that is more suitable for handling a given user input. For example, a chatbot (also referred to as a chatbot system) can include a financial bot and a retail bot, where the financial bot is configured to provide a user with information about the user's bank account balance, and where the retail bot is configured to provide a user with information about a gift card balance of a retail store. If a user provides a user input represented by the input utterance "check balance," it is not clear which of the two bots the user intends to invoke, as both the financial bot and the retail bot are capable of providing balance information. In this scenario, the user can be prompted for further input to disambiguate the user's intent. For example, the chatbot system can ask the user which of the two bots to use or which account the user wants to check the balance for.

[0026] One option for resolving user intent expressed in an utterance to a particular chatbot is to employ a machine learning (ML)-based classifier. For example, an ML-based classifier can be implemented using a neural network that has been trained on example utterances (also referred to as training utterances), where the example utterances are representative of tasks that the chatbot can perform. Each chatbot in a chatbot system can have a corresponding classifier associated with it that has been trained using example utterances specific to the chatbot. As an alternative or in addition to such classifiers, a rule-based classifier can be used. For example, in certain embodiments, a rule-based classifier is used in a chatbot system to identify another chatbot (e.g., in the form of selecting another chatbot from among multiple available chatbots) to handle user input.

[0027] As discussed below, embodiments are described in which a rule-based classifier is configured to identify a chatbot by matching an utterance to a template. In certain embodiments, the classification involves comparing an input utterance to a set of templates, where each template is a regular expression (regex) and is associated with a particular chatbot. A regex is a logical expression that specifies a pattern of strings. In certain embodiments, a regex includes pattern information plus language part-of-speech (POS) information.

[0028] Techniques are also described for generating templates based on example utterances that have been provided to a chatbot. One of the challenges in generating templates is constructing the templates in such a way that the templates match as many different utterances as possible while also ensuring that the matching utterances are representative of the same intent. For example, if a training corpus includes four example utterances that are representative of a task of querying a bank account balance (i.e., four different ways of stating a request for the task), a separate template can be generated for each of the four example utterances, where each template needs to match its corresponding example utterance exactly. Each template may, for example, require that the user utterance include at least the same sequence of words as the example utterance to which the template corresponds in order to be considered a match. Such templates would be very precise, such that the chance of matching a user utterance that is not representative of querying a bank account balance would be very low. However, requiring exact matches is suboptimal because while the precision is high, very few user utterances would match such templates, resulting in a low recall rate. Thus, the usefulness of exact match templates is limited.

[0029] Conversely, if the template is too general, there is a risk of matching it with user discourse representing different intentions. For example, a template that only requires the phrase "Check balance" would match "Check balance in savingsaccount" and "Check balance on gift card." Therefore, it is meaningful to determine how to properly construct the template—including determining which elements of the template should be mandatory and which elements can be optional.

[0030] In some embodiments, an initial template set is generated based on example utterances. This initial template set is then refined using template generalization techniques that can be performed at the word level and / or sentence level to generate a final template set for use at runtime when matching these templates against user utterances. The final template set may include one or more generalized templates derived from the initial template set, and may also include the initial template set itself.

[0031] Overview of an example chatbot system

[0032] Figure 1 This is a block diagram of an environment 100 incorporating a chatbot system according to certain embodiments. Environment 100 includes a Digital Assistant Builder Platform (DABP) 102, which enables users of DABP 102 to create and deploy digital assistant or chatbot systems. DABP 102 can be used to create one or more digital assistant (DA) or chatbot systems. For example, as... Figure 1 As shown, a user 104 representing a specific business can use DABP 102 to create and deploy a digital assistant 106 for that business's users. For example, DABP 102 can be used by a bank to create one or more digital assistants for its customers. Multiple businesses can use the same DABP 102 platform to create digital assistants. As another example, a restaurant (e.g., a pizzeria) owner can use DABP 102 to create and deploy a digital assistant that enables the restaurant's customers to order food (e.g., order pizza).

[0033] For the purposes of this disclosure, a "digital assistant" is an entity that assists users of the digital assistant in accomplishing various tasks through natural language conversations. A digital assistant can be implemented using only software (e.g., the digital assistant is a digital entity implemented using programs, code, or instructions executable by one or more processors), using hardware, or using a combination of hardware and software. A digital assistant can be embodied or implemented in various physical systems or devices such as computers, mobile phones, watches, appliances, vehicles, etc. Digital assistants are sometimes also referred to as chatbot systems. Thus, for the purposes of this disclosure, the terms digital assistant and chatbot system are interchangeable.

[0034] A digital assistant, such as digital assistant 106 built using DABP 102, can be used to perform various tasks via natural language based conversations between the digital assistant and its users 108. As part of a conversation, a user can provide one or more user inputs 110 to the digital assistant 106 and obtain returned responses 112 from the digital assistant 106. A conversation can include one or more inputs 110 as well as responses 112 in it. Via these conversations, a user can request that one or more tasks be performed by the digital assistant, and in response, the digital assistant is configured to perform the tasks requested by the user and respond to the user with appropriate responses.

[0035] User inputs 110 are typically in the form of natural language and are referred to as utterances. A user utterance 110 can be in textual form, such as when a user types a sentence, question, piece of text, or even a single word and provides the text as input to the digital assistant 106. In some embodiments, a user utterance 110 can be in the form of audio input or speech, such as when a user speaks or says something as input to the digital assistant 106. An utterance is typically in the form of a language spoken by the user 108. When the utterance is in the form of speech, the speech input is converted to a textual form of the utterance in that particular language, and then the textual utterance is processed by the digital assistant 106 as a user utterance 110. Various speech-to-text processing techniques can be used to convert speech or audio input to a textual utterance, which is then processed by the digital assistant 106. In some embodiments, the conversion of speech to text can be done by the digital assistant 106 itself.

[0036] Speech (which can be text or speech) can be fragments, a sentence, multiple sentences, one or more words, one or more questions, combinations of the above types, etc. Digital Assistant 106 is configured to apply Natural Language Understanding (NLU) technology to the speech to understand the meaning of user input. As part of the NLU processing for the speech, Digital Assistant 106 is configured to perform processing for understanding the meaning of the speech, which involves identifying one or more intentions and one or more entities corresponding to the speech. After understanding the meaning of the speech, Digital Assistant 106 can perform one or more actions or operations in response to the understood meaning or intention. For the purposes of this disclosure, it is assumed that these speeches are text speeches directly provided by the user 108 of Digital Assistant 106, or the result of converting input speech speeches into text form. However, this is not intended to limit or constrain in any way.

[0037] For example, user 108 can request to order a pizza by providing a statement such as "I want to order a pizza." Upon receiving such a statement, digital assistant 106 is configured to understand the meaning of the statement and take an appropriate action. The appropriate action may involve responding to the user's input, for example, by requesting information about the type of pizza the user wishes to order, its size, and any toppings. The responses provided by digital assistant 106 can also be in natural language and are typically in the same language as the input statement. As part of generating these responses, digital assistant 106 can perform natural language generation (NLG). To enable the user to order a pizza via a conversation between the user and digital assistant 106, the digital assistant can guide the user to provide all the necessary information for ordering the pizza and then order it at the end of the conversation. Digital assistant 106 can end the conversation by outputting information to the user indicating that the pizza has been ordered.

[0038] At the conceptual level, the digital assistant 106 performs various processes in response to utterances received from a user. In some embodiments, this processing involves a series of processing steps or a pipeline of processing steps, including, for example (using NLU), understanding the meaning of the input utterance, determining an action to be performed in response to the utterance, performing the action where appropriate, generating a response to be output to the user in response to the user's utterance, and outputting the response to the user, etc. NLU processing may include parsing the received input utterance to understand the structure and meaning of the utterance, refining and reformulating the utterance to develop a better comprehensible form (e.g., logical form) or structure of the utterance. Generating a response may include using natural language generation (NLG) techniques. Therefore, natural language processing (NLP) performed by the digital assistant may include a combination of NLU processing and NLG processing.

[0039] NLU processing performed by a digital assistant (such as digital assistant 106) may include various NLU-related processes such as sentence parsing (e.g., tokenizing, lemmatizing, identifying part-of-speech tags, identifying named entities in a sentence, generating dependency trees to represent sentence structure, splitting a sentence into clauses, analyzing individual clauses, resolving anaphoras, performing chunking, etc.). In some embodiments, NLU processing, or a portion thereof, is performed by digital assistant 106 itself. In some other embodiments, digital assistant 106 may use other resources to perform some NLU processing. For example, the syntax and structure of an input utterance sentence may be identified by processing the sentence using a parser, part-of-speech tagger, and / or named entity recognizer. In one implementation, for the English language, a parser, part-of-speech tagger, and named entity recognizer provided by the Stanford NLP Group are used to analyze sentence structure and syntax. These are provided as part of the Stanford CoreNLP toolkit.

[0040] While the various examples provided in this disclosure illustrate utterances in the English language, this is merely illustrative. In some embodiments, the digital assistant 106 is also capable of processing utterances in languages ​​other than English. The digital assistant 106 may provide subsystems (e.g., components implementing NLU functionality) configured to perform processing for different languages. These subsystems may be implemented as pluggable units that can be invoked from the NLU core server using service calls. This makes NLU processing flexible and scalable for each language, including allowing different processing sequences. Language packs may be provided for individual languages, wherein the language packs may register a list of subsystems that can be served from the NLU core server.

[0041] Digital assistants (such as...) can be enabled through various channels (such as, but not limited to, certain applications, social media platforms, various messaging services and applications (e.g., instant messaging applications), and other applications or channels). Figure 1 The digital assistant 106 depicted is available or accessible to its user 108. A single digital assistant can be configured with several channels, allowing it to run on different services simultaneously and access them through different services.

[0042] Digital assistant or chatbot systems typically incorporate or are associated with one or more skills. In some embodiments, these skills are individual chatbots (called skill-based bots) configured to interact with users and perform specific types of tasks, such as tracking inventory, submitting time cards, creating expense reports, ordering food, checking bank accounts, making appointments, purchasing widgets, etc. For example, for... Figure 1 In the depicted embodiments, the digital assistant or chatbot system 106 includes skills 116-1, 116-2, etc. For the purposes of this disclosure, the terms "skill" and "skills" are used synonymously with the terms "skill bot" and "skill bots," respectively.

[0043] Each skill associated with the digital assistant helps the user complete tasks through a conversation with the user, wherein the conversation may include a combination of text or audio input provided by the user and responses provided by the skill bot. These responses may be in the form of text or audio messages to the user and / or provided using simple user interface elements (e.g., a selection list) presented to the user for selection.

[0044] There are various methods to associate or add skills or skill bots to a digital assistant. In some instances, skill bots can be developed by a business and then added to a digital assistant using DABP 102, for example, through a user interface provided by DABP 102 for registering skill bots to a digital assistant. In other instances, skill bots can be developed and created using DABP 102 and then added to a digital assistant created using DABP 102. In still other instances, DABP 102 provides an online digital store (called the “Skill Store”) that offers multiple skills covering a wide variety of tasks. Skills offered through the Skill Store can also be exposed to various cloud services. To add a skill to a digital assistant created using DABP 102, a DABP 102 user can access the Skill Store via DABP 102, select the desired skill, and instruct the selected skill to be added to the digital assistant created using DABP 102. Skills from the Skills Store can be added to the digital assistant as is or in modified form (for example, a DABP 102 user can select and clone a specific skill bot offered by the Skills Store, customize or modify the selected skill bot, and then add the modified skill bot to the digital assistant created using the DABP 102).

[0045] Various architectures can be used to implement digital assistant or chatbot systems. For example, in some embodiments, a digital assistant created and deployed using DABP 102 can be implemented using a master bot / secondary (or sub)bot paradigm or architecture. According to this paradigm, the digital assistant is implemented as a master bot that interacts with one or more secondary bots that are skill-based bots. For example, in... Figure 1 In the depicted embodiments, digital assistant 106 includes a master robot 114 and skill robots 116-1, 116-2, etc., which are secondary robots of the master robot 114. In some embodiments, digital assistant 106 itself is considered to act as the master robot.

[0046] Digital assistants implemented using a master-slave robot architecture enable users to interact with multiple skills through a unified user interface (i.e., via the master robot). When a user engages with the digital assistant, the master robot receives the user input. The master robot then performs processing to determine the meaning of the user's input. It then determines whether the task requested in the user's utterance can be handled by the master robot itself; otherwise, the master robot selects the appropriate skill robot to handle the user's request and routes the session to the selected skill robot. This allows users to engage with the digital assistant through a common, single interface while still having the ability to use several skill robots configured to perform specific tasks. For example, for a digital assistant developed for an enterprise, the master robot can interface with skill robots with specific functionalities, such as a CRM robot for performing functions related to Customer Relationship Management (CRM), an ERP robot for performing functions related to Enterprise Resource Planning (ERP), an HCM robot for performing functions related to Human Capital Management (HCM), and so on. In this way, the end user or consumer of the digital assistant only needs to know how to access the digital assistant through a common master robot interface, while multiple skill robots are available in the background to handle user requests.

[0047] In some embodiments, in a master robot / sub-robot infrastructure, a master robot is configured to know a list of available skill robots. The master robot can access metadata identifying the various available skill robots, and for each skill robot, the skill robot's capabilities include tasks that can be performed by the skill robot. Upon receiving a user request in the form of a utterance, the master robot is configured to identify or predict from among the multiple available skill robots that a particular skill robot can best serve or handle the user request. The master robot then routes the utterance (or a portion of the utterance) to that particular skill robot for further handling. Thus, control flows from the master robot to the skill robot. The master robot can support multiple input and output channels. In some embodiments, routing can be performed by means of processing performed by one or more available skill robots. For example, as discussed below, skill robots can be trained to infer the intent of a utterance and determine whether the inferred intent matches an intent configured for a skill robot. Therefore, routing performed by the master robot can involve the skill robot transmitting an instruction to the master robot indicating whether the skill robot is already configured with an intent suitable for handling the utterance.

[0048] Although Figure 1 The embodiments shown illustrate a digital assistant 106 including a main robot 114 and skill robots 116-1, 116-2, and 116-3, but this is not intended to be limiting. A digital assistant may include various other components (e.g., other systems and subsystems) that provide the functionality of the digital assistant. These systems and subsystems may be implemented solely in software (e.g., code, instructions stored on a computer-readable medium and executable by one or more processors), solely in hardware, or in implementations using a combination of software and hardware.

[0049] The DABP 102 provides the infrastructure, services, and features that enable DABP 102 users to create digital assistants (including one or more skill bots associated with the digital assistant). In some instances, skill bots can be created by cloning existing skill bots, for example, cloning skill bots provided by a skill store. As previously mentioned, the DABP 102 can provide a skill store or skill catalog that offers multiple skill bots for performing various tasks. DABP 102 users can clone skill bots from the skill store. Cloned skill bots can be modified or customized as needed. In some other instances, DABP 102 users create skill bots from scratch using the tools and services provided by the DABP 102.

[0050] In some embodiments, at a high level, creating or customizing a skill robot involves the following activities:

[0051] (1) Configure settings for new skill robots

[0052] (2) Configure one or more intentions for the skill robot

[0053] (3) Configure one or more entities for one or more intents.

[0054] (4) Training Skills Robot

[0055] (5) Create a dialogue flow for the skill robot

[0056] (6) Add custom parts to the skill robot as needed.

[0057] (7) Test and deploy skill robots

[0058] Each of the above activities is briefly described below.

[0059] (1) Configuring settings for new skill robots: Various settings can be configured for skill robots. For example, skill robot designers can specify one or more invocation names for the skill robot being created. These invocation names (which serve as identifiers for the skill robots) can then be used by the digital assistant user to explicitly invoke the skill robot. For example, a user can include the invocation name in their speech to explicitly invoke the corresponding skill robot.

[0060] (2) Configuring one or more intents and associated example utterances for the skill robot: The skill robot designer specifies one or more intents (also called robot intents) for the skill robot being created. The skill robot is then trained based on these specified intents. These intents represent categories or classifications that the skill robot is trained to infer in response to input utterances. Upon receiving a utterance, the trained skill robot infers the intent of the utterance, wherein the inferred intent is selected from a predefined set of intents used to train the skill robot. The skill robot then takes an appropriate action to respond to the utterance based on the intent inferred for that utterance. In some instances, the intents of the skill robot represent tasks that the skill robot can perform for a digital assistant user. Each intent is assigned an intent identifier or intent name. For example, for a skill robot trained for a bank, the intents specified for the skill robot could include “CheckBalance”, “TransferMoney”, “DepositCheck”, etc.

[0061] For each intent defined for a skill bot, the skill bot designer can also provide one or more example utterances that represent and illustrate the intent. These example utterances are designed to represent the utterances a user would input to the skill bot in response to that intent. For example, for the CheckBalance intent, example utterances could include “What's my savings account balance?”, “How much is in my checking account?”, “How much money do I have in my account?”, etc. Therefore, various permutations of typical user utterances can be designated as example utterances for the intent.

[0062] These intentions and their associated example utterances are used as training data for training the skill robot. Various training techniques can be used. As a result of this training, a predictive model is generated, which is configured to take utterances as input and output the inferred intentions for those utterances. In some instances, the input utterances are provided to an intention analysis engine (e.g., a rule-based or machine learning-based classifier executed by the skill robot), which is configured to use the trained model to predict or infer the intentions of the input utterances. The skill robot can then take one or more actions based on the inferred intentions.

[0063] (3) Configuring Entities for One or More Intents of a Skilled Robot: In some instances, it may be necessary to attach context so that the skilled robot can respond correctly to user utterances. For example, there may be cases where user input utterances are broken down into the same intent in the skilled robot. For example, in the example above, the utterances “What's my savings account balance?” and “How much is in my checking account?” are both broken down into the same CheckBalance intent, but these utterances are different requests asking different things. To clarify such requests, one or more entities can be added to the intent. Using the example of a banking skilled robot, an entity called Account_Type (whose definitions are referred to as the values ​​of “checking” and “saving”) enables the skilled robot to parse user requests and respond appropriately. In the example above, although these utterances are broken down into the same intent, the values ​​associated with the Account_Type entity for the two utterances are different. This allows the skilled robot to perform potentially different actions for the two utterances, even though the two utterances are broken down into the same intent. One or more entities can be specified for certain intents configured for a skill bot. Entities are thus used to add context to the intent itself. Entities help to more fully describe the intent and enable the skill bot to fulfill the user's request.

[0064] In some embodiments, there are two types of entities: (a) built-in entities provided by DABP 102; and (2) custom entities that can be specified by the skill bot designer. Built-in entities are general-purpose entities that can be used with a wide variety of bots. Examples of built-in entities include, but are not limited to, entities relating to time, date, address, number, email address, duration, cycle time, currency, phone number, URL, etc. Custom entities are used for more customized applications. For example, for banking skills, the Account_Type entity can be defined by the skill bot designer to enable various banking transactions by examining keywords entered by the user (such as current account, savings, credit card, etc.).

[0065] (4) Training the Skill Robot: The skill robot is configured to receive user input in the form of utterances, parse or otherwise process the received input, and identify or select intents associated with the received user input. As indicated above, for this purpose, the skill robot must be trained. In some embodiments, the skill robot is trained based on intents configured for the skill robot and example utterances associated with those intents (collectively, training data), such that the skill robot can decompose user input utterances into one of its configured intents. In some embodiments, the skill robot uses a predictive model trained using the training data and allows the skill robot to discern what the user says (or, in some cases, is attempting to say). DABP 102 provides a variety of different training techniques that can be used by skill robot designers to train skill robots, including various machine learning-based training techniques, rule-based training techniques, and / or combinations thereof. In some embodiments, a portion of the training data (e.g., 80%) is used to train the skill robot model and another portion (e.g., the remaining 20%) is used to test or validate the model. Once trained, the trained model (sometimes referred to as the trained skill robot) can be used to process and respond to the user's utterances. In some cases, a user's utterance may be a question that requires only a single answer and no further conversation. To handle this, a Q&A (question and answer) intent can be defined for a skill bot. Q&A intents are created in a similar manner to regular intents. The dialogue flow used for a Q&A intent may differ from that used for a regular intent. For example, unlike a regular intent, the dialogue flow used for a Q&A intent may not involve prompts to solicit additional information from the user (e.g., the value of a specific entity).

[0066] (5) Creating a Dialogue Flow for the Skilled Robot: The dialogue flow specified for the skilled robot describes how it reacts when different intentions of the skill robot are broken down in response to received user input. The dialogue flow defines the actions or behaviors the skill robot will take (e.g., how the skill robot responds to user utterances, how it prompts the user for input, and how it returns data). The dialogue flow is like the process followed by the skill robot. Figure 1 Similarly, skill bot designers specify the dialogue flow using languages ​​such as Markdown. In some embodiments, a version of YAML called OBotML can be used to specify the skill bot's dialogue flow. The dialogue flow definition for the skill bot acts as a model of the session, a model that allows the skill bot designer to carefully orchestrate the interactions between the skill bot and the users it serves.

[0067] In some embodiments, the dialogue flow definition of a skill robot comprises three parts, as described below:

[0068] (a) Context section

[0069] (b) Default Transformation Section

[0070] (c) State section

[0071] Context Section: Skill bot designers can define variables used in the session flow in the context section. Other variables that can be named in the context section include, but are not limited to: variables for error handling, variables for built-in or custom entities, and user variables that enable the skill bot to recognize and save user preferences.

[0072] Default Transition Section: Transitions for the skill bot can be defined in the dialogue flow state section or in the default transition section. Transitions defined in the default transition section act as fallbacks and are triggered when no applicable transition is defined within the state or when the conditions required to trigger a state transition cannot be met. The default transition section can be used to define routes that allow the skill bot to handle unexpected user actions appropriately.

[0073] State Section: The dialogue flow and its associated operations are defined as a sequence of temporary states that manage the logic within the dialogue flow. Each state node within the dialogue flow definition is named a component, which provides the functionality required at that time in the dialogue. Therefore, states are built around components. States contain component-specific properties and define transitions to other states that are triggered after a component's execution.

[0074] Special scenarios can be handled using the state section. For example, sometimes you might want to offer users the option to temporarily leave the first skill they are interacting with to do something in a second skill within the digital assistant. For instance, if a user is busy with a conversation with the shopping skill (e.g., the user has made some purchase choices), the user might want to jump to a banking skill (e.g., the user might want to make sure he / she has enough money for the purchase) and then return to the shopping skill to complete the user's order. To address this, the state section in the first skill's dialogue flow definition can be configured to initiate an interaction with a second, different skill within the same digital assistant and then return to the original dialogue flow.

[0075] (6) Adding Custom Parts to the Skill Robot: As described above, the states specified in the skill robot's dialogue flow are named with parts that provide the required functionality corresponding to those states. Parts enable the skill robot to perform functions. In some embodiments, DABP 102 provides a set of pre-configured parts for performing a wide variety of functions. The skill robot designer can select one or more of these pre-configured parts and associate them with states in the skill robot's dialogue flow. The skill robot designer can also use the tools provided by DABP 102 to create custom parts or new parts and associate custom parts with one or more states in the skill robot's dialogue flow.

[0076] (7) Testing and Deploying Skilled Robots: DABP 102 provides several features that enable skilled robot designers to test the skilled robots they are developing. The skilled robots can then be deployed and included in digital assistants.

[0077] While the above description outlines how to create skill bots, similar techniques can be used to create digital assistants (or master bots). At the master bot or digital assistant level, built-in system intents can be configured for the digital assistant. These built-in system intents are used to identify general tasks that the digital assistant (i.e., the master bot) can handle without invoking the skill bots associated with it. Examples of system intents defined for the master bot include: (1) Exit: applicable when a user signals that they want to exit the current session or context within the digital assistant; (2) Help: applicable when a user requests help or direction; and (3) UnresolvedIntent: applicable to user input that doesn't quite match the Exit and Help intents. The digital assistant also stores information about one or more skill bots associated with it. This information enables the master bot to select the specific skill bot to handle utterances.

[0078] At the master robot or digital assistant level, when a user inputs a phrase or utterance into the digital assistant, the digital assistant is configured to perform processing to determine how to route the utterance and related conversation. The digital assistant uses a routing model to determine this, which can be rule-based, ML-based, or a combination thereof. The digital assistant uses the routing model to determine whether the conversation corresponding to the user's input utterance should be routed to a specific skill for handling, handled by the digital assistant or the master robot itself according to built-in system intent, or handled as a different state within the current conversation flow.

[0079] In some embodiments, as part of this process, the digital assistant determines whether the user input utterance explicitly identifies a skill bot using its invocation name. If the invocation name is present in the user input, it is considered an explicit invocation of the skill bot corresponding to that name. In this scenario, the digital assistant can route the user input to the explicitly invoked skill bot for further processing. In some embodiments, if no specific invocation or explicit invocation is present, the digital assistant evaluates the received user input utterance and calculates a confidence score for the skill bot and system intent associated with the digital assistant. The score calculated for the skill bot or system intent indicates how likely the user input represents a task that the skill bot is configured to perform or represents a system intent. Any system intent or skill bot whose associated calculated confidence score exceeds a threshold (e.g., a confidence threshold routing parameter) is selected as a candidate for further evaluation. The digital assistant then selects a specific system intent or skill bot from the identified candidates for further processing of the user input utterance. In some embodiments, after one or more skill bots are identified as candidates, for each intent, the intents associated with those candidate skills are evaluated (using a model trained for each skill) and a confidence score is determined. Any intent with a confidence score exceeding a threshold (e.g., 70%) is generally considered a candidate intent. If a specific skill bot is selected, the user utterance is routed to that skill bot for further processing. If a system intent is selected, the master bot itself performs one or more actions based on the selected system intent.

[0080] Classification of digital assistant tiers or skill bot tiers

[0081] The template-based classification technique described in this paper can be applied to the master bot / digital assistant level and / or the skill bot level. For example, in a chatbot system comprising a digital assistant or master bot and one or more skill bots, template-based classification processing can be performed by a classifier for the master bot / digital assistant and / or a classifier for the skill bots. For example, as... Figure 2 The main robot described may include a classifier 224 that can be implemented using a template.

[0082] Figure 2 This is a simplified block diagram of a master robot (MB) system 200 according to certain embodiments. The MB system 200 may be implemented in software only, in hardware only, or in a combination of hardware and software. The MB system 200 includes a preprocessing subsystem 210, a routing subsystem 220, and a data storage device 250. Figure 2The depicted MB system 200 is merely an example of component arrangement in the main robot. Those skilled in the art will recognize many possible variations, alternatives, and modifications. For example, in some embodiments, the MB system 200 may have a more... Figure 2 The systems or components shown may have more or fewer subsystems, may combine two or more subsystems, or may have different subsystem configurations or arrangements.

[0083] Language processing subsystem 210 is configured to process utterances 202 provided by a user. As discussed in the chatbot system overview section above, this processing may involve performing NLU processing to understand the meaning of the utterances. Therefore, language processing subsystem 210 may include a language detector 212 configured to detect the language of utterance 202 and a language parser 214 configured to parse utterance 202 to understand its meaning. Processing performed by language processing subsystem 210 may include, for example, sentence parsing (e.g., token parsing, part-of-speech tagging, identifying part-of-speech tags in a sentence, identifying named entities in a sentence, generating a dependency tree to represent sentence structure, splitting a sentence into clauses, analyzing individual clauses, decomposing pronouns, performing chunking, etc.). Language processing subsystem 210 outputs utterance 202 along with extracted information 205 generated as a result of the above processing. For example, extracted information 205 may include a dependency tree, a list of part-of-speech tags, etc. In some embodiments, language processing subsystem 210 may reformat utterance 202 for input to routing subsystem 220. Reformatting can involve, for example, rearranging the word order in utterance 202, dividing utterance 202 into different parts to process each part separately, correcting grammatical errors, etc.

[0084] Routing subsystem 220 is configured to determine which robot (e.g., one of a set of available skill robots 216-1 to 216-3 or MB system 200 itself) should handle the user utterance (e.g., utterance 202). Specifically, classifier 224 of routing subsystem 220 is configured to determine whether the user utterance represents a specific robot (primary or secondary / skillful) and / or whether the user utterance represents a specific intent already configured for a specific robot. For example, as discussed earlier, a skill robot may be configured with one or more robot intents. Each robot intent may have its own dialogue flow and be associated with one or more tasks that the skill robot can perform. Upon determining that utterance 202 represents an intent configured for a specific robot or for a specific robot, routing subsystem 220 may invoke the specific robot and transmit utterance 202 (possibly along with extracted information 205) as input 235 to the specific robot.

[0085] The classifier 224 can be implemented using a rule-based model, an ML-based model, or both. For example, in some embodiments, the classifier 224 may include a neural network trained on training data 254. For each skill robot 216, the training data 254 may include a set of corresponding example utterances (e.g., two or more example utterances for each intention configured for the skill robot 216). The training data 254 may be used to pre-train a neural network to infer the probability that a user utterance represents the skill robot 216 or one of the intentions already configured for the skill robot 216.

[0086] Additionally, in some embodiments, classifier 224 may be configured to apply a set of rules 252 to determine the likelihood that a user utterance represents either the skill robot 216 or one of the intentions already configured for the skill robot 216. The determinations made as a result of applying the rules may be combined with determinations made using an ML model. For example, the neural network of classifier 224 may output a first confidence score, the result of applying the rules may include a second confidence score, and classifier 224 may mathematically combine these two confidence scores or compare them with each other to determine which robot the utterance 202 is most likely to represent.

[0087] Rule 252 can be implemented in various ways. For example, a rule can be expressed as a logical statement including one or more conditions relating to the sentence structure of a utterance. In some embodiments, rule 252 is expressed as a template, and more specifically, as a regex. As indicated above, a regex is a logical expression that specifies a string pattern. For example, in order for a utterance to be matched with a pizza ordering robot, the regex of the pizza ordering robot could specify that the utterance must contain a combination of the word "pizza" and at least one word from a set of synonyms used for placing an order (e.g., "order", "buy", "purchase", etc.).

[0088] A string pattern for a template can be obtained from example utterances (e.g., the same utterances included in training data 254). An example of how such a string pattern is obtained is described below. Typically, a string pattern for a template is formed by iteratively refining a group of one or more example utterances to produce a more general form of the example utterance(s). Refinement may involve analyzing the syntax and structure of the example utterances (e.g., to identify parts of speech (POS), named entities, etc.). Therefore, the string pattern can include POS information such that the string pattern is formed from a sequence of words, where each word in the sequence corresponds to a specific part of speech (e.g., noun, pronoun, verb, etc.).

[0089] Classifier 224 can calculate a confidence score for each template that matches a user utterance. This confidence score indicates the degree to which the user utterance matches the string pattern of the template. For example, if utterance 202 exactly matches the string pattern, classifier 224 can calculate a confidence score of "1". If utterance 202 does not match (e.g., the utterance does not contain any words in the sequence specified in the string pattern), classifier 224 can calculate a confidence score of "0". Classifier 224 can assign the confidence score to the robot intent corresponding to the template. If the confidence score of any particular robot intent exceeds a threshold, classifier 224 can consider utterance 202 to represent that particular robot intent.

[0090] In some embodiments, classifier 224 may support partial template matching. For example, classifier 224 may calculate a template confidence score (e.g., a score between 0 and 1) based on the percentage of user utterances that match a template. Additionally, classifier 224 may take into account the length of template matches as part of calculating the template score (e.g., weighting the template score by the number of matched words / elements) or when ranking template scores across different intents. For example, if templates corresponding to different intents for the same user utterance have similar scores, classifier 224 may determine that the intent for which its template has the longest match is the intent most likely represented by the user utterance.

[0091] Furthermore, in some embodiments, classifier 224 may calculate an overall confidence score for the robot based on individual scores for the intentions already configured for the robot (e.g., the average of all template scores for a particular robot). If the overall score for a particular robot exceeds a threshold (which may be the same as or different from the threshold for template scores), then classifier 224 may consider utterance 202 to represent that particular robot.

[0092] like Figure 2 As shown, rule 252 and training data 254 can be stored in a data storage device 250 accessible to the MB system 200. The data storage device 250 can also store skill robot information 258. Skill robot information 258 may include information about the capabilities of skill robots 216 and configuration information for skill robots 216. For example, skill robot information 258 may include a list of robot intentions for each skill robot 216 and a description of the tasks associated with these robot intentions.

[0093] Templates can be used to perform matching at the sentence or utterance level. In some embodiments, classifier 224 can also perform matching at the word level. For example, rule 252 may include a list of words with associated confidence scores, where the confidence score of a word indicates the probability that the word represents a particular intent. Thus, classifier 224 can be configured to determine the confidence score of each word in the user utterance by matching words in the user utterance with words in the list. Word-level scoring can be combined with template-based scoring to more accurately determine which robot intent the user utterance best represents. When a user utterance scores highly based on template matching, it is reasonable to determine that the user utterance represents the intent corresponding to the template. However, there may be cases where the user utterance does not closely match any template but still has a high word score (e.g., one or more words in the user utterance exceed a threshold).

[0094] After identifying a robot (e.g., based on an assessment of a confidence score generated as a result of matching a template), the routing subsystem 220 may hand over processing of utterance 202 to the identified robot. As part of this handover, the routing subsystem 220 may determine what content to provide as input 235 to the identified robot. In some instances, the routing subsystem 220 may provide less than all of utterance 202 as input 235. This may occur, for example, when utterance 202 does not contain information to be processed by the identified robot (e.g., utterance 202 is only intended to invoke the identified robot). Further, in some embodiments, the routing subsystem 220 may include at least some of the extracted information 205 in the input 235.

[0095] Figure 3 This is a simplified block diagram of a skill robot system 300 according to certain embodiments. The SB system 300 can be implemented. Figure 2 Any skill robot 216, and can be implemented in software only, hardware only, or a combination of hardware and software. Figure 3 As shown, the SB system 300 may include a classifier 320 and a session manager 330.

[0096] Classifier 320 can be compared with Figure 2The classifier 320 operates in a similar manner to classifier 224, and classifier 320 can provide the functionality described above regarding classifier 224. Classifier 320 can be implemented using a rule-based model or an ML-based model, or both, and can use the same inputs as classifier 224 (e.g., utterance 202 and extracted information 205). In some embodiments, classifier 320 is configured to infer the intent of utterance 202 by applying a template, which is stored as rule 352 in a data storage device 350 accessible to the skill robot system 300. Figure 3 As shown, the data storage device 350 can be configured to store rules 352 and intent information 354. Rules 352 may, for example, include a separate regex for each intent of the skill robot system 300. Intent information 354 may include example utterances from which rules 352 are derived and / or other intent-related configuration information. For example, intent information 354 may include one or more confidence thresholds, wherein these values ​​are specified by the skill robot designer as part of configuring the skill robot system 300.

[0097] Classifier 320 can use rule 352 and possibly also a trained AI model to determine whether utterance 202 represents a specific intent already configured for the skill bot (i.e., one of the bot intents associated with the skill bot system 300). Each skill bot in the chatbot system can include its own classifier 320. Further, in some embodiments, classifier 224 is configured to make its determination after the digital assistant / master bot has determined that utterance 202 represents a specific skill bot. For example, master bot system 200 can use classifier 224 to determine that utterance 202 represents the skill bot corresponding to skill bot system 300. Based on this determination, master bot system 200 can provide utterance 202 and extracted information 204 as input to classifier 320, allowing classifier 320 to match utterance 202 with one or more templates to identify the bot intent that utterance 202 best represents. In this way, each skill bot can manage and apply its own templates derived from example utterances specific to the intents already configured for the skill bot.

[0098] After identifying the intent most representative of utterance 202, classifier 320 can transmit the indication 322 of the identified intent to session manager 330. Figure 3 In some embodiments, the session manager 330 is shown as being local to the skill robot system 300. However, the session manager 330 can be shared between the master robot and multiple skill robots. Therefore, in some embodiments, the session manager 330 is local to the digital assistant or the master robot.

[0099] In response to receiving instruction 322, session manager 330 can determine an appropriate response 335 to dialogue 202. For example, response 335 could correspond to... Figure 1 One of the DA responses 112 in the embodiments, and may be an action or message specified in the dialogue flow definition 355 configured for the skill robot system 300. For example, the session manager 330 may access the dialogue flow definition 355 from the data storage device 350 and, based on the dialogue flow definition 355, determine the dialogue flow state associated with the indicated intent as the next state to transition to. The session manager 330 may determine the response 335 based on further processing of the utterance 202 and / or information obtained from the utterance 202 (e.g., extracted information 204). For example, if the utterance 202 is “Checkbalance in savings,” the session manager 330 may transition to a dialogue flow state that presents the user with a dialogue related to the user’s savings account. The session manager 330 may transition to this state based on an indication 322 indicating that the identified intent is the “CheckBalance” intent configured for the skill robot system 300, and also based on recognizing the value of “saving” already extracted for the “Account_Type” entity.

[0100] Example techniques for generating templates

[0101] One method for determining the degree of match between a user utterance and a template is to identify the longest common subsequence (LCS) shared by the user utterance and the template. Based on the LCS, a confidence score can be calculated for the intent associated with the template. For example, even though the utterance contains additional words, the utterance “I want to order pizza” can be considered a 100% match with a template that includes the pattern “I order pizza”. In some embodiments, such a match returns a score of 1.0, which represents a perfect template match. This matching technique essentially attempts to match at least one substring of the template, and ideally, the entire template. User utterances can be matched against templates associated with different robot intents to classify the utterance as corresponding to the intent of any template that has the highest score among the different robot intents. As explained below, LCS can also be used to create templates representing multiple example utterances.

[0102] Figure 4 An example template set 400 according to certain embodiments is illustrated. Template set 400 includes templates 402, 404, 406, and 418, and may correspond to the same robot intent (e.g., for...). Figure 3The templates 402, 404, and 406 are associated with the intent configured in the skill robot system 300. Templates 402, 404, and 406 are derived from example utterances 412, 414, and 416, respectively, such that each example utterance has a corresponding template. For example, templates 402, 404, and 406 can be based on the exact string patterns of their corresponding example utterances 412, 414, and 416. To determine whether a user utterance represents a robot intent, the user utterance can be matched against each of the templates 402, 404, and 416. The matching results can include a confidence score for each template 402, 404, and 416, where the confidence score indicates the degree to which the user utterance represents a robot intent.

[0103] Creating and matching templates for each example utterance provided to a chatbot (e.g., example utterances 412, 414, and 416) can be cumbersome. If multiple variations or permutations of the same example utterance exist, it's possible to generate separate templates for each permutation, resulting in a large number of templates to match. The number of templates can be reduced, for example, by using an LCS-based inductive process to identify subsequences common to (shared between) two example utterances and using these subsequences as templates for both example utterances. Figure 4 In some embodiments, if example utterances 412 and 414 share the same subsequence, templates 402 and 404 can be replaced with a single template that captures that shared subsequence. However, templates obtained solely through LCS-based induction cannot adequately capture the full meaning behind the example utterances. For example, simply identifying the longest common subsequence between an example utterance and another example utterance does not take into account which parts of the example utterance are optional to its meaning and which parts are mandatory.

[0104] Regex provides a method for reducing the number of templates to match while capturing the semantic meaning behind example utterances, generalizing these templates in a way that preserves the meaning of the example utterances. Furthermore, regex is advantageous because it is not only machine-understandable but also human-understandable, which facilitates rule debugging and customization by robot developers. In some embodiments, a regex may include patterns (e.g., sequences of words / substrings), and for each word or substring, the desired number of instances of that word or substring (0 or more, 1 or more, etc.). A regex may further include linguistic information (e.g., information about nouns, verbs, etc.). As an example, in addition to the part of speech of a particular word in the context of the user utterance, a regex may also specify the order in which a particular word should appear in the user utterance.

[0105] Ideally, each intent configured for a chatbot has only one template generated for it, allowing the determination of whether a user's utterance represents that intent to be made by matching it against a single template. For example, as Figure 4 The description allows for the generation of a generalized template 418 that captures the structure and meaning of each of the example utterances 412, 414, and 416. User utterances can be matched against the generalized template 418, and this may be sufficient to determine whether the user utterance represents a robot intent associated with the example utterances 412, 414, and 416. User utterances can optionally be matched against templates 402, 404, and 406 to determine which of the example utterances 412, 414, or 416 the user utterance most closely resembles.

[0106] Generalized template 418 can be generated by simplifying templates 402, 404, and 406 into more general forms through applied language understanding. The simplification process can involve generalizing templates 402, 404, and 406 based on parts of speech (such as nouns, subjects, predicates, etc.) and the relationships between these parts of speech. Examples of template generalization techniques applied at the word and sentence levels are described below.

[0107] In some embodiments, the template is generated by a rule generator. The rule generator can be part of a digital assistant / master robot. For example, such as... Figure 2 As depicted, the main robot system 200 may include a rule generator 260. Alternatively, each skill robot (e.g., Figure 3 The skill-based chatbot system 300 may include its own rule generator. In another embodiment, the rule generator may be part of a separate computer system outside the chatbot system. The rule generator may be implemented in hardware and / or software (e.g., as program code) and may be configured to perform language processing for identifying parts of speech (e.g., key nouns, verbs, etc.) in example utterances. The rule generator may be further configured to determine the sentence structure of the example utterances. Based on language processing, the rule generator may generate a separate regex for each example utterance as an initial template set (e.g., templates 402, 404, and 406).

[0108] After generating an initial template set, the rule generator can compare the templates with each other and, based on the comparisons, identify patterns to learn rules encoded as one or more new regexes that supplement or replace the initial template set. The new regexes are generalizations of the initial template set (e.g., generalized template 418). For example, a new regex can be identified as capturing shared features of two or more previous regexes associated with the same intent. This can be done iteratively, one pair of regex at a time. A new regex can also be a more generalized version of a previous regex. Thus, generalized template 418 can be formed over multiple iterations, where each iteration involves further refining or simplifying at least one existing template.

[0109] Additionally, in some embodiments, the rule generator is configured to perform a filtering process, whereby if the same regex is associated with different robot intentions, the regex is discarded because it is not specific enough (since matching such a regex is insufficient to indicate any specific robot intention among different robot intentions). For example, in Figure 4 In this embodiment, if the rule generator determines during the generation of generalized template 418 that there are two alternative methods to further simplify an existing template, but the first method results in a template that is identical to a generalized template already generated for a different chatbot intent (the same chatbot or another chatbot), the rule generator may ignore the template generated by the first method. Therefore, the rule generator can compare templates / regex across intents to determine which regex to retain.

[0110] To generate generalized templates, such as Figure 2 Rule generators such as Rule Generator 260 can be configured to apply one or more heuristics. Generalization can be performed at the word level, for example, using stemming, system entity generalization, custom entity generalization, generalized parts of speech, word synonyms, and / or case neutralization.

[0111] Stem extraction: In stem extraction, words are replaced with their roots / stems. For example, “paid” (past tense of “pay”) becomes “pay” (verb base form of “pay”), so if the template includes the sequence “When was I last paid” (the time when I last paid (past tense)), the sequence will become “When was I last pay” (the time when I last paid (verb base form)).

[0112] System entity generalization: As indicated above, a chatbot system may be configured with system-level entities enumerated according to entity types. When such an entity appears in a template sequence, it can be replaced with its named entity type. For example, "pay Lauren $20" may become "pay PERSON CURRENCY".

[0113] Custom entity generalization: As indicated above, a chatbot system may be configured with custom entities (e.g., entities specific to one or more skillbots). When defining a custom entity, a list of valid values can be specified for the custom entity. For example, the Account_Type entity may be a custom entity defined for a financial bot, where the Account_Type entity may have values such as "checking" or "saving". Thus, "pay Mom from checking" may become "pay Mom from Account_Type".

[0114] Generalizing word classes: In some embodiments, certain word classes (e.g., symbols and interjections) are generalized. For example, "Hello$%^(" may become "UH SYM SYM SYM", where "UH" represents a greeting interjection and "SYM" represents a symbol.

[0115] Word synonyms: Words can be replaced with their synonyms (e.g., synonyms from VerbNet, WordNet, or some other source of known synonyms). For example, "pay Mom $20" and "send Mom $20" have the same meaning because "send" and "pay" are synonyms. In this case, "send" and "pay" can be included as options for the same template element, e.g., "(send|pay) Mom $20". In some embodiments, synonyms are learned by encoding words from example utterances into word vectors and determining that the positions of the word vectors in the vector space are close to each other. For example, the word vectors for "red" and "green" may be close to each other in the vector space because both terms are related to the concept of color.

[0116] Case neutralization: To allow users to change the capitalization of their speech, case neutralization can be used to standardize the template derived from the example speech. For example, “I want to order a pizza” might become “i want to order a pizza”.

[0117] In some embodiments, the rule generator is configured to use sentence-level generalization in combination with or instead of word-level generalization. Sentence-level generalization may include, for example, stop words, induction, subexpression / language grouping, subpattern sorting, optional subexpressions, subexpression merging, synonyms, linguistic features, and a priori subexpressions of language.

[0118] Stop words: In some instances, stop words can be removed from the template. For example, "I want to send money" might become "I want send money".

[0119] In summary: As indicated above, the example utterances can be used for generalization based on their longest common subsequence. For example, “I to send money” can be a generalization of “I want to send money for something” and “I would like to send money”.

[0120] Sub-expressions / Language Grouping: Linguistic reasoning can be applied to group parts of a utterance together into sub-patterns. For example, “I want to send $20 from my checking account to Mom” can be broken down into the following sub-patterns:

[0121] Request type: I want to

[0122] Main verb: send (to send by mail)

[0123] Direct object: $20 (20 US dollars)

[0124] Preposition / Entity Phrase 1: from my checking account

[0125] Preposition / Entity Phras 2: to Mom

[0126] The verbs, nouns, and prepositional phrases appearing in the sub-patterns can also be further broken down into their constituent parts. For example, a prepositional phrase can be divided into a preposition followed by a noun phrase. Similarly, a noun phrase can be divided into an adjective followed by a pronoun, a determiner, and then a noun. For example, in the sentence "I am going to school," the words "going to" are part of the verb phrase, while "school" is part of the noun phrase; therefore, "going" and "to" should be grouped separately from the noun phrase element (e.g., school).

[0127] Language grouping can also be performed across templates. For example, suppose there are two templates for the same intent. These two templates are of the form: AB E1 E2 (template 1) and AB E3 E4 (template 2), where E1, E2, E3, and E4 are entities. Therefore, it can be inferred that the more general form of these two templates is: AB(E1 E2|E3 E4). As another example, suppose these two templates are of the form: AB DO1 E1 and AB DO2 E2. Therefore, the general form of these two templates could be: AB(DO1 E1|DO2 E2).

[0128] Subpattern sorting: Words grouped into subpatterns can be sorted in different ways. For example, "I want to send $20 from checking to Mom" ​​can be reordered as "$20 to Mom from checking I want to send".

[0129] Optional sub-expressions: Sub-patterns can include optional and / or mandatory elements (e.g., A is optional, either A or B, either A or B, etc.). For example, “I want to send money to Mom” can be expressed in the template form: (I want to), send, money, (to Mom)*((I want), send, money, (to Mom)*), making “to Mom” optional. Optional elements can be identified by comparing templates to determine which parts are redundant. For example, if two templates for the same intent are in the form: ABC (template 1) and ABCD (template 2), then element D can be inferred to be optional. In some instances, certain types of sub-patterns (such as prepositional phrases) can be considered optional. Additionally, optional elements can be identified based on their part of speech. For example, verbs / actions and objects are often mandatory, while entities are often requested parameters and can therefore be set as optional. Adverbs are another example of elements that can be set as optional.

[0130] Combining subexpressions: Subexpressions with equivalent meanings can be combined by replacing them with compound expressions. For example, “(I would like to),send,money((I want to),send,money)” and “(I want to),send,money((I want to),send,money)” can be generalized to: (I(would like)|want to),send,money(I(want to)send,money).

[0131] Synonyms: In some embodiments, the rule generator can be configured to identify synonyms across example utterances / templates. If templates generated from two or more example utterances differ by only one word and the word has the same part of speech, the rule generator can identify the different words as synonyms. For example, the rule generator can determine that "send" and "pay" are synonyms based on the fact that "sendMom$20" and "payMom$20" are provided as example utterances for the same robot intent.

[0132] Language (semantic) features: In some embodiments, the rule generator can generate templates with subexpressions based on the presence or absence of semantic features. For example, a generalized template can specify whether a subexpression element is negated, the verb tense of the subexpression element, the requested quantity, or the request type.

[0133] Prior subexpressions for the language: In some embodiments, the rule generator can be configured with subexpressions or partial templates provided independently of any example utterances. Such subexpressions can be based on common language-related usages, such as idioms. For example, in the English language, the following might be a prior subexpression:

[0134] please*,((I want)|(I would like)|(could you)|(would you)|can you)*,(show|tell|help)me*,? *(Please*,((I want)|(could you)|(would you)|can you)*,(show|tell|help)me*,?*)

[0135] Therefore, if the rule generator generates a generalized template based on an example utterance that only includes "I want to", the rule generator can incorporate the above subexpressions into the generalized template, thereby capturing variations of the same subexpression, even though such variations are not represented in the example utterance.

[0136] As will be understood from the above discussion, generalization can involve techniques other than LCS, such as generalization based on entity knowledge and linguistic information such as synonyms and subpattern expressions. In some embodiments, generalization is performed considering individual word scores. As mentioned earlier, the classifier can perform word-level matching based on a list of words with corresponding confidence scores. Word scores can also be used to generalize templates. For example, templates can be made more general by excluding words with scores below a certain threshold, making words with scores above a certain threshold mandatory, or both. Candidate templates can be rejected if they do not contain any individual high-scoring words or consist entirely of stop words or stop phrases (help, show, can you help me to, etc.).

[0137] As indicated above, the classifier can employ templates obtained using template generation and generalization techniques. Given a set of example utterances as training examples, the rule generator can use these example utterances as a starting point for generating a corresponding initial template set. In this way, each training example can provide a basis for a corresponding template or rule. To match user utterances different from the training examples, the initial template set can be generalized to generate one or more generalized templates, allowing the classifier to match a larger set of similar utterances. For example, as discussed above, templates corresponding to the same intent can be compared to generate a more general template based on the longest common subsequence. To obtain the final generalized template (e.g., generalized template 418), multiple candidate templates can be generated in this way, and these candidate templates can then be iteratively refined until they cannot be further simplified without causing conflicts across robot intents. For example, candidate templates across different intents (e.g., intents configured for robots with the same skill or intents configured for robots with different skills). If the same candidate template is generated for two different robot intents, the candidate template can be excluded from consideration as a template for both intents.

[0138] In some embodiments, template generalization may involve backtracking or reverting to an earlier form (e.g., a previous form) of a candidate template in response to determining that the current form of a candidate template conflicts with another template (e.g., a candidate template or a final generalized template) associated with a different robot intention. For example, a rule generator may be configured to undo the last generalization step for at least one of the two templates when a conflict is detected between a template for a first robot intention and a template for a second robot intention. For example, when reverting to the earlier form of the two templates, the rule generator may generalize the templates differently so that the templates do not conflict with each other. Further, the rule generator may store a record of any generalization steps that lead to a conflict, for example, by adding such generalization steps to a blacklist. In some embodiments, the rule generator maintains a separate blacklist for each robot intention. To improve performance, the rule generator may be further configured to mark portions of a candidate template that cannot be further generalized without causing a conflict as "frozen," thereby indicating that the marked portions are in their final form.

[0139] Because generating generalized templates can involve numerous iterations, performing processing to obtain a "final" generalized template only to discover later that the generalized template conflicts with another template (e.g., during the evaluation of candidate templates for another intent) can be computationally inefficient. Therefore, in some embodiments, rule generators (e.g., rule generator 260 or rule generators for different skill-based bots operating collaboratively) are configured to perform generalization in parallel for multiple intents. For example, generalization can be performed simultaneously for all intents already configured within the chatbot system. This parallel processing may involve first performing a more conservative or "safe" generalization on an initial template for two or more configured intents, checking for conflicts between candidate templates after each generalization step, and rolling back candidate templates if a conflict is found. Examples of conservative generalization include stemming and system entity generalization, both of which can be performed with little or no risk of altering the meaning or unintentionally removing elements crucial to the meaning. After conservative generalization has been completed, the rule generator can proceed to less conservative generalization, then again check for conflicts after each generalization step and revert to an earlier template form if a conflict occurs.

[0140] Backtracking Example: Suppose "How much did I spend last month" is an example utterance of the "Balances" intent, and "How much did I spend on gas last month" is an example utterance of the "TrackSpending" intent. The rule generator can perform a generalization step that makes "on gas" optional because it is a prepositional phrase. After making "on gas" optional, the rule generator determines that there is now a conflict between the current candidate template for the TrackSpending intent and the initial template for the Balances intent because the two templates have the same pattern. In response to this determination, the rule generator can make "on gas" required again. Furthermore, the rule generator can attempt to generalize "on gas" in different ways. For example, "on gas" can be generalized to the "ExpenseType" entity, or grouped into subexpressions such as "on(gas|food|clothes)(on(gasoline|food|clothes))". If "on gas" cannot be further generalized, the rule generator can mark "on gas" as frozen and attempt to generalize different parts of the candidate template for the TrackSpending intent.

[0141] In some embodiments, the rule generator can be configured to incorporate user input (e.g., input from a skill robot developer) as part of determining the final set of generalization templates. Specifically, the rule generator can be configured to request user feedback on whether a generalization step should be accepted or rejected. For example, as indicated above, the rule generator can generate synonyms based on the proximity of word vectors in a vector space. Proximity in the vector space can indicate that words are closely related to each other (e.g., words have the same utility or meaning), but this is not always the case. In some instances, words whose vectors are close in the vector space are not synonyms. Therefore, the rule generator can ask the user to confirm that the identified words are indeed synonyms after identifying words that may be synonyms. In this way, machine learning (based on applying heuristics to the training data) can be combined with human understanding to improve the quality of the resulting templates.

[0142] Template generalization example

[0143] Figure 5 The illustration shows example utterance 510 and generalized template 520 for the robot's intent. Figure 5 In one embodiment, the robot is intended to be associated with the task of querying the balances of various bank accounts. For example... Figure 5 As shown, example statement 510 includes the following:

[0144] How much do I owe on all my credit cards?

[0145] What's the current balance on my bank account?

[0146] How much money do I have in all of my accounts?

[0147] How much money did I save last year?

[0148] What's the value of my trading account?

[0149] How much money do I have in my checking account?

[0150] What's my balance?

[0151] What's my available credit on my Visa?

[0152] What's my balance?

[0153] As indicated above, entity generalization is one of the more conservative types of generalization. Therefore, as an initial generalization step, the following elements can be generalized to the "Account_Type" entity: credit cards, cc, trading, checking, and Visa. This is in... Figure 6 It is shown in box 610.

[0154] Figure 6 The diagram illustrates the basis Figure 5 The result of a series of generalization steps in example utterance 510. Box 610 corresponds to the above combination. Figure 5 The results of the initial generalization steps discussed. For example... Figure 6 As shown, the element corresponding to the Account_Type entity has been replaced with "Account_Type". Additional processing that can be performed to produce the result shown in box 610 includes correcting "Whats" to "What's".

[0155] Box 620 shows the results of further generalization, including combining the phrases “What's my balance?” and “What's my balance?” into “What's my balance[?]”, where the parentheses indicate that the question mark is optional.

[0156] Box 620 also illustrates the result of logical reasoning that a rule generator can make based on the determination that Account_Type is followed by the word "account". Based on this determination, the rule generator can infer that Account_Type can optionally be followed by "account", or infer that "account" can replace Account_Type. This can be expressed as: [Account_Type account]([account_type account]).

[0157] Box 630 illustrates the result of applying another logical reasoning. Specifically, the rule generator can determine that in the patterns generated so far, Account_Type is often preceded by a preposition and / or modifier, such as "on all my," "on my," "in all of my," "of my," "in," and "on my." These phrases include combinations of prepositions, determiners, and possessive pronouns. Since the determiner is not important, these phrases can be rewritten as [on[all]my|in[allof my]] ([on [all]my|in [all of my]) or possibly [on in of all my] ([on all]my]). Therefore, in the context of the intent "Balances," the Account_Type entity can be expressed as the entity phrase: [on in of all my][Account_Type account] ([on all]my][Account_Type account]). Since these are optional elements, and entity phrases can appear anywhere in the string pattern (before or after other elements), entity phrases are removed to produce the string pattern shown in box 630.

[0158] Figure 7 The diagram illustrates the results of a series of generalization steps, which can be followed by... Figure 6 The generalization shown ultimately leads to Figure 5 The generalization template 520 is shown. As discussed above, in box 630, entity phrases relating to the Account_Type entity can be deleted. However, it turns out that the deletion operation weakens the meaning to some extent, because the meaning of the parts before the preposition (e.g., “What's the value”) is now unclear.

[0159] As shown in box 710, the next logical deduction can be based on the understanding that “the value,” “my balance,” “my available credit,” and “the current balance” are phrases that precede the entity phrases and are on the other side of the prepositions “in,” “of,” and “on.” Therefore, it can be inferred that these phrases are semantically equivalent and can thus be combined as follows:

[0160] What's the value, my balance, my available credit, and the current balance?

[0161] Box 720 shows that inferring "money" is an optional result. The rule generator can make this inference in different ways. One approach is to recognize that "how much" is followed by the noun "money," and then by "do." Another approach is to determine that "how much" should perhaps be followed by a noun or something being counted, but in some instances it can be omitted.

[0162] Box 730 shows an example result of inferring that "last year" is an entity representing time. In this case, the entity is used to represent the relevant time period for which balance information is requested. If it were possible to provide balances for times other than the current time, such an entity might have already been declared (e.g., as a custom entity associated with the "Balance" intent). However, if no such entity has been declared, the rule generator can still assume that time is the entity associated with the Balance intent and can omit any terms corresponding to the time entity.

[0163] Box 740 shows the result of inferring that “owe,” “have,” and “save” are synonyms. If these terms appear in a synonym list such as VerbNet, they can be combined with other synonyms in the list to create a set of alternative terms as part of a template. However, these terms are not typically presented as synonyms in a synonym list. Instead, they are only synonyms for the purpose of Balance. Therefore, the rule generator can infer that these terms are synonyms in other ways. For example, the rule generator can determine that these terms are synonyms based on the proximity of the word vectors of each of these terms in the vector space and / or based on user input confirming that these terms are indeed synonyms. Additionally, in Box 740, the question mark “?” is made optional. Generalization can be repeated until the template cannot be further simplified. For illustrative purposes, it can be assumed that the final unsimplified form of the template corresponds to the two string patterns shown in Box 740.

[0164] The two string patterns shown in box 740 correspond to Figure 5 The generalized template in [the document] is 520. Return to reference. Figure 5The generalized template 520 includes the two string patterns from box 740 as alternative string patterns 522 and 524. However, in some implementations, each final pattern can be considered an independent template. Additionally, the generalized template 520 includes the above combination. Figure 6 The optional elements discussed in box 630.

[0165] Figures 5-7 This is merely an example illustrating how a series of generalizations are performed to obtain the final form of the generalized template. However, the final form does not necessarily have to be the same as the generalized template 520. Instead, it is a combination of the above... Figures 5-7 Modifications to the generalization steps discussed are possible. For example, in Figure 7 The processing depicted in box 740 can be followed by additional steps closely related to determining the "what is" and "how much" questions for the purpose of Balances. The rule generator can perform this termination based on, for example, ontology or knowledge graphs.

[0166] As discussed earlier, templates can be expressed in the form of regex, which includes pattern information and part-of-speech (POS) information. Therefore, although Figures 5-7 The discussion focuses on the derivation of the pattern information of the generalized template 520, but it should be understood that the generalized template 520 can include more than just the pattern information. Figure 5 The two string patterns 522 and 524 described herein, along with optional elements, are used. For example, a generalized template can include information about the parts of speech corresponding to different parts of string patterns 522 and 524.

[0167] Example training process

[0168] Figure 8 A simplified flowchart 800 according to some embodiments is illustrated, which depicts the process for generating generalized templates as part of training a classifier. Figure 8 The described process can be implemented in software (e.g., code, instructions, programs) executed by one or more processing units (e.g., processors, cores) of a corresponding system, in hardware, or in a combination thereof. The software can be stored on a non-transitory storage medium (e.g., a memory device). Figure 8 The methods presented and described below are intended to be illustrative rather than restrictive. Although Figure 8 The individual processing steps, occurring in a specific sequence or order, are described, but this is not intended to be limiting. In some alternative embodiments, the steps may be performed in a different order, or some steps may be performed in parallel. In some embodiments, such as Figure 2 In the embodiments described herein, the rule generator, such as rule generator 260, can perform the operation.Figure 8 The processing described in the document.

[0169] At 802, example utterances associated with different intents are received. For example, a chatbot system may include multiple skill bots, and one or more example utterances may be provided to each skill bot. Example utterances may, for example, be utterances created and stored by skill bot developers for use as training data. In some embodiments, each skill bot is trained independently. However, as indicated above, template generalization may involve performing generalization in parallel for multiple chatbots. In some instances, the example utterances received at 802 may include utterances associated with different intents already configured for the same chatbot (e.g., a master bot or a skill bot). Further, in some instances, the example utterances received at 802 may include utterances associated with different intents already configured for different chatbots (e.g., utterances for a first skill bot and utterances for a second skill bot). Each intent may have certain declared entities / variables associated with it. For example, a cost tracking intent may be configured with entities such as "Expense_Type" and "Amount". Additionally, there may be certain system-level entities that are not specific to any particular intent, such as "Date" and "Time".

[0170] At position 804, part-of-speech tags are extracted from the example utterance received at position 802. For example, each example utterance can be input into a language processing subsystem (such as...). Figure 2 The language processing subsystem 210 in the language processing subsystem is used to generate POS tags for example utterances. POS tags can be generated using a standard language parser and using generic parts of speech.

[0171] At 806, using the example utterance received in 802 and the POS extracted in 804, an initial template as a regex is generated. Each template in the initial template can be expressed as a regex including a string pattern. A regex can support the ability to specify any / all / one of the following concepts for elements in its corresponding string pattern: mandatory / optional, sequential, ordinal, and reorderable. For example, as explained earlier, a regex can be used to specify which elements are mandatory or optional, and the order in which the elements may appear in the string pattern. Further, each regex can include POS information. For example, for each POS in its string pattern, the regex can include information indicating the corresponding POS label (e.g., noun, verb, preposition, etc.).

[0172] At 808, generalization is performed to obtain a generalized template from the initial template generated in 806. Generalization in 808 can be performed using any of the word-level generalization techniques and / or sentence-level generalization techniques described above. For example, in some embodiments, generalization in 808 may involve the following activities:

[0173] 1) Use the declared entities for each intent to identify entity phrases in order to generalize entities. For example, for the intent to track spending, the utterance “how much I did I spend on entertainment” can be generalized to “how much did I spend on EXPENSE_TYPE”.

[0174] 2) Based on the position of the entity phrase in the example discourse and the common part of speech of entity phrases associated with the same intent, the regex pattern of the entity phrase is obtained.

[0175] 3) Obtaining prepositional phrase patterns. For example, if one phrase has "on Expense_Type" and another phrase has "for Expense_Type", then the pattern "(on|for)Expense_Type((on...|for)expense_type)" can be obtained. Obtaining this pattern can involve applying heuristics to perform logical reasoning based on language understanding. The POS extracted in 804 aids in this reasoning.

[0176] 4) Break down the discourse into its components, including verb phrases, "Wh" phrases (e.g., phrases containing the words "what" or "which"), noun phrases, and prepositional phrases. This reveals the patterns of verb, wh, and noun phrases.

[0177] 5) Within a given intent, compare utterances associated with that intent to determine how subphrase patterns can be merged (e.g., send and pay can be merged into “send|pay”) or how subphrase patterns can be made optional (e.g., determine that the pattern “last month” for specifying a date range might be optional). Additionally, for example, synonyms can be determined based on words used in the same context and / or from word vector similarity.

[0178] As discussed earlier, some types of generalization are safer than others (e.g., stemming and entity generalization are generally the safest), and such generalization can be performed first as part of the processing in box 808. Also as discussed above, if a generalization is tested and found to be invalid, backtracking (undoing the generalization) can be performed to try a different generalization instead, and the generalization that was rejected / considered invalid can be remembered to avoid retrying it. A generalization can be considered invalid if the overall pattern it results in is: (i) not specific enough (e.g., "what do" is a vague pattern unlikely to be specific to a particular intent), or (ii) matches a pattern from another generalization derived from a different intent.

[0179] The processing in box 808 (e.g., steps 3 through 5) can be repeated to pair all combinations of the remaining known templates until no further generalization can be made. Additionally, user input (e.g., input from a skill robot developer) can be used to test the effectiveness of the generalization. For example, in step 5 of box 808, user input can be used instead of comparison utterances. Moreover, new training examples introduced via user input (e.g., additional example utterances provided by the user) can restart training, allowing the generation of additional templates and generalizations derived from these additional templates.

[0180] At 810, the initial template from 806 and the generalized template from 808 are stored in association with their corresponding intents. For example, the templates can be stored in the runtime engine (e.g., Figure 2 The routing subsystem 220 in the runtime engine can access the storage (e.g., stored in a database). The generalized template forms at least a portion of the rules to be applied by the classifier (e.g., classifier 224) of the runtime engine.

[0181] According to the above technology (e.g., based on about Figure 8 The generalized templates generated (through the processing discussed) offer several advantages. For example, generalization can be performed in a non-simplifying manner by making certain template elements optional rather than removing them entirely (e.g., element-based POS). Conversely, relying solely on LCS (simplification techniques) to generalize templates can lead to overgeneralization. Furthermore, new patterns can be learned from existing patterns to achieve further generalization (e.g., replacing one word with another word of similar meaning based on Verb / WordNet or word vectors), thereby improving the ability of the resulting templates to match a wider range of user discourse. The accuracy with which the resulting templates represent their corresponding intents can be improved by testing candidate templates (e.g., through repeated pairwise template comparisons across different intents in multiple rounds).

[0182] Example runtime process (template-based intent classification)

[0183] Figure 9 A simplified flowchart 900 according to some embodiments is illustrated, which depicts the process for classifying the intent of user utterances based on a generalized template. Figure 9 The described process can be implemented in software (e.g., code, instructions, programs) executed by one or more processing units (e.g., processors, cores) of a corresponding system, in hardware, or in a combination thereof. The software can be stored on a non-transitory storage medium (e.g., a memory device). Figure 9 The methods presented and described below are intended to be illustrative rather than restrictive. Although Figure 9 The individual processing steps, occurring in a specific sequence or order, are described, but this is not intended to be limiting. In some alternative embodiments, the steps may be performed in a different order, or some steps may be performed in parallel. In some embodiments, such as in Figure 2 In the embodiments depicted, this can be performed by a classifier such as classifier 224. Figure 9 The processing described in the document.

[0184] At position 902, speech is received from the user. For example, the speech received at position 902 could correspond to... Figure 2 The words in 202.

[0185] At 904, the utterance received in 902 is processed to extract part-of-speech tags. The processing in 904 can be performed, for example, by a language parser such as language parser 214.

[0186] At 906, one or more candidate templates are identified for use when matching against the utterance received at 902. In the context of intent classification, candidate templates may be a subset of all templates available to the classifier. Candidate templates include generalized templates and, in some instances, initial (non-generalized) templates. In some embodiments, the identification at 906 involves using an index to filter out a set of templates that may match. For example, the index may be a "bag of words" index of individual word scores. For each word in the index, the bag of words index may include an associated value indicating the probability that the word represents a certain robot intent. This allows for the rapid elimination of a subset of robot intents from consideration, as well as templates associated with those intents, based on the word scores of the utterance. Specifically, word scores may be used to identify intents most likely to be represented by the utterance, and then templates associated with these identified intents are selected as candidates for further evaluation.

[0187] At 908, the utterance received in 902 is matched against the template identified in 906. As indicated above, the template can be expressed in the form of a regex, which includes a string pattern and optional part-of-speech information. Therefore, the matching in 908 can involve calculating a score for each candidate template based on the degree of matching between the utterance in 902 and the string pattern and / or the part-of-speech information specified by the candidate template. For example, in some embodiments, the score is calculated such that: (i) if not all mandatory elements match, the score is 0, and (ii) to obtain a perfect score of 1.0, all words in the user utterance must match the candidate template, thus preserving the optional portion of the candidate template pattern. In other words, according to some embodiments, a perfect score can be obtained when the user utterance includes all mandatory elements of the template and all optional elements of the template. However, other scoring methods are also possible (e.g., calculating a score between 0 and 1.0 based on partial matching).

[0188] As part of the 908 matching process, part-of-speech information can be applied to verify that a matching element from a user utterance has the same part of speech as its corresponding template element. If a word in the user utterance is the same as another word in the template, but has a different part of speech associated with it (e.g., one is a noun and the other is an adjective), then the word in the user utterance can be considered a mismatch.

[0189] At point 910, candidate templates are ranked by score to identify the intent associated with the highest-scoring template as the robot intent corresponding to the user's utterance, i.e., the intent most representative of the user's utterance. In some embodiments, a threshold criterion may be applied such that the highest score must exceed a certain threshold of the second-highest score. If the highest score does not exceed the threshold of the second-highest score, the classifier may prompt the user for input to indicate which of the multiple intents associated with the multiple highest-scoring templates is the intent the user wishes to continue. For example, the classifier may present the user with a list of intents and ask the user to select from the list. Alternatively, in cases where a partial template match results in a highest score, or in cases where two high-scoring templates match, the classifier may ask questions about words unknown to the training model (not represented in any template) and prevent full or explicit matches. Questions may be asked to prompt the user to clarify the meaning of the word when it appears in the user's utterance.

[0190] At point 912, input is generated for the chatbot that has been configured with the robot intent identified in point 910. For example, the input generated in point 912 could correspond to... Figure 2The input in 912 is 235 and may include the entire user utterance. Alternatively, as previously described, the chatbot's input may include only a portion of the user utterance and / or extracted information. Examples of extracted information include POS tags and entity values ​​(e.g., the "checking" value for the Account_Type entity). In some embodiments, the input generated in 912 includes a parsed form of the user utterance, where the user utterance is broken down into its components. Further, in some embodiments, the input generated in 912 may include an indication of which template is considered a matching template (e.g., by virtue of having the highest score). This additional information can be used by the chatbot receiving the information to perform specific actions. For example, the chatbot may determine its response based on its dialogue configuration, depending on the matched template, such as starting the session with a different dialogue flow state depending on the matched template.

[0191] Figure 9 The classification-related processing described herein is not limited to identifying intents configured for the skill bot. System-level intents (e.g., Exit, Help, or other intents configured for the main bot / digital assistant) can also be identified. In some embodiments, user utterances related to system-level intents are prioritized, such that if the highest-scoring template is associated with a system-level intent and the user is currently in a conversation with the skill bot, the digital assistant interrupts the conversation to trigger a conversation between the user and the digital assistant. After the conversation between the user and the digital assistant concludes, the digital assistant can allow the user to switch back to the interrupted conversation.

[0192] Example implementation

[0193] Figure 10 A simplified diagram of a distributed system 1000 for implementing embodiments is depicted. In the illustrated embodiment, the distributed system 1000 includes one or more client computing devices 1002, 1004, 1006, and 1008 coupled to a server 1012 via one or more communication networks 1010. The client computing devices 1002, 1004, 1006, and 1008 may be configured to execute one or more applications.

[0194] In various embodiments, server 1012 may be adapted to run one or more services or software applications that enable the generation of templates for matching user utterances.

[0195] In some embodiments, server 1012 may also provide other services or software applications that may include non-virtual and virtual environments. In some embodiments, these services may be provided as web-based services or cloud services (such as under a Software as a Service (SaaS) model) to users of client computing devices 1002, 1004, 1006, and / or 1008. Users operating client computing devices 1002, 1004, 1006, and / or 1008 may then use one or more client applications to interact with server 1012 to utilize the services provided by these components.

[0196] exist Figure 10 In the depicted configuration, server 1012 may include one or more components 1018, 1020, and 1022 that implement the functions performed by server 1012. These components may include software components that can be executed by one or more processors, hardware components, or a combination thereof. It should be understood that various different system configurations, different from distributed system 1000, are possible. Therefore, Figure 10 The illustrated embodiment is an example of a distributed system for implementing the system of the embodiment and is not intended to be limiting.

[0197] Users can interact with the chatbot system using client computing devices 1002, 1004, 1006, and / or 1008. Users of client computing devices 1002, 1004, 1006, and / or 1008 can include end users who provide utterances during a session with the chatbot system. In accordance with the teachings of this disclosure, users of client computing devices 1002, 1004, 1006, and / or 1008 can also include users who provide input for generating templates as part of training an intent classifier. The client devices can provide an interface that enables users of the client devices to interact with them. The client devices can also output information to users via this interface. Although... Figure 10 It describes only four client computing devices, but can support any number of client computing devices.

[0198] Client devices can include various types of computing systems, such as portable handheld devices, general-purpose computers like personal computers and laptops, workstation computers, wearable devices, gaming systems, thin clients, various messaging devices, sensors, or other sensing devices. These computing devices can run various types and versions of software applications and operating systems (e.g., Microsoft). Apple Or a UNIX-like operating system, Linux, or a Linux-like operating system (such as Google Chrome). TMOS), including various mobile operating systems (e.g., Microsoft Windows). Windows Android TM , Palm Portable handheld devices can include cellular phones, smartphones (e.g., ), tablet computers (e.g., Wearable devices include Google's personal digital assistants (PDAs), personal digital assistants (PDAs), and more. Head-mounted displays and other devices. Gaming systems can include various handheld gaming devices, internet-enabled gaming devices (e.g., with or without internet access). Microsoft gesture input devices Game console, Sony System, by This includes various game systems and other devices. Client devices can run a variety of different applications, such as various Internet-related applications, communication applications (e.g., email applications, Short Message Service (SMS) applications), and can use various communication protocols.

[0199] (Multiple) networks 1010 can be any type of network familiar to those skilled in the art that supports data communication using any of the various available protocols, including but not limited to TCP / IP (Transmission Control Protocol / Internet Protocol), SNA (System Network Architecture), and IPX (Internet Packet Switching). Etc. By way of example only, (multiple) networks 1010 can be Local Area Networks (LANs), Ethernet-based networks, Token Ring, Wide Area Networks (WANs), the Internet, Virtual Networks, Virtual Private Networks (VPNs), Intranets, Extranets, Public Switched Telephone Networks (PSTNs), Infrared Networks, Wireless Networks (e.g., according to the IEEE 802.11 protocol suite), etc. (and / or any other wireless protocol operating on a network) and / or any combination of these networks and / or other networks.

[0200] Server 1012 may consist of: one or more general-purpose computers, dedicated server computers (including PC (personal computer) servers by way of example), Server 1012 may be a server, mid-range server, mainframe computer, rack server, etc., server cluster, server group, or any other suitable arrangement and / or combination. Server 1012 may include one or more virtual machines running a virtual operating system or other computing architectures involving virtualization (such as logical storage devices that can be virtualized to maintain one or more flexible pools of virtual storage devices for the server). In various embodiments, server 1012 may be adapted to run one or more services or software applications that provide the functionality described in the foregoing disclosure.

[0201] The computing system in server 1012 can run one or more operating systems, including any of the operating systems discussed above and any commercially available server operating system. Server 1012 can also run any of a variety of additional server applications and / or middleware applications, including HTTP (Hypertext Transfer Protocol) servers, FTP (File Transfer Protocol) servers, CGI (Common Gateway Interface) servers, etc. Servers, database servers, etc. Exemplary database servers include, but are not limited to, those that can be accessed from... Those database servers acquired through purchases by companies such as IBM.

[0202] In some implementations, server 1012 may include one or more applications to analyze and merge data feeds and / or event updates received from users of client computing devices 1002, 1004, 1006, and 1008. As an example, data feeds and / or event updates may include, but are not limited to, those provided in the original text. feed, The server 1012 may receive real-time updates from one or more third-party information sources and continuous data streams. These real-time updates may include real-time events related to sensor data applications, financial reporting machines, network performance measurement tools (e.g., network monitoring and traffic management applications), clickstream analysis tools, vehicle traffic monitoring, and the like. The server 1012 may also include one or more applications to display data feeds and / or real-time events via one or more display devices of client computing devices 1002, 1004, 1006, and 1008.

[0203] The distributed system 1000 may also include one or more data repositories 1014, 1016. In some embodiments, these data repositories may be used to store data and other information. For example, one or more of the data repositories 1014, 1016 may be used to store templates and / or other information used during intent classification. The data repositories 1014, 1016 may reside in various locations. For example, the data repository used by server 1012 may be local to server 1012 or may be located remotely to server 1012 and communicate with server 1012 via a network-based or dedicated connection. The data repositories 1014, 1016 may be of different types. In some embodiments, the data repository used by server 1012 may be a database, such as a relational database, like one created by Oracle. Databases from other vendors. One or more of these databases can be adapted to respond to SQL-formatted commands to store, update, and retrieve data from the database.

[0204] In some embodiments, one or more of the data stores 1014 and 1016 may also be used by an application to store application data. The data store used by the application may be of different types, such as, for example, a key-value store, an object store, or a general-purpose storage store supported by a file system.

[0205] In some embodiments, the classification-related functions described in this disclosure may be provided as a service via a cloud environment. Figure 11 This is a simplified block diagram of a cloud-based system environment according to certain embodiments, in which various category-related services can be provided as cloud services. Figure 11 In the depicted embodiments, cloud infrastructure system 1102 can provide one or more cloud services that can be requested by a user using one or more client computing devices 1104, 1106, and 1108. Cloud infrastructure system 1102 may include one or more computers and / or servers, which may include those described above with respect to server 1012. The computers in cloud infrastructure system 1102 may be organized as general-purpose computers, dedicated server computers, server clusters, server groups, or any other suitable arrangement and / or combination.

[0206] Multiple networks 1110 can facilitate data communication and exchange between clients 1104, 1106, and 1108 and cloud infrastructure system 1102. Multiple networks 1110 may include one or more networks. The networks may be of the same or different types. Multiple networks 1110 may support one or more communication protocols (including wired and / or wireless protocols) to facilitate communication.

[0207] Figure 11 The described embodiments are merely one example of a cloud infrastructure system and are not intended to be limiting. It should be understood that in some other embodiments, the cloud infrastructure system 1102 may have more... Figure 11 The depicted components may have more or fewer components, may combine two or more components, or may have different component configurations or arrangements. For example, although... Figure 11 Three client computing devices are depicted, but in alternative embodiments, any number of client computing devices can be supported.

[0208] The term cloud service is generally used to refer to services that become available to users on demand through a service provider's systems (e.g., cloud infrastructure system 1102) and via communication networks such as the Internet. Typically, in a public cloud environment, the servers and systems that make up the cloud service provider's systems are different from the customer's own on-premises servers and systems. The cloud service provider's systems are managed by the cloud service provider. Therefore, customers can utilize cloud services provided by the cloud service provider without having to purchase separate licenses, support, or hardware and software resources for the services. For example, the cloud service provider's systems can host applications, and users can subscribe to and use the applications on demand via the Internet without having to purchase the infrastructure resources to run the applications. Cloud services are designed to provide easy, scalable access to applications, resources, and services. Several providers offer cloud services. For example, Oracle, based in Redwood Shores, California. It provides several cloud services, such as middleware services, database services, Java cloud services, and other services.

[0209] In some embodiments, cloud infrastructure system 1102 may provide one or more cloud services using different models, such as Software as a Service (SaaS), Platform as a Service (PaaS), Infrastructure as a Service (IaaS), and other models (including hybrid service models). Cloud infrastructure system 1102 may include a set of applications, middleware, databases, and other resources that enable the provisioning of various cloud services.

[0210] The SaaS model enables applications or software to be delivered as a service to customers via communication networks such as the Internet, without requiring customers to purchase the underlying application's hardware or software. For example, the SaaS model can be used to provide customers with access to on-demand applications hosted by a cloud infrastructure system. Oracle Examples of the SaaS services offered include, but are not limited to, various services for human resources / capital management, customer relationship management (CRM), enterprise resource planning (ERP), supply chain management (SCM), enterprise performance management (EPM), analytics services, social applications, and others.

[0211] The IaaS model is typically used to provide customers with infrastructure resources (such as servers, storage, hardware, and networking resources) as cloud services to offer elastic computing and storage capabilities. (Produced by Oracle) It provides a variety of IaaS services.

[0212] The PaaS model is typically used to provide a platform and environment resources as a service that enable customers to develop, run, and manage applications and services, without requiring customers to purchase, build, or maintain such resources. (Provided by Oracle) Examples of PaaS services provided include, but are not limited to, Oracle Java Cloud Service (JCS), Oracle Database Cloud Service (DBCS), Data Management Cloud Service, various application development solutions services, and other services.

[0213] Cloud services are typically delivered in an on-demand, self-service, subscription-based, elastically scalable, reliable, highly available, and secure manner. For example, a customer can subscribe to one or more services provided by cloud infrastructure system 1102 via a subscription order. Cloud infrastructure system 1102 then performs processing to deliver the services requested in the customer's subscription order. For instance, a customer can place a subscription order to deploy a skills bot in a chatbot system. As part of configuring the skills bot, the customer can provide one or more sample utterances to generate templates associated with intents already configured for the skills bot. Cloud infrastructure system 1102 can be configured to provide one or more cloud services.

[0214] Cloud infrastructure system 1102 can provide cloud services through different deployment models. In a public cloud model, cloud infrastructure system 1102 may be owned by a third-party cloud service provider, and cloud services are provided to any general public customer, which may be an individual or a business. In some other embodiments, under a private cloud model, cloud infrastructure system 1102 may operate within an organization (e.g., within a business organization), and services are provided to customers within that organization. For example, customers may be various departments within the organization, such as human resources or payroll departments, or even individuals within the organization. In some other embodiments, under a community cloud model, cloud infrastructure system 1102 and the services provided may be shared by several organizations in the relevant community. Various other models, such as hybrids of the models mentioned above, may also be used.

[0215] Client computing devices 1104, 1106, and 1108 can be of different types (e.g., Figure 10 The depicted devices 1002, 1004, 1006, and 1008 may also be capable of operating one or more client applications. Users can use the client devices to interact with the cloud infrastructure system 1102, such as requesting services provided by the cloud infrastructure system 1102. For example, a user can use the client device to request the classification-related services described in this disclosure.

[0216] In some embodiments, the processing performed by the cloud infrastructure system 1102 for providing classification-related services may involve big data analytics. This analytics may involve using, analyzing, and manipulating large datasets to detect and visualize various trends, behaviors, relationships, etc., within the data. This analytics may be performed by one or more processors, potentially processing the data in parallel, performing simulations using the data, etc. The data used for this analytics may include structured data (e.g., data stored in a database or structured according to a structured model) and / or unstructured data (e.g., data blocks (binary large objects)).

[0217] like Figure 11 As depicted in the embodiments, the cloud infrastructure system 1102 may include infrastructure resources 1130 used to facilitate the provision of various cloud services offered by the cloud infrastructure system 1102. Infrastructure resources 1130 may include, for example, processing resources, storage or memory resources, networking resources, etc.

[0218] In some embodiments, to facilitate efficient provisioning of these resources to support various cloud services provided by cloud infrastructure system 1102 to different customers, resources may be bundled into resource groups or resource modules (also referred to as "pods"). Each resource module or pod may include a pre-integrated and optimized combination of one or more types of resources. In some embodiments, different pods may be pre-provisioned for different types of cloud services. For example, a first pod may be provisioned for a database service, and a second pod may be provisioned for a Java service (the second pod may include a different combination of resources than the pods in the first pod), etc. For some services, resources allocated for provisioning the service may be shared between services.

[0219] The cloud infrastructure system 1102 itself can internally use services 1132 shared by different components of the cloud infrastructure system 1102 and that facilitate the provisioning of services by the cloud infrastructure system 1102. These internal shared services may include, but are not limited to, security and identity services, integration services, enterprise repository services, enterprise manager services, virus scanning and whitelisting services, high availability, backup and recovery services, services for enabling cloud support, email services, notification services, file transfer services, etc.

[0220] Cloud infrastructure system 1102 may include multiple subsystems. These subsystems may be implemented in software or hardware, or a combination thereof. Figure 11 The described subsystem may include a user interface subsystem 1112 that enables users or customers of cloud infrastructure system 1102 to interact with cloud infrastructure system 1102. User interface subsystem 1112 may include various interfaces such as a web interface 1114, an online store interface 1116 (where advertising and customers can purchase cloud services provided by cloud infrastructure system 1102), and other interfaces 1118. For example, a customer may use a client device to request (service request 1134) one or more services provided by cloud infrastructure system 1102 using one or more of interfaces 1114, 1116, and 1118. For example, a customer may access an online store, browse cloud services provided by cloud infrastructure system 1102, and place a subscription order for one or more services provided by cloud infrastructure system 1102 that the customer wishes to subscribe to. A service request may include information identifying the customer and the one or more services the customer wishes to subscribe to. For example, a customer may place a subscription order for category-related services provided by cloud infrastructure system 1102. As part of the order, the customer may provide information identifying a specific chatbot and / or specific intents configured for the chatbot. Customers can further provide example utterances for training the classifier to infer that user utterances correspond to a specific chatbot and / or a specific intent.

[0221] In some embodiments (such as) Figure 11 In the depicted embodiment, cloud infrastructure system 1102 may include an order management subsystem (OMS) 1120 configured to process new orders. As part of this process, OMS 1120 may be configured to: create an account for a customer (if not already created); receive billing and / or billing information from the customer to be used to bill the customer for the requested services; verify customer information; place an order for the customer after verification; and schedule various workflows to prepare the order for delivery.

[0222] Once correctly verified, OMS 1120 can invoke the Order Provisioning Subsystem (OPS) 1124, configured as order provisioning resources (including processing resources, storage resources, and networking resources). Provisioning may include allocating resources for an order and configuring resources to facilitate the service requested by the customer's order. The manner in which resources are provisioned to an order and the type of resources provided may depend on the type of cloud service the customer has subscribed to. For example, according to a workflow, OPS 1124 may be configured to determine the specific cloud service being requested and identify the number of groups that may have been pre-configured for that specific cloud service. The number of groups allocated to an order may depend on the size / volume / tier / scope of the requested service. For example, the number of groups to be allocated may be determined based on the number of users the service is to support, the duration of the requested service, etc. The allocated groups can then be customized for a specific requesting customer to provide the requested service.

[0223] Cloud infrastructure system 1102 may send a response or notification 1144 to the requesting customer to indicate when the requested service is now ready for use. In some instances, information (e.g., links) enabling the customer to begin using and taking advantage of the benefits of the requested service may be sent to the customer. In some embodiments, for customers requesting category-related services, the response may include a final set of templates (e.g., generalized templates) generated from a set of example utterances.

[0224] Cloud infrastructure system 1102 can provide services to multiple customers. For each customer, cloud infrastructure system 1102 is responsible for managing information related to one or more subscription orders received from the customer, maintaining customer data related to the orders, and providing the requested services to the customer. Cloud infrastructure system 1102 can also collect usage statistics about customers' use of subscribed services. For example, it can collect statistics such as storage usage, data transfer volume, number of users, system uptime, and system downtime. This usage information can be used to issue bills to customers. Billing can be done, for example, on a monthly basis.

[0225] Cloud infrastructure system 1102 can provide services to multiple customers in parallel. Cloud infrastructure system 1102 can store information about these customers (potentially including proprietary information). In some embodiments, cloud infrastructure system 1102 includes an Identity Management Subsystem (IMS) 1128 configured to manage customer information and provide separation of the managed information such that information associated with one customer cannot be accessed by another customer. IMS 1128 can be configured to provide various security-related services, such as identity services, information access management, authentication and authorization services, services for managing customer identities and roles, and related functions.

[0226] Figure 12 An exemplary computer system 1200 is illustrated that can be used to implement certain embodiments. For example, in some embodiments, computer system 1200 can be used to implement the main robot system 200 described above, the skill robot system 300, and any of various server and computer systems. Figure 12 As shown, the computer system 1200 includes various subsystems, including a processing subsystem 1204 that communicates with multiple other subsystems via a bus subsystem 1202. These other subsystems may include a processing acceleration unit 1206, an I / O subsystem 1208, a storage subsystem 1218, and a communication subsystem 1224. The storage subsystem 1218 may include non-transitory computer-readable storage media, including storage medium 1222 and system memory 1210.

[0227] Bus subsystem 1202 provides mechanisms for enabling the various components and subsystems of computer system 1200 to communicate with each other as intended. While bus subsystem 1202 is schematically shown as a single bus, alternative embodiments of the bus subsystem may utilize multiple buses. Bus subsystem 1202 may be any of several types of bus structures, including a memory bus or memory controller, a peripheral bus, a local bus using any of various bus architectures, etc. For example, such architectures may include an Industry Standard Architecture (ISA) bus, a Micro Channel Architecture (MCA) bus, an Enhanced ISA (EISA) bus, a Video Electronics Standards Association (VESA) local bus, and a Peripheral Component Interconnect (PCI) bus (which may be implemented as a mezzanine bus manufactured to the IEEE P1386.1 standard), etc.

[0228] Processing subsystem 1204 controls the operation of computer system 1200 and may include one or more processors, application-specific integrated circuits (ASICs), or field-programmable gate arrays (FPGAs). Processors may include single-core or multi-core processors. The processing resources of computer system 1200 may be organized into one or more processing units 1232, 1234, etc. A processing unit may include one or more processors, one or more cores from the same or different processors, a combination of cores and processors, or other combinations of cores and processors. In some embodiments, processing subsystem 1204 may include one or more dedicated coprocessors such as graphics processors or digital signal processors (DSPs). In some embodiments, some or all of the processing units of processing subsystem 1204 may be implemented using custom circuitry such as ASICs or FPGAs.

[0229] In some embodiments, the processing units in the processing subsystem 1204 can execute instructions stored in system memory 1210 or on computer-readable storage medium 1222. In various embodiments, the processing units can execute various program or code instructions and can maintain multiple concurrently executing programs or processes. At any given time, some or all of the program code to be executed can reside in system memory 1210 and / or on computer-readable storage medium 1222 (potentially including residing on one or more storage devices). With appropriate programming, the processing subsystem 1204 can provide the various functions described above. In an instance where the computer system 1200 is executing one or more virtual machines, one or more processing units can be assigned to each virtual machine.

[0230] In some embodiments, a processing acceleration unit 1206 may optionally be provided for performing custom processing or for offloading some of the processing performed by the processing subsystem 1204, thereby accelerating the overall processing performed by the computer system 1200.

[0231] I / O subsystem 1208 may include devices and mechanisms for inputting information to and / or outputting information from or via computer system 1200. Generally, the term input device is intended to include all possible types of devices and mechanisms for inputting information to computer system 1200. User interface input devices may include, for example, keyboards, pointing devices such as mice or trackballs, touchpads or touchscreens integrated into displays, scroll wheels, click wheels, dial pads, buttons, switches, keypads, audio input devices with voice command recognition systems, microphones, and other types of input devices. User interface input devices may also include motion sensing and / or gesture recognition devices, such as Microsoft's [unclear - possibly a specific technology or feature], which enables users to control and interact with input devices. Motion sensor, Microsoft The 360 ​​game controller provides an interface for receiving input using gestures and spoken commands. The user interface input device may also include eye gesture recognition devices, such as detecting eye movements from the user (e.g., "blinking" when taking a photo and / or making menu selections) and translating the eye gestures into the input device (such as Google). Google input Blink detector. Additionally, the user interface input device may include features that enable the user to interact with a voice recognition system (e.g., via voice commands). Voice recognition sensing devices for navigators.

[0232] Other examples of user interface input devices include, but are not limited to, 3D mice, joysticks or pointing sticks, game controllers and graphics tablets, and audio / visual devices such as speakers, digital cameras, digital camcorders, portable media players, webcams, image scanners, fingerprint scanners, barcode readers, 3D scanners, 3D printers, laser rangefinders, and eye-tracking devices. Additionally, user interface input devices may include, for example, medical imaging input devices such as computed tomography (CT), magnetic resonance imaging (MRI), positron emission tomography (PET), and medical ultrasound examination equipment. User interface input devices may also include, for example, audio input devices such as MIDI keyboards, digital musical instruments, etc.

[0233] Generally, the term "output device" is intended to encompass all possible types of devices and mechanisms for outputting information from computer system 1200 to a user or other computer. User interface output devices may include display subsystems, indicator lights, or non-visual displays such as audio output devices. Display subsystems may be cathode ray tube (CRT), flat panel devices (such as those using liquid crystal displays (LCDs) or plasma displays), projection devices, touchscreens, etc. For example, user interface output devices may include, but are not limited to, various display devices that visually convey text, graphics, and audio / video information, such as monitors, printers, speakers, headsets, car navigation systems, plotters, voice output devices, and modems.

[0234] Storage subsystem 1218 provides a repository or data storage device for storing information and data used by computer system 1200. Storage subsystem 1218 provides a tangible, non-transitory, computer-readable storage medium for storing basic programming and data constructs that provide the functionality of some embodiments. Storage subsystem 1218 may store software (e.g., programs, code modules, instructions) that provides the functionality described above when executed by processing subsystem 1204. The software may be executed by one or more processing units of processing subsystem 1204. Storage subsystem 1218 may also provide a repository for storing data used, in accordance with the teachings of this disclosure.

[0235] The storage subsystem 1218 may include one or more non-transitory memory devices, which may include volatile memory devices and non-volatile memory devices. For example... Figure 12 As shown, the storage subsystem 1218 includes system memory 1210 and computer-readable storage medium 1222. System memory 1210 may include multiple memories, including volatile main random access memory (RAM) for storing instructions and data during program execution and non-volatile read-only memory (ROM) or flash memory where fixed instructions are stored. In some embodiments, a basic input / output system (BIOS) containing basic routines, such as those that help transfer information between elements within computer system 1200 during startup, may typically be stored in ROM. RAM may contain data and / or program modules currently operated and executed by processing subsystem 1204. In some embodiments, system memory 1210 may include various different types of memory, such as static random access memory (SRAM), dynamic random access memory (DRAM), etc.

[0236] By using examples rather than restrictions, such as Figure 12 As depicted, system memory 1210 can load an executing application 1212 (which may include various applications such as web browsers, middleware applications, relational database management systems (RDBMS), etc.), program data 1214, and operating system 1216. By way of example, operating system 1216 may include various versions of Microsoft... Apple and / or Linux operating system, and various commercially available... Or a UNIX-like operating system (including but not limited to various GNU / Linux operating systems, Google...) OS, etc.) and / or such as iOS, Telephone, OS OS OS operating system and other mobile operating systems.

[0237] Computer-readable storage medium 1222 can store programming and data structures that provide functionality for some embodiments. Computer-readable storage medium 1222 can provide storage for computer-readable instructions, data structures, program modules, and other data for computer system 1200. Software (programs, code modules, instructions) that provides the functionality described above, when executed by processing subsystem 1204, can be stored in storage subsystem 1218. By way of example, computer-readable storage medium 1222 may include, for example, hard disk drives, disk drives, optical disc drives (such as CD ROMs, DVDs, Blu-ray discs, etc.). Non-volatile memory such as disks or other optical media. Computer-readable storage medium 1222 may include, but is not limited to, [other types of storage media]. Drives, flash memory cards, Universal Serial Bus (USB) flash memory drives, Secure Digital (SD) cards, DVD discs, digital videotapes, etc. Computer-readable storage media 1222 may also include, for example, flash memory-based solid-state drives (SSDs), enterprise-class flash memory drives, SSDs based on non-volatile memory such as solid-state ROM, SSDs based on volatile memory such as solid-state RAM, dynamic RAM, static RAM, etc., DRAM-based SSDs, magnetoresistive RAM (MRAM) SSDs, and hybrid SSDs using a combination of DRAM and flash memory-based SSDs.

[0238] In some embodiments, the storage subsystem 1218 may further include a computer-readable storage medium reader 1220 that can be further connected to the computer-readable storage medium 1222. The reader 1220 may receive data from a storage device such as a disk, flash memory drive, etc., and is configured to read data from said storage device.

[0239] In some embodiments, computer system 1200 may support virtualization technologies, including but not limited to virtualization of processing and memory resources. For example, computer system 1200 may provide support for executing one or more virtual machines. In some embodiments, computer system 1200 may execute programs such as hypervisors that facilitate the configuration and management of virtual machines. Each virtual machine may be allocated memory, computing (e.g., processor, core), I / O, and networking resources. Each virtual machine typically runs independently of other virtual machines. Virtual machines typically run their own operating system, which may be the same as or different from the operating systems executed by other virtual machines executed by computer system 1200. Therefore, multiple operating systems may potentially be run simultaneously by computer system 1200.

[0240] The communication subsystem 1224 provides an interface to other computer systems and networks. The communication subsystem 1224 serves as an interface for receiving data from other systems and transmitting data from computer system 1200 to other systems. For example, the communication subsystem 1224 can enable computer system 1200 to establish a communication channel via the Internet to one or more client devices for receiving and sending information to client devices. For example, the communication subsystem can be used to transmit messages between a user and a chatbot with which the user is having a conversation.

[0241] The communication subsystem 1224 may support both wired and / or wireless communication protocols. For example, in some embodiments, the communication subsystem 1224 may include radio frequency (RF) transceiver components (e.g., advanced data network technologies such as cellular phone technology, 3G, 4G, or EDGE (Global Evolution Enhanced Data Rate), WiFi (IEEE 802.XX Home Standard, or other mobile communication technologies, or any combination thereof), GPS receiver components, and / or other components for accessing wireless voice and / or data networks. In some embodiments, in addition to or as an alternative to a wireless interface, the communication subsystem 1224 may provide wired network connectivity (e.g., Ethernet).

[0242] The communication subsystem 1224 can receive and transmit data in various forms. For example, in some embodiments, the communication subsystem 1224 can also receive input communications in the form of structured and / or unstructured data feeds 1226, event streams 1228, event updates 1230, etc., among others. For example, the communication subsystem 1224 can be configured to receive (or send) data feeds 1226 in real time from users of social media networks and / or other communication services, such as... feed, Updates, web feeds (such as rich site summary (RSS) feeds) and / or real-time updates from one or more third-party information sources.

[0243] In some embodiments, the communication subsystem 1224 may be configured to receive data in the form of a continuous data stream, which may include an event stream 1228 and / or event updates 1230 of real-time events (which may be inherently continuous or unbounded and may not have an explicit end). Examples of applications that generate continuous data may include, for example, sensor data applications, financial reporting machines, network performance measurement tools (e.g., network monitoring and traffic management applications), clickstream analysis tools, vehicle traffic monitoring, etc.

[0244] The communication subsystem 1224 can also be configured to transmit data from computer system 1200 to other computer systems or networks. Data can be transmitted in various forms, such as structured and / or unstructured data feeds 1226, event streams 1228, event updates 1230, etc., to one or more databases that can communicate with one or more streaming data source computers coupled to computer system 1200.

[0245] Computer system 1200 can be of a variety of types, including handheld portable devices (e.g., Cellular phone Computing tablets, PDAs), wearable devices (e.g., Google...) Head-mounted displays, personal computers, workstations, mainframes, self-service kiosks, server racks, or any other data processing systems. Due to the constantly evolving nature of computers and networks, [the following is relevant:] Figure 12 The description of the computer system 1200 is intended only as a concrete example. It has a higher... Figure 12 Many other configurations with more or fewer components are possible for the system depicted. Based on the disclosure and teachings provided herein, those skilled in the art will understand other ways and / or methods for implementing the various embodiments.

[0246] While specific embodiments have been described, various modifications, alterations, alternative constructions, and equivalents are possible. The embodiments are not limited to operation in a particular data processing environment but are free to operate in multiple data processing environments. Furthermore, although certain embodiments have been described using specific series of transactions and steps, it will be apparent to those skilled in the art that this is not intended to be restrictive. While some flowcharts describe operations as sequential processes, many operations may be performed in parallel or simultaneously. Additionally, the order of operations can be rearranged. Processes may have additional steps not included in the figures. Various features and aspects of the embodiments described above can be used individually or in combination.

[0247] Furthermore, while certain embodiments have been described using specific combinations of hardware and software, it should be recognized that other combinations of hardware and software are also possible. Some embodiments may be implemented solely in hardware, solely in software, or using a combination thereof. The various processes described herein may be implemented on the same or different processors in any combination.

[0248] When a device, system, component, or module is described as being configured to perform certain operations or functions, this configuration can be accomplished, for example, by designing electronic circuitry to perform operations, by programming programmable electronic circuitry (such as a microprocessor) to perform operations (such as by executing computer instructions or code), or by a processor or core programmed to execute code or instructions stored on a non-transitory memory medium, or any combination thereof. Processes can communicate using a variety of techniques, including but not limited to conventional techniques for inter-process communication, and different pairs of processes may use different techniques, or the same pair of processes may use different techniques at different times.

[0249] Specific details are set forth in this disclosure to provide a thorough understanding of the embodiments. However, embodiments may be practiced without these specific details. For example, well-known circuits, processes, algorithms, structures, and techniques have been shown without unnecessary detail to avoid obscuring the embodiments. This description provides only exemplary embodiments and is not intended to limit the scope, applicability, or configuration of other embodiments. Rather, the foregoing description of the embodiments will provide those skilled in the art with an enabling description for implementing the various embodiments. Various changes can be made to the function and arrangement of the elements.

[0250] Therefore, the specification and drawings should be viewed in an illustrative rather than restrictive sense. However, it will be apparent that additions, omissions, deletions, and other modifications and changes may be made without departing from the broader spirit and scope set forth in the claims. Thus, while specific embodiments have been described, these examples are not intended to be limiting. Various modifications and equivalents are within the scope of the following claims.

Claims

1. A computer-implemented method, comprising: A computer-based chatbot system receives input from users; The chatbot system determines, based on a template set, that the input utterance represents a task configured to be performed by a first chatbot in the chatbot system, wherein the determination includes: The input utterance is compared with various templates in the template set, wherein the first template in the template set is a regular expression including a string pattern corresponding to a generalized form of one or more example utterances provided to the first chatbot, and the one or more example utterances represent a task that the first chatbot is configured to perform, and wherein the first template is generated by: identifying one or more example utterances representing the task that the first chatbot is configured to perform; generating an initial template as a regular expression including a string pattern for each of the one or more example utterances representing the task; repeatedly performing at least one of sentence-level generalization or word-level generalization on the generated initial template until no further simplification of the initial template can be performed to generate the first template; and confirming that the first template does not conflict with one or more templates associated with other chatbots in the chatbot system. For each template in the template set, a corresponding score is calculated based on the degree of matching between the input utterance and the template; and Based on the calculated score, it is determined that the first template in the template set best matches the input utterance; and In response to determining that the input utterance represents a task configured to be performed by the first chatbot, the chatbot system transmits at least a portion of the input utterance as input to the first chatbot.

2. The method as described in claim 1, wherein, The first template includes language part-of-speech information, which indicates the part of speech of different elements of the string pattern.

3. The method as described in claim 2, wherein, Determining that the input utterance represents a task configured to be performed by the first chatbot further includes: The elements of the input utterance are determined to have the same part of speech as the corresponding elements of the string pattern.

4. The method of claim 1, wherein, Determining that the input utterance represents a task configured to be performed by the first chatbot further includes: Determine the score of each word in the input utterance, wherein the score of a word indicates the probability that the word represents a specific task that the chatbot is configured to perform; and Based on the scores of each word in the input utterance, the template set is identified as a subset of a larger template set.

5. The method of claim 1, wherein, At least one instance of sentence-level generalization or word-level generalization involves determining a more general pattern that applies to two or more string patterns.

6. The method of claim 5, wherein, The more general pattern grouped synonyms together as alternative template elements.

7. The method of claim 1, wherein, At least one instance of sentence-level generalization or word-level generalization involves: The specific element determining the string pattern is one of several possible values ​​for the named entity already configured for the first chatbot; and Replace the specific element with the named entity.

8. The method of claim 1, wherein, At least one instance of sentence-level generalization or word-level generalization involves: An element of the first string pattern is identified as an optional element based on the following: (i) the element does not appear in the longest common subsequence between the first string pattern and the second string pattern, or (ii) the part-of-speech tag of the element.

9. The method of claim 1, wherein, Generating the first template further includes: The first string pattern obtained during a specific instance of sentence-level or word-level generalization is invalid because it is identical to a second string pattern representing a task different from the one represented by the first string pattern.

10. A computer system, comprising: One or more processors; as well as A memory coupled to the one or more processors, the memory storing instructions that, when executed by the one or more processors, cause the one or more processors to perform processing including the following operations: Receive input from the user; Based on a template set, the input utterance is determined to represent a task configured to be performed by the first chatbot, wherein the determination includes: The input utterance is compared with various templates in the template set, wherein the first template in the template set is a regular expression including a string pattern corresponding to a generalized form of one or more example utterances provided to the first chatbot, and the one or more example utterances represent a task that the first chatbot is configured to perform, and wherein the first template is generated by: identifying one or more example utterances representing the task that the first chatbot is configured to perform; generating an initial template as a regular expression including a string pattern for each of the one or more example utterances representing the task; repeatedly performing at least one of sentence-level generalization or word-level generalization on the generated initial template until no further simplification of the initial template can be performed to generate the first template; and confirming that the first template does not conflict with one or more templates associated with other chatbots in the chatbot system. For each template in the template set, a corresponding score is calculated based on the degree of matching between the input utterance and the template; and Based on the calculated score, it is determined that the first template in the template set best matches the input utterance; and In response to determining that the input utterance represents a task that the first chatbot is configured to perform, at least a portion of the input utterance is transmitted as input to the first chatbot.

11. The computer system of claim 10, wherein, The first template includes language part-of-speech information, which indicates the part of speech of different elements of the string pattern.

12. The computer system of claim 11, wherein, Determining that the input utterance represents a task configured to be performed by the first chatbot further includes: The elements of the input utterance are determined to have the same part of speech as the corresponding elements of the string pattern.

13. The computer system of claim 10, wherein, Determining that the input utterance represents a task configured to be performed by the first chatbot further includes: Determine the score of each word in the input utterance, wherein the score of a word indicates the probability that the word represents a specific task that the chatbot is configured to perform; and Based on the scores of each word in the input utterance, the template set is identified as a subset of a larger template set.

14. The computer system of claim 10, wherein, At least one instance of sentence-level generalization or word-level generalization involves determining a more general pattern that applies to two or more string patterns.

15. The computer system of claim 10, wherein, Generating the first template further includes: The first string pattern obtained during a specific instance of sentence-level or word-level generalization is invalid because it is identical to a second string pattern representing a task different from the one represented by the first string pattern.

16. A non-transitory computer-readable medium storing instructions that, when executed by one or more processors within a computer system, cause the one or more processors to perform processing comprising the following operations: Receive input from the user; Based on the template set, the input utterance is determined to represent the task configured to be performed by the first chatbot, wherein... The determination includes: The input utterance is compared with various templates in the template set, wherein the first template in the template set is a regular expression including a string pattern corresponding to a generalized form of one or more example utterances provided to the first chatbot, and the one or more example utterances represent a task that the first chatbot is configured to perform, and wherein the first template is generated by: identifying one or more example utterances representing the task that the first chatbot is configured to perform; generating an initial template as a regular expression including a string pattern for each of the one or more example utterances representing the task; repeatedly performing at least one of sentence-level generalization or word-level generalization on the generated initial template until no further simplification of the initial template can be performed to generate the first template; and confirming that the first template does not conflict with one or more templates associated with other chatbots in the chatbot system. For each template in the template set, a corresponding score is calculated based on the degree of matching between the input utterance and the template; and Based on the calculated score, it is determined that the first template in the template set best matches the input utterance; and In response to determining that the input utterance represents a task that the first chatbot is configured to perform, at least a portion of the input utterance is transmitted as input to the first chatbot.

17. The non-transitory computer-readable medium of claim 16, wherein, The first template includes language part-of-speech information, which indicates the part of speech of different elements of the string pattern.

18. A chatbot system comprising means for performing the steps of the method as claimed in any one of claims 1-9.

Images