Entity-Level Data Augmentation in Chatbots for Robust Inherent Expression Recognition

Abstract

In one particular aspect, a data augmentation technique for training a chatbot system in natural language processing is provided, the method comprising: generating a list of values ​​to cover an entity; selecting an utterance having a context of the entity from a dataset; and converting the utterance into a plurality of templates, each template of the plurality of templates including a slot that maps to a list of values ​​of the entity; and generating an artificial utterance based on the selected template and the selected value, wherein generating the artificial utterance comprises inserting the selected value into the slot of the selected template that maps to the list of values ​​of the entity.

Description

Technical Field

[0001] Priority Claim This application is a non-provisional application of U.S. Provisional Application No. 63 / 038,337, filed on Jun. 12, 2020, and claims the benefit thereof, and the entire disclosure thereof is incorporated herein by reference.

[0002] Field of the Invention The present disclosure generally relates to chatbot systems, and more particularly, to data augmentation techniques for training chatbot systems in natural language processing.

Background Art

[0003] Background Many users around the world are using instant messaging or chat platforms to get instant responses. Organizations often use these instant messaging or chat platforms to communicate directly with customers (or end users). However, it can be very costly for an organization to hire service staff to communicate directly with customers or end users. In particular, chatbots or bots have begun to be developed to simulate conversations with end users over the Internet. End users can communicate with the bot through the messaging apps they have already installed and are using. Generally, intelligent bots equipped with artificial intelligence (AI) can enable more intelligent and contextual communication in live conversations, so that a more natural conversation can be possible between the bot and the end user, and the conversation experience may be improved. Instead of learning a fixed set of keywords or commands that the end user knows how the bot will respond, intelligent bots can understand the end user's intention based on the end user's utterance in natural language and respond accordingly.

[0004] However, building chatbots is challenging because these automated solutions require specific knowledge in a particular domain and the application of specific technologies that may only exist within the scope of the expertise of specialized developers. As part of building such a chatbot, developers can first understand the needs of the company and end users. Developers can then perform analyses and decisions related to, for example, selecting the dataset to be used for analysis, preparing the input dataset for analysis (e.g., data cleansing, pre-analysis data extraction, formalization, and / or transformation, performing data feature engineering, etc.), identifying the appropriate machine learning (ML) techniques or models for performing the analysis, and improving the techniques or models to improve results / outcomes based on feedback. The task of identifying the appropriate model may, in some cases, involve developing multiple models in parallel and then identifying one (or more) specific models to use after iterative testing and experimentation using these models. Furthermore, supervised learning-based solutions typically involve a training phase, a subsequent application (i.e., inference) phase, and an iterative loop between the training and application phases. Developers may be responsible for ensuring these phases are carefully implemented and monitored to achieve the optimal solution. For example, to train an ML technique or model, accurate training data is needed so that the algorithm can understand and learn specific patterns or features that the ML technique or model will use to predict desired outcomes (e.g., in the case of a chatbot, not only raw language processing but also intent extraction and careful parsing). To ensure that the ML technique or model learns these patterns and features properly, developers may be responsible for selecting, enhancing, and optimizing the training dataset for the ML technique or model. [Overview of the project] [Means for solving the problem]

[0005] concise summary This disclosure relates to data augmentation techniques for training chatbot systems in natural language processing.

[0006] Methods are provided in various embodiments, each comprising: a data processing system generating a list of values ​​to cover entities; a data processing system selecting utterances from a dataset that have the context of entities; and a data processing system converting the utterances into a plurality of templates, each of which includes slots of entities that map to a list of values. The method further comprises: a data processing system selecting a template from the plurality of templates; a data processing system selecting values ​​from a list of values ​​based on mappings between slots in the selected template and a list of values ​​for entities; and a data processing system generating an artificial utterance using the selected template and the selected values, wherein generating the artificial utterance includes inserting the selected values ​​into slots in the selected template.

[0007] In some embodiments, the method further comprises the data processing system training a model using a training dataset before selecting utterances from a dataset, the training dataset being a dataset or a subset thereof, and the method further comprises the data processing system testing the model using a test dataset or validation dataset, the data processing system evaluating the performance of the model based on the testing of the model, and the data processing system identifying entities based on the evaluation.

[0008] In some embodiments, the method further comprises: a data processing system augmenting a dataset or training dataset with artificial utterances to generate an augmented training dataset; the data processing system training a model using the augmented training dataset; the data processing system retesting the model using a test dataset or validation dataset, or other test datasets or other validation datasets; the data processing system evaluating the performance of the model based on the retesting of the model; the data processing system determining that the performance of the model meets predetermined criteria; and deploying the model as part of a chatbot system in response to the model meeting the predetermined criteria.

[0009] In some embodiments, the method further comprises: a data processing system determining that a slot in a selected template includes a first element that maps to a list of values ​​and a second element that maps to a function; the data processing system processing the function to generate a function value based on the mapping between the slot in the selected template and the function; and the data processing system combining the selected values ​​and the function value to obtain a slot value, wherein generating artificial utterance includes inserting the slot value into a slot in the selected template.

[0010] In some embodiments, the method further comprises the data processing system determining that a selected template maps to a nested set of slot mappings, which include a slot and other slots; the data processing system determining that the other slots in the nested set of slot mappings map to a list of other values; and the data processing system selecting other values ​​from the list of other values ​​based on the mapping between the other slots in the selected template and the list of other values, and generating artificial utterances includes inserting the selected values ​​into the slots of the selected template and inserting the selected other values ​​into the other slots of the selected template.

[0011] In some embodiments, the method further comprises a data processing system augmenting a dataset or training dataset with artificial utterances to generate an augmented training dataset for training a model to recognize and classify a plurality of entities, including entities.

[0012] In some embodiments, templates are selected based on heuristics that include the following conditions: (i) whether to choose a template with multiple entity classes or a template with a single entity class; (ii) a trade-off between template diversity and template quality; (iii) a context-dependent classification of templates; (iv) whether the model requires a minimum number of occurrences per template / specific context that the user wants to teach the model; or (v) any combination of these.

[0013] The system is provided in various embodiments, each comprising one or more data processors and a non-temporary computer-readable storage medium containing instructions that, when executed on one or more data processors, cause one or more data processors to perform some or all of the methods disclosed herein.

[0014] Computer program products are provided in various embodiments, each computer program product being tangibly embodied in a non-temporary machine-readable storage medium and including instructions configured to cause one or more data processors to perform some or all of the methods disclosed herein.

[0015] The technologies described above and below can be implemented in many forms and in many contexts. Several examples of implementations and contexts are provided with reference to the following diagram, as will be explained in more detail below. However, the following implementations and contexts represent only a fraction of the many. [Brief explanation of the drawing]

[0016] [Figure 1] This is a simplified block diagram showing a distributed environment incorporating an exemplary embodiment. [Figure 2] This is a simplified block diagram showing a computing system that implements a master bot according to a specific embodiment. [Figure 3] This is a simplified block diagram showing a computing system that implements a skill bot according to a specific embodiment. [Figure 4] This is a simplified block diagram illustrating a chatbot training and deployment system according to various embodiments. [Figure 5A] This figure shows the process flow for training a predictive model using augmented training data according to various embodiments. [Figure 5B] This figure shows a process flow for generating augmented training data according to various embodiments. [Figure 5C] This figure shows a process flow for generating artificial speech from a template according to various embodiments. [Figure 6] This is a simplified diagram showing a distributed system for implementing various embodiments. [Figure 7] This is a simplified block diagram showing one or more components of a system environment in which services provided by one or more components of an embodiment of the system may be provided as cloud services, according to various embodiments. [Figure 8] This figure shows examples of computer systems that may be used to implement various embodiments. [Modes for carrying out the invention]

[0017] Detailed explanation In the following description, specific details are set forth for the purpose of providing a thorough understanding of particular embodiments. However, it will be apparent that various embodiments may be practiced without these specific details. The figures and description are not intended to be restrictive. The term "exemplary" as used herein means "serving as an example, instance, or illustration." Any embodiment or design described herein as "exemplary" is not necessarily to be construed as preferred or advantageous over other embodiments or designs.

[0018] Introduction A digital assistant is an AI-driven interface that helps users accomplish various tasks in natural language conversations. For each digital assistant, a customer can combine one or more skills. Skills (also described herein as chatbots, bots, or skillbots) are individual bots focused on specific types of tasks such as inventory tracking, time card submission, and expense report creation. When an end user engages with a digital assistant, the digital assistant evaluates the end user's input and routes the conversation to the appropriate chatbot. A digital assistant can be made available to end users through various channels such as FACEBOOK (registered trademark) Messenger, SKYPE MOBILE (registered trademark) messenger, or Short Message Service (SMS). A channel carries chats back and forth between an end user and the digital assistant and its various chatbots on various messaging platforms. A channel can also support user agent escalation, conversations at event start, and testing.

[0019] Through intents, the chatbot becomes able to understand what the user wants from the chatbot. An intent is composed of a permutation of typical user requests and statements, which are also called utterances (e.g., get account balance, purchase, etc.). As used herein, an utterance or message may refer to a set of words (e.g., one or more sentences) exchanged during a conversation with the chatbot. An intent can be created by providing a name that exemplifies a certain user action (e.g., order pizza) and compiling the actual user statements, or set of utterances, generally associated with triggering the action. Since the chatbot's understanding is derived from these intents, each intent can be created from a robust (1 to 20 or so utterances) and diverse dataset so that the chatbot can interpret ambiguous user input. When receiving messages with different expressions but the same meaning, such as "Please forget this order!" or "Please cancel the delivery!", the rich set of utterances enables the chatbot to understand the user's desire. Such intents and the utterances belonging to them together constitute the training corpus of the chatbot. By training a model using this corpus, the customer can basically make it a reference tool for resolving the end-user's input into one intent. The customer can improve the chatbot's understanding ability by repeating the intent testing and intent training.

[0020] However, building a chatbot that can determine the end user's intent based on user utterances is a partially challenging task due to the nuances and ambiguities of natural language, as well as the dimensionality of the input space (e.g., possible user utterances) and the size of the output space (number of intents). Thus, it may be necessary to train, monitor, debug, and retrain chatbots to improve their performance and the user experience using them. Traditional systems have training systems in place for training and retraining machine learning models for digital assistants or chatbots in spoken language understanding (SLU) and natural language processing (NLP). Traditionally, models used in chatbot systems are trained in NLP with "fabricated" utterances for arbitrary intents. For example, a chatbot system's classifier model can be trained to classify the utterance "Do you want to change the price?" into the intent "Do you want to perform price matching?". Training models with fabricated utterances helps in initially training a chatbot system to provide a service, and then allows for retraining once the chatbot system is deployed and begins receiving actual utterances from users.

[0021] As part of the NLP processing of speech, digital assistants are trained to understand the meaning of utterances, and this training includes identifying one or more intents and one or more entities corresponding to the utterance. Entity extraction in digital assistants has two stages: named entity recognition via a named entity recognizer and entity resolution. The specific training problem addressed here relates to named entity recognition. Entities can be understood as nouns and are sometimes called slots. Entities are typically things like dates, times, cities, names, brands, etc. (Some common entities used as examples herein are domain-independent and are called system entities such as PERSON, NUMBER, CURRENCY, and DATE_TIME. However, it should be understood that this disclosure is not limited to these system entities and is applicable to any entity type, such as composite entities, entity roles, and entity lists.) For example, in the case of a travel bot, capturing origin, destination, travel mode, price, date, and time is the basis of the interface. However, because users input data randomly and out of order in various languages ​​and contexts, entity extraction is a challenging task for digital assistants. Table 1 below shows some examples of input (utterances) and output (entities) for entity extraction problems.

[0022] [Table 1]

[0023] However, even so, incorporating these entities is crucial for digital assistants to take actions based on user intent.

[0024] Traditional entity recognition models are trained starting with pre-labeled data. In supervised machine learning settings, the central challenge is the lack of sufficient training data with pre-labeled data for the model to learn from, especially in this problem. More specific challenges in building a robust named entity recognition model for digital assistants include: (i) not covering a wider variety of CURRENCY symbols in the training data (a condition for handling different currencies); (ii) not covering different notations of CURRENCY amounts in the training data (a condition for handling different notations of amounts); (iii) not having specific CURRENCY unit, amount combination utterances in the training data (a condition for handling utterances of large and small units for each currency, and utterances of large units, small units, and amount formats); (iv) not having abbreviated TIME_DATE utterances (a condition for handling abbreviations such as "Sep 5" for September 5th); and (v) not having different formats of PERSON and NUMBER utterances (a condition for handling all formats of names and numbers). Any machine learning model can only achieve good quality by being trained on pre-labeled training data. Therefore, the quality of the training data determines the model's performance.

[0025] The system entities described herein are very common, and therefore, a large number of publicly available datasets exist for use. However, the problem with publicly available pre-labeled data is that it is not diverse enough to train models to detect entities with all kinds of variations (currency type, numerical format, data time format, etc.). One option for obtaining diverse labeled data is to write labeled additional utterances into the training data yourself. Another option is to outsource the writing of labeled additional utterances to freelancers or specialized data labeling companies. There is also the option of using crowdsourcing, which effectively scales up the manual work by utilizing cloud workers. However, these approaches can be difficult to implement in enterprise systems that employ many chatbot systems trained for many different tasks in multiple languages, and receive a wide variety of utterances (many of which are very short) for each task. Systems employing such chatbots need to automatically acquire entity diversity in a synthesis-independent manner in order to quickly and efficiently generate large corpora of multilingual training data for many different chatbots.

[0026] Therefore, different approaches are needed to address these problems. The developed approach uses data augmentation techniques to generate more utterances that have all the characteristics necessary to obtain diversity of entities, and trains a model to detect entities that have all kinds of variations. Methods are provided in various embodiments, one method comprising: a data processing system generating a list of values ​​that cover entities; a data processing system selecting utterances from a dataset having the context of entities; and a data processing system transforming the utterances into a plurality of templates, each of which includes slots of entities that map to a list of values; the method further comprises: a data processing system selecting a template from the plurality of templates; a data processing system selecting values ​​from a list of values ​​based on the mapping between slots in the selected template and a list of values ​​for entities; and a data processing system generating an artificial utterance using the selected template and the selected values, the generation of the artificial utterance comprising inserting the selected values ​​into slots in the selected template.

[0027] Bots and analytics systems A bot (also called a skill, chatbot, chatterbot, or talkbot) is a computer program that can engage in conversations with end users. Bots can typically respond to natural language messages (e.g., questions or comments) through messaging applications that use natural language messaging. Businesses may use one or more bot systems to interact with end users through messaging applications. The messaging application, sometimes called a channel, can be a messaging application that the end user already has installed and is familiar with, or even prefers. Therefore, end users do not need to download and install a new application to chat with the bot system. Messaging applications may include, for example, over-the-top (OTT) messaging channels (such as Facebook Messenger, Facebook WhatsApp, WeChat, Line, Kik, Talk, Skype, Slack, or SMS), virtual private assistants (such as Amazon Dot, Echo, or Show, Google Home, or Apple HomePod), mobile and web application extensions that extend native or hybrid / responsive mobile or web applications with chat capabilities, or voice-based input (such as devices or applications with interfaces that use Siri®, Cortana, Google Voice, or other speech input for conversation).

[0028] In some examples, a bot system may be associated with a URI (Uniform Resource Identifier). A URI can identify a bot system using a string. A URI can be used as a webhook for one or more messaging application systems. A URI may include, for example, a Uniform Resource Locator URL or a Uniform Resource Name (URN). A bot system may be designed to receive messages from a messaging application system (for example, via hypertext transfer protocol: HTTP). HTTP post-call messages may be directed from the messaging application system to the URI. In some embodiments, messages may differ from HTTP post-call messages. For example, a bot system may receive messages from a Short Message Service (SMS). While the description herein refers to communications that a bot system receives as messages, it should be understood that messages may be HTTP post-call messages, SMS messages, or any other type of message between two systems.

[0029] End users can interact with bot systems through conversational interactions (sometimes called conversational user interfaces (UI)), similar to interactions between people. In some cases, this interaction involves the end user saying "hello" to the bot, the bot responding with "hi," and asking the end user what they can do. In other cases, this interaction could be a transactional interaction with a banking bot, such as transferring money from one account to another; an informational interaction with an HR bot, such as checking vacation time; or an interaction with a retail bot, such as returning a purchased item or inquiring about technical support.

[0030] In some embodiments, a bot system can intelligently handle interactions with end users without interacting with the bot system's administrator or developer. For example, an end user may send one or more messages to the bot system to achieve a desired purpose. Messages may include specific content such as text, emojis, voice, images, videos, or other methods of message delivery. In some embodiments, the bot system can convert this content into a standard form (e.g., a REST (representational state transfer) call to an enterprise service with appropriate parameters) and generate a natural language response. The bot system may also prompt the end user for additional input parameters or request other additional information. In some embodiments, the bot system can initiate an interaction with the end user rather than passively responding to their utterance. Described herein are various techniques for identifying explicit calls to a bot system and determining the input to the bot system being called. In certain embodiments, parsing of explicit calls is performed by a master bot based on detecting the call name in the utterance. In response to the detection of the call name, the utterance may be refined for input to the skill bot associated with the call name.

[0031] A conversation with a bot may follow a specific conversational flow that includes multiple states. This flow can predetermine what happens next based on the input. In some embodiments, a bot system can be implemented using a state machine that includes user-defined states (e.g., end-user intents) and actions to be performed in or for each state. The conversation can take various paths based on end-user inputs that may influence the decisions the bot makes about this flow. For example, in each state, the bot may determine the end-user's intent based on the end-user's input to determine the appropriate action to take next. In the context of an utterance, the term “intent” as used herein refers to the intent of the user who provided the utterance. For example, a user may intend to converse with a bot to order a pizza, and therefore the user’s intent can be expressed by the utterance “Please order a pizza.” User intent can be directed to a specific task that the user wants the chatbot to perform on their behalf. Thus, an utterance can be expressed as a question, command, request, etc., that reflects the user’s intent. The intent may include goals that the end-user wants to achieve.

[0032] In this specification, in the context of chatbot configuration, the term “intent” is used to refer to configuration information for mapping user utterances to specific tasks / actions or categories of tasks / actions that the chatbot can perform. To distinguish between utterance intents (i.e., user intents) and chatbot intents, the latter may also be referred to herein as “bot intents.” A bot intent may consist of one or more sets of utterances associated with an intent. For example, the intent to order a pizza may have various reorderings of utterances that express the desire to order a pizza. These associated utterances can be used to train the intent classifier so that it can later determine whether an input utterance from the user matches the pizza order intent. A bot intent may be associated with one or more dialogue flows to initiate a conversation with the user in a particular state. For example, the first message for the pizza order intent may be the question, “What kind of pizza do you prefer?” In addition to associated utterances, a bot intent may further comprise named entities associated with the intent. For example, a pizza order intent might include variables or parameters used to perform the task of ordering a pizza, such as topping 1, topping 2, pizza type, pizza size, and pizza quantity. The entity's values ​​are generally obtained through conversation with the user.

[0033] Figure 1 is a simplified block diagram of an environment 100 incorporating a chatbot system according to a particular embodiment. Environment 100 includes a digital assistant builder platform (DABP) 102, which enables users of DABP 102 to create and deploy digital assistant or chatbot systems. DABP 102 can be used to create one or more digital assistants (DAs) or chatbot systems. For example, as shown in Figure 1, a user 104 representing a particular company can use DABP 102 to create and deploy a digital assistant 106 for users of that company. For example, DABP 102 can be used by a bank to create one or more digital assistants for use by its customers. The same DABP 102 platform may be used by multiple companies to create digital assistants. As another example, a restaurant owner (e.g., a pizzeria) may use DABP 102 to create and deploy a digital assistant that enables restaurant customers to order food (e.g., order a pizza).

[0034] For the purposes of this disclosure, “digital assistant” is an entity that helps users of a digital assistant accomplish various tasks through natural language conversation. A digital assistant can be implemented using software alone (for example, a digital assistant is a digital entity implemented using programs, code, or instructions executable by one or more processors), using hardware, or using a combination of hardware and software. A digital assistant can be embodied or implemented in a variety of physical systems or devices, such as computers, mobile phones, watches, home appliances, and vehicles. A digital assistant is also sometimes referred to as a chatbot system. Therefore, for the purposes of this disclosure, the terms digital assistant and chatbot system are interchangeable.

[0035] A digital assistant, such as a digital assistant 106 built using DABP102, can be used to perform various tasks through natural language-based conversations between the digital assistant and its user 108. As part of the conversation, the user can provide one or more user inputs 110 to the digital assistant 106 and receive responses 112 from the digital assistant 106. The conversation may include one or more of the inputs 110 and responses 112. Through these conversations, the user can request one or more tasks to be performed by the digital assistant, and accordingly, the digital assistant is configured to perform the tasks requested by the user and respond to the user with appropriate responses.

[0036] User input 110 is generally in natural language form and is called utterance. User utterance 110 may also be in text form, for example, when the user types a sentence, question, text fragment, or single word and provides it as input to the digital assistant 106. In some embodiments, user utterance 110 may also be in voice input or speech form, for example, when the user says or speaks something that is provided as input to the digital assistant 106. The utterance is typically the language spoken by the user 108. For example, the utterance may be in English or another language. If the utterance is in speech form, the speech input is converted into a text-form utterance in that particular language, and the text utterance is then processed by the digital assistant 106. Various speech-to-text processing techniques may be used to convert the speech or voice input into a text utterance, which is then processed by the digital assistant 106. In some embodiments, the speech-to-text conversion may be performed by the digital assistant 106 itself.

[0037] Utterances, which may be text or voice utterances, may include fragments, sentences, multiple sentences, one or more words, one or more questions, and combinations of the aforementioned types. The digital assistant 106 is configured to apply natural language understanding (NLU) technology to utterances to understand the meaning of user input. As part of the NLU processing of an utterance, the digital assistant 106 is configured to perform processing to understand the meaning of the utterance, which includes identifying one or more intents and one or more entities corresponding to the utterance. Once the meaning of the utterance is understood, the digital assistant 106 can perform one or more actions or operations in response to the understood meaning or intent. For the purposes of this disclosure, utterances are assumed to be either text utterances directly provided by the user 108 of the digital assistant 106, or the result of converting input voice utterances into text format. However, this is not intended to be limiting or restrictive in any way.

[0038] For example, user input 108 may request to order a pizza by providing an utterance such as, "I would like to order a pizza." Upon receiving such an utterance, the digital assistant 106 is configured to understand the meaning of the utterance and take appropriate action. Appropriate action may include responding to the user with a question requesting user input regarding the type of pizza the user wishes to order, the size of the pizza, the pizza toppings, etc. The responses provided by the digital assistant 106 may also be in natural language form and typically in the same language as the input utterance. As part of generating these responses, the digital assistant 106 may perform natural language generation (NLG). For a user ordering a pizza, the digital assistant 106 may provide all the necessary information for ordering the pizza through a conversation between the user and the digital assistant 106, and then guide the user to order the pizza at the end of the conversation. The digital assistant 106 may terminate the conversation by outputting information to the user indicating that the pizza has been ordered.

[0039] At a conceptual level, the digital assistant 106 performs various processes in response to utterances received from the user. In some embodiments, this process includes a series of or pipelined processing steps, such as understanding the meaning of the input utterance (sometimes called natural language understanding (NLU)), determining what action to take in response to the utterance, performing the action as necessary, generating a response to be output to the user in response to the user's utterance, and outputting the response to the user. NLU processing may include parsing the received input utterance to understand its structure and meaning, and refining and improving the utterance to develop a more understandable form (e.g., logical form) or structure of the utterance. Generating a response may include using NLG techniques.

[0040] NLU processing performed by a digital assistant such as digital assistant 106 may include a variety of NLP-related processes, such as sentence parsing (e.g., tokenization, headword creation, identification of part-of-speech tags in the sentence, identification of named entities in the sentence, generation of dependency trees to represent sentence structure, splitting the sentence into clauses, parsing individual clauses, anaphora resolution, performing chunking, etc.). In certain embodiments, the NLU processing or a portion of it is performed by digital assistant 106 itself. In some other embodiments, digital assistant 106 may use other resources to perform a portion of the NLU processing. For example, the syntax and structure of an input utterance may be identified by processing the sentence using a parser, part-of-speech tagger, and / or named entity recognizer. In one embodiment, for English, a parser, part-of-speech tagger, and named entity recognizer, such as those provided by the Stanford Natural Language Processing (NLP) Group, are used to parse the sentence structure and syntax. These are provided as part of the Stanford CoreNLP Toolkit.

[0041] The various examples provided in this disclosure demonstrate English utterances, but these are intended as examples only. In certain embodiments, the digital assistant 106 may also process utterances in languages ​​other than English. The digital assistant 106 may provide subsystems (e.g., components implementing NLU functionality) configured to perform processing for different languages. These subsystems may be implemented as pluggable units that can be invoked using service calls from the NLU core server. This allows for flexible and scalable NLU processing on a language-by-language basis, such as changing the order of processing. Language packs may be provided for individual languages, and language packs may register a list of subsystems that can be provided by the NLU core server.

[0042] A digital assistant, such as the digital assistant 106 shown in Figure 1, can be made available or accessible to its user 108 through various different channels, including, but not limited to, specific applications, social media platforms, various messaging services and applications, and other applications or channels. A single digital assistant may have multiple channels configured to run and be accessible simultaneously on different services.

[0043] A digital assistant or chatbot system generally includes or is associated with one or more skills. In certain embodiments, these skills are individual chatbots (referred to as skillbots) configured to interact with the user and perform specific types of tasks, such as inventory tracking, time card submission, expense report creation, food ordering, bank account verification, reservations, and widget purchases. For example, in the embodiment shown in Figure 1, the digital assistant or chatbot system 106 includes skills 116-1, 116-2, and so on. For the purposes of this disclosure, the terms “one skill” and “multiple skills” are used synonymously with the terms “one skillbot” and “multiple skillbots,” respectively.

[0044] Each skill associated with a digital assistant helps the user complete tasks through conversations with the user, and these conversations may include a combination of text or voice input provided by the user and responses provided by the skill bot. These responses may be in the form of text or voice messages to the user, and / or in the form of simple user interface elements (e.g., selection from a list) presented to the user for making a choice.

[0045] There are various ways in which skills or skillbots can be associated with or added to digital assistants. In some cases, skillbots are developed by companies and then added to digital assistants using DABP102. In other cases, skillbots are developed and created using DABP102 and then added to digital assistants created using DABP102. In yet another case, DABP102 provides an online digital store (referred to as the "skill store") offering multiple skills aimed at a wide range of tasks. Skills offered through the skill store can also be published to various cloud services. To add skills to digital assistants generated using DABP102, DABP102 users can access the skill store via DABP102, select the desired skills, and instruct DABP102 to add the selected skills to the digital assistant generated using DABP102. Skills from the Skill Store can be added to a digital assistant either as is or in a modified form. (For example, a DABP102 user may select and clone a specific skill bot from the Skill Store, customize or modify the selected skill bot, and then add the modified skill bot to a digital assistant created using DABP102.)

[0046] Various different architectures may be used to implement a digital assistant or chatbot system. For example, in a particular embodiment, a digital assistant created and deployed using DABP102 may be implemented using a master-bot / child (or sub)bot paradigm or architecture. According to this paradigm, the digital assistant is implemented as a master bot that interacts with one or more child bots, which are skill bots. For example, in the embodiment shown in Figure 1, the digital assistant 106 consists of a master bot 114 and skill bots 116-1, 116-2, etc., which are child bots of the master bot 114. In a particular embodiment, the digital assistant 106 is considered to function as a master bot itself.

[0047] A digital assistant implemented according to a master-bot-child-bot architecture allows users of the digital assistant to interact with multiple skills through a unified user interface, i.e., via the master bot. When a user engages with the digital assistant, user input is received by the master bot. The master bot then performs processing to determine the meaning of the user's input utterance. Next, the master bot determines whether it can handle the task requested by the user in the utterance, and furthermore, the master bot selects a skill bot suitable for handling the user's request and routes the conversation to the selected skill bot. This allows users to converse with the digital assistant through a common, single interface, while still providing the ability to use multiple skill bots configured to perform specific tasks. For example, in the case of a digital assistant developed for an enterprise, the digital assistant's master bot can interface with skill bots with specific functions, such as a CRM bot to perform functions related to customer relationship management (CRM), an ERP bot to perform functions related to enterprise resource planning (ERP), and an HCM bot to perform functions related to human capital management (HCM). Thus, end-users or consumers of digital assistants only need to know how to access them through a common master bot interface, with multiple skill bots in place behind the scenes to handle user requests.

[0048] In certain embodiments, the master-bot / child-bot infrastructure is configured so that the master-bot is aware of the available list of skill-bots. The master-bot can access metadata that identifies the various available skill-bots and the capabilities of the skill-bots, including the tasks that each skill-bot can perform. Upon receiving a user request in the form of an utterance, the master-bot is configured to identify or predict from among several available skill-bots which particular skill-bot is best suited to respond to and process the user request. The master-bot then routes the utterance (or part of the utterance) to that particular skill-bot for further processing. In this way, control flows from the master-bot to the skill-bots. The master-bot can support multiple input and output channels.

[0049] The embodiment shown in Figure 1 illustrates a digital assistant 106 comprising a master bot 114 and skill bots 116-1, 116-2, and 116-3, but this is not intended to be limiting. The digital assistant may include various other components (e.g., other systems and subsystems) that provide the functionality of the digital assistant. These systems and subsystems may be implemented solely in software (e.g., code, instructions stored on a computer-readable medium and executable by one or more processors), solely in hardware, or in implementations using a combination of software and hardware.

[0050] DABP102 provides the infrastructure, services, and functions that enable DABP102 users to create digital assistants that include one or more skillbots associated with the digital assistant. In some cases, skillbots can be created by cloning existing skillbots, for example, by cloning skillbots provided by the skill store. As previously mentioned, DABP102 provides a skill store or skill catalog that offers multiple skillbots for performing various tasks. DABP102 users can clone skillbots from the skill store. They can modify or customize the cloned skillbots as needed. In some other cases, DABP102 users create skillbots from scratch using the tools and services provided by DABP102. As previously mentioned, the skill store or skill catalog provided by DABP102 may offer multiple skillbots for performing various tasks.

[0051] In a particular embodiment, creating or customizing a skill bot involves, in general terms, the following steps:

[0052] (1) Steps to set up a new skillbot (2) Steps to set up one or more intents for SkillBot (3) Steps to constitute one or more entities for one or more intents (4) Steps to train the skillbot (5) Steps to create a SkillBot dialogue flow (6) Steps to add custom components to the skillbot as needed (7) Steps to test and deploy the skillbot Each of the above steps is briefly explained below.

[0053] (1) Steps to configure a new skillbot - various settings can be configured for a skillbot. For example, a skillbot designer can specify one or more invocation names for the skillbot being created. Users of the digital assistant can then explicitly invoke the skillbot using these invocation names. For example, a user can explicitly invoke the corresponding skillbot by typing the invocation name in their utterance.

[0054] (2) Steps to configure one or more intents and associated utterance examples for a skillbot - The skillbot designer specifies one or more intents (also called bot intents) for the skillbot being created. The skillbot is then trained based on these specified intents. These intents represent categories or classes of input utterances that the skillbot is trained to infer. Upon receiving an utterance, the trained skillbot infers the intent for the utterance, and the inferred intent is selected from a predefined set of intents used to train the skillbot. The skillbot then responds to the utterance by taking an appropriate action based on the intent inferred for that utterance. In some cases, the skillbot's intents represent tasks that the skillbot can perform for the user of the digital assistant. Each intent is given an intent identifier or intent name. For example, for a skillbot trained for banking, the intents specified for this skillbot might include "CheckBalance," "TransferMoney," and "DepositCheck."

[0055] For each intent defined for a skillbot, the skillbot designer may also provide one or more example utterances that represent and describe the intent. These example utterances are intended to represent utterances that a user might input to the skillbot for that intent. For example, for the CheckBalance intent, example utterances might include "What is the balance in my savings account?", "How much is in my checking account?", and "How much is in my account?". Thus, various permutations of typical user utterances may be designated as example utterances for an intent.

[0056] Intents and their associated utterance examples are used as training data to train a skillbot. Various different training techniques may be used. As a result of this training, a predictive model is generated, which is configured to take an utterance as input and output the intent of that utterance as inferred by the predictive model. In some cases, the input utterance is provided to an intent analysis engine configured to predict or infer the intent of the input utterance using the trained model. The skillbot can then take one or more actions based on the inferred intent.

[0057] (3) Steps to configure entities for one or more intents of the skillbot - In some cases, additional context may be needed to enable the skillbot to respond appropriately to user utterances. For example, there may be situations where user input utterances resolve to the same intent in the skillbot. For example, in the example above, the utterances "What is the balance in my savings account?" and "How much is in my checking account?" both resolve to the same CheckBalance intent, but these utterances are different requests asking for different things. To clarify such requests, one or more entities are added to the intent. Using the example of a banking skillbot, an entity called AccountType that defines values ​​called "checking" and "savings" may enable the skillbot to parse user requests and respond appropriately. In the example above, the utterances resolve to the same intent, but the values ​​associated with the AccountType entity are different for these two utterances. This allows the skillbot to perform different actions for the two utterances, even though they resolve to the same intent. One or more entities can be specified for a particular intent configured for the skillbot. Therefore, we use entities to add context to the intent itself. Entities help to further explain the intent and enable the skillbot to complete the user request.

[0058] In certain embodiments, there are two types of entities: (a) built-in entities provided by DABP102, and (2) custom entities that can be specified by the skillbot designer. Built-in entities are general-purpose entities that can be used with a wide variety of bots. Examples of built-in entities include, but are not limited to, time, date, address, number, email address, duration, recurring duration, currency, number of calls, URL, etc. Custom entities are used for more customized applications. For example, in a banking skill, the skillbot designer may define an AccountType entity that enables various banking transactions by checking user input for keywords such as checking, savings, and credit card.

[0059] (4) Steps to train the skillbot - The skillbot is configured to receive user input in the form of utterances, and to parse or process the received input to identify or select an intent associated with the received user input. As described above, the skillbot needs to be trained for this purpose. In certain embodiments, the skillbot is trained on intents configured for the skillbot and utterance examples associated with those intents (collectively, the training data) so that the skillbot can resolve a user input utterance to one of its configured intents. In certain embodiments, the skillbot uses a predictive model that is trained with the training data to enable the skillbot to discern what the user is saying (or, in some cases, what the user is trying to say). DABP102 provides a variety of different training techniques that skillbot designers can use to train the skillbot, and these training techniques include a variety of machine learning-based training techniques, rule-based training techniques and / or combinations thereof. In certain embodiments, a portion of the training data (e.g., 80%) is used to train the skillbot model, and another portion (e.g., the remaining 20%) is used to test or validate the model. Once trained, a trained model (sometimes referred to as a trained skillbot) can be used to process and respond to user utterances. In certain cases, a user utterance may be a question that requires only one answer and does not require further conversation. To address such situations, a Q&A (question and answer) intent may be defined for the skillbot. This allows the skillbot to output an answer to a user request without having to update the dialogue definition. Q&A intents are created in the same manner as regular intents. The dialogue flow for Q&A intents may differ from that of regular intents.

[0060] (5) Steps to create a dialogue flow for a skillbot - The dialogue flow specified for a skillbot describes how the skillbot responds as its various intents are resolved in response to received user input. This dialogue flow defines the actions or behaviors of the skillbot, such as how the skillbot responds to user utterances, how the skillbot requests input from the user, and how the skillbot returns data. The dialogue flow is like a flowchart that the skillbot follows. The skillbot designer specifies the dialogue flow using a language such as Markdown. In certain embodiments, a version of YAML called OBotML may be used to specify the dialogue flow for the skillbot. The dialogue flow definition for a skillbot serves as a model of the conversation itself, i.e., a model that allows the skillbot designer to construct the interaction between the skillbot and the user that the skillbot serves.

[0061] In a particular embodiment, the dialogue flow definition for the skillbot includes the following three sections:

[0062] (a) Context section (b) Default transition section (c) State section Context Section - Skillbot designers can define variables used in the conversation flow within the context section. Other variables that can be named within the context section include, but are not limited to, variables for error handling, variables for built-in or custom entities, and user variables that allow Skillbot to recognize and persist user preferences.

[0063] Default Transition Section - Skillbot transitions can be defined in either the Dialogflow State section or the Default Transition section. Transitions defined in the Default Transition section act as fallbacks and are triggered when there are no applicable transitions defined within a state, or when the conditions necessary to trigger a state transition cannot be met. The Default Transition section can be used to define routing that allows Skillbot to handle unexpected user actions in a sophisticated manner.

[0064] The State Section – Dialog Flow and its associated behavior are defined as a set of transient states that govern the logic within the dialog flow. Each state node in the dialog flow definition names a component that provides the functionality required for the dialog at that moment. Thus, states are built around components. States contain characteristics specific to the component and define transitions to other states that are triggered after the component has been executed.

[0065] Special case scenarios may be handled using the state section. For example, you might want to give a user the option to temporarily leave the first skill they are involved with and do something with a second skill within the digital assistant. For instance, a user might be conversing with a shopping skill (e.g., having made some purchase choices), then want to jump to a banking skill (e.g., to make sure they have enough money for the purchase), and then return to the shopping skill to complete the user's order. To address this, the actions in the first skill can be configured to return to the original flow after initiating an interaction with a second, different skill within the same digital assistant.

[0066] (6) Steps to add custom components to the SkillBot - As described above, the states specified in the SkillBot's dialog flow name the components that provide the necessary functionality corresponding to those states. Components enable the SkillBot to perform functions. In certain embodiments, DABP102 provides a set of pre-configured components for performing a wide range of functions. The SkillBot designer can select one or more of these pre-configured components and associate them with states in the SkillBot's dialog flow. Alternatively, the SkillBot designer can use the tools provided by DABP102 to create custom or new components and associate these custom components with one or more states in the SkillBot's dialog flow.

[0067] (7) Steps to test and deploy the skillbot - DABP102 provides several features that allow the skillbot designer to test the skillbot being developed. The skillbot can then be deployed and included in the digital assistant.

[0068] The above describes how to create a skillbot, but similar techniques may be used to create a digital assistant (or masterbot). At the masterbot or digital assistant level, built-in system intents may be configured for the digital assistant. These built-in system intents are used to identify common tasks that the digital assistant itself (i.e., the masterbot) can handle without calling the skillbot associated with the digital assistant. Examples of system intents defined for a masterbot include (1) Exit, which is applied when the user expresses a desire to end the current conversation or context in the digital assistant; (2) Help, which is applied when the user requests help or orientation; and (3) UnresolvedIntent, which is applied to user input that does not well match the exit and help intents. The digital assistant also stores information about one or more skillbots associated with it. This information allows the masterbot to select a specific skillbot to process the utterance.

[0069] At the master bot or digital assistant level, when a user inputs a phrase or utterance into the digital assistant, the digital assistant is configured to perform processing to determine how to route the utterance and the associated conversation. The digital assistant uses a routing model to make this determination. This routing model may be rule-based, AI-based, or a combination of both. The digital assistant uses this routing model to determine whether the conversation corresponding to the user-input utterance is routed to a specific skill for processing, processed by the digital assistant or master bot itself on a per-built-in system intent basis, or processed as a different state in the current conversation flow.

[0070] In certain embodiments, as part of this process, the digital assistant determines whether the user input utterance explicitly identifies a skillbot using its invocation name. If an invocation name is present in the user input, it is treated as an explicit invocation of the skillbot corresponding to the invocation name. In such scenarios, the digital assistant may route the user input to the explicitly invoked skillbot for further processing. If there is no specific or explicit invocation, in certain embodiments, the digital assistant evaluates the received user input utterance to calculate a confidence score for the system intent and the skillbot associated with the digital assistant. The calculated score for a skillbot or system intent indicates how likely the user input is to represent a task that the skillbot is configured to perform, or to represent a system intent. Any system intent or skillbot whose associated calculated confidence score exceeds a threshold (e.g., a confidence threshold routing parameter) is selected as a candidate for further evaluation. The digital assistant then selects a specific system intent or skillbot from the identified candidates for further processing of the user input utterance. In a particular embodiment, after one or more skill bots are identified as candidates, the intents associated with those candidate skills are evaluated (according to a per-skill intent model), and a confidence score is obtained for each intent. Generally, any intent with a confidence score exceeding a threshold (e.g., 70%) is treated as a candidate intent. Once a specific skill bot is selected, the user utterance is routed to that skill bot for further processing. Once a system intent is selected, one or more actions are performed by the master bot itself according to the selected system intent.

[0071] Figure 2 is a simplified block diagram showing a master bot (MB) system 200 according to a particular embodiment. The MB system 200 can be implemented using software only, hardware only, or a combination of hardware and software. The MB system 200 includes a preprocessing subsystem 210, a multiple intent subsystem (MIS) 220, an explicit invocation subsystem (EIS) 230, a skillbot caller 240, and a data store 250. The MB system 200 shown in Figure 2 is merely one example of the arrangement of components in a master bot. Those skilled in the art will recognize many possible variations, alternatives, and modifications. For example, in some implementations, the MB system 200 may have more or fewer systems or components than those shown in Figure 2, may combine two or more subsystems, or may have different configurations or arrangements of subsystems.

[0072] The preprocessing subsystem 210 receives the utterance "A" 202 from the user and processes it via the language detector 212, the language syntactic analyzer 214, and the named entity recognition system 216. As described above, the utterance can be provided in various ways, including speech or text. The utterance 202 may be a sentence fragment, a complete sentence, multiple sentences, etc. The utterance 202 may contain punctuation. For example, if the utterance 202 is provided as speech, the preprocessing subsystem 210 can convert the speech to text using a speech-to-text converter (not shown) that inserts punctuation, such as commas, semicolons, periods, etc., into the resulting text.

[0073] The language detector 212 detects the language of utterance 202 based on the text of utterance 202. Since each language has its own grammar and semantics, the method of processing utterance 202 is language-dependent. Differences between languages ​​are taken into consideration when analyzing the syntactic and structural aspects of the utterance.

[0074] The language parser 214 parses the utterance 202 and extracts part-of-speech (POS) tags for individual linguistic units (e.g., words) within the utterance 202. POS tags include, for example, nouns (noun: NN), pronouns (pronoun: PN), and verbs (verb: VB). The language parser 214 can also tokenize the linguistic units of the utterance 202 (e.g., convert each word into a separate token) and turn the words into headwords. A headword is the primary form of a set of words as represented in a dictionary (e.g., "run" is the headword for run, runs, ran, running, etc.). Other types of preprocessing that the language parser 214 can perform include chunking compound expressions, for example, combining "credit" and "card" into a single expression "credit_card". The language parser 214 can also identify relationships between words within the utterance 202. For example, in some embodiments, the language parser 214 generates a dependency tree that indicates which parts of an utterance (e.g., a specific noun) are direct objects, which parts of an utterance are prepositions, and so on.

[0075] The named entity recognition (NAME) recognition machine learning model 216 identifies named entities in text (e.g., utterances) and classifies them into predefined categories such as people, organizations, places, time representations, currencies, and Universal Resource Language addresses. Identity identification involves detecting words or sequences of words that form predefined entities. For example, each word represents a token, and "Pacific" is a string of three tokens representing one entity. In some cases, inside-outside-beginning tagging, which indicates the beginning and end of an entity, may be used to facilitate entity identification. Each identified entity can be classified into a predefined category. For example, a named entity recognition machine learning model might detect the word "oracle" in a text string, identify the word as an entity, and classify it as "company." To learn what constitutes a relevant entity, what constitutes an unrelated entity, and how to perform the classification, the named entity recognition machine learning model implemented by the named entity recognition recognition model 216 is trained on a corpus of training data. The more relevant the training data is to a task or skill, the more accurately each model will perform named entity recognition for that task or skill. Once defined, entities and categories can be used to label data and create training datasets (in some cases, named entity recognition data programs can do this automatically). This training dataset can then be used to train one or more algorithms to predictively label text for entities and categories in newly presented utterances.

[0076] The results of processing performed by the language detector 212, the language syntactic analyzer 214, and the named entity recognition system 216 form extracted information 205, which is provided as input to the MIS 220 along with the utterance 202 itself.

[0077] As described above, utterance 202 may contain two or more sentences. For the purpose of detecting multiple intents and explicit calls, utterance 202 may be treated as a single unit even if it contains multiple sentences. However, in certain embodiments, preprocessing may be performed, for example, by the preprocessing subsystem 210, to identify a single sentence among multiple sentences for multiple intent analysis and explicit call analysis. In general, whether utterance 202 is processed at the level of individual sentences or as a single unit containing multiple sentences, the results produced by MIS220 and the results produced by EIS230 are substantially the same.

[0078] MIS220 determines whether utterance 202 represents multiple intents. While MIS220 can detect the presence of multiple intents in utterance 202, the processing performed by MIS220 does not include determining whether the intents in utterance 202 match any intent configured for the bot. Instead, the processing to determine whether the intents in utterance 202 match a bot intent can be performed by the intent classifier 242 of the MB system 200 or the skill bot's intent classifier (shown in the embodiment of Figure 3). The processing performed by MIS220 assumes the existence of a bot capable of processing utterance 202 (e.g., a specific skill bot or the master bot itself). Therefore, the processing performed by MIS220 does not require knowledge of what bots are in the chatbot system (e.g., the identity of a skill bot registered with the master bot) or what intents are configured for a particular bot.

[0079] To determine that utterance 202 contains multiple intents, MIS220 applies one or more rules from a set of rules 252 in datastore 250. The rule applied to utterance 202 is determined by the language of utterance 202 and may include sentence patterns indicating the presence of multiple intents. For example, a sentence pattern may include a coordinating conjunction (e.g., a conjunction) that connects two parts of a sentence, where both parts correspond to separate intents. If utterance 202 matches a sentence pattern, it can be inferred that utterance 202 represents multiple intents. Note that utterances with multiple intents do not necessarily have different intents (e.g., intents directed to different bots or intents directed to different intents within the same bot). Instead, the utterance may have separate instances of the same intent, for example, "order a pizza using payment account X, and then order a pizza using payment account Y."

[0080] As part of its determination that utterance 202 represents multiple intents, MIS220 also determines which parts of utterance 202 are associated with each intent. For each intent represented in an utterance containing multiple intents, MIS220 constructs a new utterance for alternative processing, such as utterances "B" 206 and "C" 208 shown in Figure 2, to replace the original utterance. Thus, the original utterance 202 can be split into two or more separate utterances, each processed one at a time. Using extracted information 205 and / or from the analysis of utterance 202 itself, MIS220 determines which of the two or more utterances should be processed first. For example, MIS220 may determine that utterance 202 contains a marker word indicating that a particular intent should be processed first. The newly formed utterance corresponding to this particular intent (e.g., either utterance 206 or utterance 208) is then sent first for further processing by EIS230. After the conversation triggered by the first utterance has ended (or been temporarily interrupted), the next highest priority utterance (for example, utterance 206 or the other of utterance 208) may be sent to the EIS230 for processing.

[0081] The EIS230 determines whether the utterance it receives (for example, utterance 206 or utterance 208) contains a skillbot invocation name. In certain embodiments, each skillbot in the chatbot system is assigned a unique invocation name that distinguishes it from other skillbots in the chatbot system. A list of invocation names can be maintained in the data store 250 as part of the skillbot information 254. If an utterance contains a word that matches an invocation name, the utterance is considered an explicit invocation. If the bot is not explicitly invoked, the utterance received by the EIS230 is considered an implicit invocation utterance 234 and is fed into the master bot's intent classifier (for example, intent classifier 242) to determine which bot to use to process the utterance. In some cases, the intent classifier 242 determines that the master bot should process the implicit invocation utterance. In other cases, the intent classifier 242 determines which skillbot the utterance is routed to for processing.

[0082] The explicit call functionality provided by EIS230 offers several advantages. It can reduce the amount of processing the master bot must perform. For example, with explicit calls, the master bot does not need to perform intent classification analysis (e.g., using intent classifier 242), or it can reduce the amount of intent classification analysis it must perform to select a skill bot. Thus, explicit call analysis can enable the selection of a specific skill bot without relying on intent classification analysis.

[0083] Furthermore, there may be overlapping functionalities among multiple skill bots. This can occur, for example, when two skill bots process the same or very similar intents. In such situations, it becomes difficult for the master bot to determine which of the multiple skill bots to select based solely on intent classification analysis. In these scenarios, an explicit call clarifies which specific skill bot should be used.

[0084] In addition to determining that an utterance is an explicit invocation, the EIS230 is responsible for determining whether any part of the utterance should be used as input to the explicitly invoked skillbot. Specifically, the EIS230 can determine whether any part of the utterance is not associated with an invocation. The EIS230 can make this determination by parsing the utterance and / or parsing the extracted information 205. Instead of sending the entire utterance received by the EIS230, the EIS230 can send only the parts of the utterance that are not associated with an invocation to the invoked skillbot. In some cases, the input to the invoked skillbot is formed simply by removing any part of the utterance that is associated with the invocation. For example, "I want to order a pizza using PizzaBot" can be shortened to "I want to order a pizza" because "using PizzaBot" is related to the invocation of PizzaBot, but irrelevant to the process performed by PizzaBot. In some cases, the EIS230 may reformat the parts sent to the invoked bot, for example, to form a complete sentence. Therefore, the EIS230 determines not only whether there is an explicit call, but also what to send to the skillbot if there is an explicit call. In some cases, there may be no text to input to the bot being called. For example, if the utterance is "pizzabot", the EIS230 can determine that pizzabot is being called, but there is no text to process by pizzabot. In such a scenario, the EIS230 may inform the skillbot caller 240 that there is nothing to send.

[0085] The skillbot caller 240 can invoke skillbots in various ways. For example, the skillbot caller 240 can invoke a bot in response to receiving an indication 235 that a particular skillbot has been selected as a result of an explicit invocation. The indication 235 may be sent by the EIS 230 along with input about the explicitly invoked skillbot. In this scenario, the skillbot caller 240 delegates control of the conversation to the explicitly invoked skillbot. The explicitly invoked skillbot determines an appropriate response to the input from the EIS 230 by treating the input as a standalone utterance. For example, the response may be to perform a specific action or to start a new conversation in a specific state where the initial state of the new conversation is determined by the input sent from the EIS 230.

[0086] Another way in which the SkillBot Caller 240 can invoke a SkillBot is through an implicit invocation using the Intent Classifier 242. The Intent Classifier 242 can be trained using machine learning and / or rule-based training techniques to determine the likelihood that an utterance represents a task that a particular SkillBot is configured to perform. The Intent Classifier 242 is trained in different classes, i.e., one class for each SkillBot. For example, whenever a new SkillBot is registered with the MasterBot, the Intent Classifier 242 can be trained to determine the likelihood that a particular utterance represents a task that the new SkillBot can perform, using a list of utterance examples associated with this new SkillBot. The parameters generated as a result of this training (e.g., a set of parameter values ​​for a machine learning model) can be stored as part of the SkillBot Information 254.

[0087] In certain embodiments, the intent classifier 242 is implemented using a machine learning model, as described further in this specification. Training the machine learning model may involve inputting at least a subset of utterances from utterance examples associated with various skill bots to generate, as output of the machine learning model, inferences about which bot is the correct bot to process any particular training utterance. For each training utterance, a representation of the correct bot to use for that training utterance may be provided as ground truth information. The behavior of the machine learning model can then be adapted (for example, via backpropagation) to minimize the difference between the generated inferences and the ground truth information.

[0088] In certain embodiments, the intent classifier 242 calculates a confidence score for each skill bot registered with the master bot, indicating the likelihood that the skill bot can process an utterance (e.g., an implicit calling utterance 234 received from the EIS 230). The intent classifier 242 may also calculate a confidence score for each configured system-level intent (e.g., help, exit). If a particular confidence score meets one or more conditions, the skill bot caller 240 invokes the bot associated with that particular confidence score. For example, it may need to meet a threshold confidence score value. Therefore, the output 245 of the intent classifier 242 is either an identification of a system intent or an identification of a specific skill bot. In some cases, in addition to meeting a threshold confidence score value, the confidence score must exceed the next highest confidence score by a certain win margin. Imposing such conditions allows routing to a specific skill bot if each of the confidence scores of multiple skill bots exceeds a threshold confidence score value.

[0089] After identifying a bot based on its confidence score, the skillbot caller 240 hands over processing to the identified bot. In the case of a system intent, the identified bot is the master bot. Otherwise, the identified bot is a skillbot. Furthermore, the skillbot caller 240 determines what to provide as input 247 to the identified bot. As described above, in the case of an explicit call, input 247 may be based on a portion of an utterance not associated with this call, or input 247 may be nothing (e.g., an empty string). In the case of an implicit call, input 247 may be the entire utterance.

[0090] The data store 250 includes one or more computing devices that store data used by various subsystems of the master bot system 200. As described above, the data store 250 includes rules 252 and skill bot information 254. Rules 252 include, for example, rules for MIS 220 to determine when an utterance represents multiple intents and how to divide an utterance that represents multiple intents. Rules 252 further include rules for EIS 230 to determine which parts of an utterance that explicitly invokes a skill bot are sent to the skill bot. Skill bot information 254 includes the invocation names of skill bots in the chatbot system, for example, a list of invocation names of all skill bots registered with a particular master bot. Skill bot information 254 may also include confidence scores for each skill bot in the chatbot system, for example, information used by the intent classifier 242 to determine parameters for a machine learning model.

[0091] Figure 3 is a simplified block diagram of a SkillBot System 300 according to a particular embodiment. The SkillBot System 300 is a computing system that can be implemented with software only, hardware only, or a combination of hardware and software. In certain embodiments, such as the embodiment shown in Figure 1, the SkillBot System 300 can be used to implement one or more SkillBots in a digital assistant.

[0092] The Skillbot system 300 includes an MIS 310, an intent classifier 320, and a conversation manager 330. The MIS 310 is similar to the MIS 220 in Figure 2 and provides similar functionality, including being operable to determine (1) whether an utterance represents multiple intents, and if so, (2) how to split this utterance into separate utterances for each of the multiple intents, using rules 352 in the data store 350. In a particular embodiment, the rules applied by the MIS 310 to detect multiple intents and to split an utterance are the same as the rules applied by the MIS 220. The MIS 310 receives an utterance 302 and extracted information 304. The extracted information 304 is similar to the extracted information 205 in Figure 2 and can be generated using a language detector 212, a language parser 214 and / or a named entity recognizer 216, or a language detector, language parser and / or named entity recognizer local to the Skillbot system 300.

[0093] The intent classifier 320 can be trained in a manner similar to that of the intent classifier 242 described above in relation to the embodiment of Figure 2, as described in further detail herein. For example, in a particular embodiment, the intent classifier 320 is implemented using a machine learning model. The machine learning model of the intent classifier 320 is trained for a particular skill bot using at least a subset of utterance examples associated with that particular skill bot as training utterances. The ground truth for each training utterance would be the specific bot intent associated with that training utterance.

[0094] Utterance 302 can be received directly from the user or supplied via a master bot. If utterance 302 is supplied via a master bot as a result of processing via MIS220 and EIS230 in the embodiment shown in Figure 2, for example, MIS310 can be bypassed to avoid repeating processing already performed by MIS220. However, if utterance 302 is received directly from the user during a conversation that occurs after routing to a skill bot, for example, MIS310 can process utterance 302 to determine whether utterance 302 represents multiple intents. In that case, MIS310 applies one or more rules to split utterance 302 into separate utterances for each intent, for example, utterance "D" 306 and utterance "E" 308. If utterance 302 does not represent multiple intents, MIS310 forwards utterance 302 to the intent classifier 320 for intent classification without splitting it.

[0095] The intent classifier 320 is configured to match received utterances (e.g., utterances 306 or 308) with intents associated with the skillbot system 300. As described above, a skillbot can consist of one or more intents, each intent including at least one example utterance associated with the intent and used to train the classifier. In the embodiment of Figure 2, the intent classifier 242 of the masterbot system 200 is trained to determine confidence scores for individual skillbots and system intents. Similarly, the intent classifier 320 can be trained to determine confidence scores for each intent associated with the skillbot system 300. The classification performed by the intent classifier 242 is at the bot level, while the classification performed by the intent classifier 320 is at the intent level and therefore more granular. The intent classifier 320 accesses intent information 354. The intent information 354 includes a list of utterances for each intent associated with the skillbot system 300, where this list of utterances represents and explains the meaning of the intent and is generally associated with the tasks that can be performed by that intent. The intent information 354 may further include parameters generated as a result of training on this list of utterances.

[0096] The conversation manager 330 receives an indication 322 as output from the intent classifier 320, which indicates that a particular intent identified by the intent classifier 320 best matches the utterance input to the intent classifier 320. In some cases, the intent classifier 320 may not be able to determine any match. For example, if the utterance is directed to a system intent or an intent of a different skillbot, the confidence score calculated by the intent classifier 320 may fall below a threshold confidence score value. When this occurs, the skillbot system 300 may delegate the utterance to the master bot for processing, for example, to route it to a different skillbot. However, if the intent classifier 320 successfully identifies an intent within the skillbot, the conversation manager 330 will initiate a conversation with the user.

[0097] A conversation initiated by the conversation manager 330 is a conversation specific to an intent identified by the intent classifier 320. For example, the conversation manager 330 may be implemented using a state machine configured to execute a dialogue flow for the identified intent. This state machine may include a default initiation state (e.g., an indent is invoked without additional input) and one or more further states, each associated with an action performed by the skillbot (e.g., executing a purchase transaction) and / or a dialogue presented to the user (e.g., a question, answer). Thus, the conversation manager 330 can determine an action / dialogue 335 upon receiving an indication 322 that identifies an intent, and can determine further actions or dialogues in response to subsequent utterances received during the conversation.

[0098] The data store 350 includes one or more computing devices that store data used by various subsystems of the skillbot system 300. As shown in Figure 3, the data store 350 includes rule 352 and intent information 354. In certain embodiments, the data store 350 can be integrated with the data store of the master bot or digital assistant, for example, the data store 250 in Figure 2.

[0099] Entity Data Extension Models used for entity recognition and extraction have been found to produce poor results for texts containing entities that can be expressed in a wide range of languages ​​and formats. To overcome this problem, various embodiments have been directed toward techniques for augmenting training data with a diverse set of entities in any number of languages ​​and formats (e.g., adding different texts and text formats used to describe one or more entities) so that the model can detect entities with all kinds of variations (currency types, numeric formats, date_time formats, etc.). By augmenting the training data with a diverse set of entities in various languages ​​and formats, better entity recognition and extraction of the model becomes possible. As illustrated with respect to Figures 1, 2, and 3, a model trained with augmented entity data can be implemented in a chatbot system. Advantageously, these models and chatbots perform better for utterances in various languages ​​and / or text formats because the model has a greater ability to recognize and extract entities in utterances. Furthermore, since the augmentation is applied automatically in a synthesis-independent manner, customers or clients do not need to worry about manually adding utterances in various languages ​​and / or text formats to the training data.

[0100] Figure 4 is a block diagram showing an embodiment of a chatbot system 400 configured to train one or more models, for example, one or more models implemented as named entity recognizers, based on text data. As shown in Figure 4, the entity recognition performed by the chatbot system 400 in this example includes various stages: a predictive model training stage 410 in which models are built and trained; an evaluation stage 415 in which the performance of the trained models is evaluated; and a chatbot implementation stage 420 in which one or more chatbots are implemented. The predictive model training stage 410 builds and trains one or more predictive models 425a to 425n (where "n" is any natural number) to be used in the other stages (in this specification, they may be referred to individually as predictive models 425 or collectively as predictive models 425). For example, predictive models 425 may include a model for recognizing one or more entities in an utterance, another model for determining the likelihood that the utterance represents a task configured to be performed by a particular skillbot, another model for predicting intent from an utterance for a first type of skillbot, and another model for predicting intent from an utterance for a second type of skillbot. Further types of predictive models may be implemented in other examples relating to this disclosure.

[0101] The predictive model 425 may be a machine learning (ML) model such as a convolutional neural network ("CNN") (e.g., an inception neural network, a residual neural network ("Resnet")), or a recurrent neural network (e.g., a long short-term memory ("LSTM") model or a gated recurrent unit ("GRU") model), or another variation of a deep neural network ("DNN") (e.g., a multi-label n binary DNN classifier or a multi-class DNN classifier for single-intent classification). Furthermore, the predictive model 425 may be a naive Bayes classifier, a linear classifier, a support vector machine, a bagging model such as a random forest model, a boosting model, a shallow neural network, or a combination of one or more such techniques (for example, other suitable ML models trained for natural language processing, such as a CNN-HMM or a Multi-Scale Convolutional Neural Network (MCNN). The chatbot system 400 may use the same type of predictive model or a different type of predictive model to recognize one or more entities in an utterance, determine whether the utterance is likely to represent a task configured to be performed by a particular skillbot, predict an intent from the utterance for a first type of skillbot and for a second type of skillbot. Yet another type of predictive model may be realized in other examples relating to this disclosure.

[0102] To train various predictive models 425, the training phase 410 consists of three main components: dataset preparation 430, template engineering 435, and model training 440. Dataset preparation 430 includes the process of loading the data asset 445, splitting the data asset 445 into training and validation sets 445a-n so that the system can train and test the predictive models 425, and performing basic preprocessing. Splitting the data asset 445 into training and validation sets 445a-n may be done randomly (e.g., 90 / 10% or 70 / 30%), or this split may be done according to more complex validation techniques such as K-Fold Cross-Validation, Leave-one-out Cross-Validation, Leave-one-group-out Cross-Validation, or Nested Cross-Validation to minimize sampling bias and overfitting.

[0103] The training data 445a may include a subset of at least one utterance from utterance examples associated with one or more skillbots. As shown above, utterances can be provided in various forms, including audio or text. Utterances may be sentence fragments, complete sentences, and multiple sentences, etc. For example, if an utterance is provided as audio, the data preparation 430 may convert the audio to text using a speech-to-text converter (not shown) that inserts punctuation marks, such as commas, semicolons, periods, etc., into the resulting text. In some cases, utterance examples are provided by a client or customer. In other cases, utterance examples are automatically generated from a pre-existing library of utterances (for example, identifying utterances specific to the skill that the chatbot is designated to learn from the library). The training data 445a for the predictive model 425 may include input text or audio (or input features of text or audio frames) and labels 450 corresponding to the input text or audio (or input features) as a matrix or table of values. For example, for each training utterance, a representation of the correct entity and its classification that should be inferred by the predictive model 425 may be provided as ground truth information for label 450. The behavior of the predictive model 425 can then be adapted (for example, by backpropagation) to minimize the difference between the inferences generated for various entities and the ground truth information.

[0104] Template engineering 435 involves the process of augmenting the training data 445a to include artificial utterances with a diverse set of any number of linguistic and / or text-format entities in order to make the predictive model 425 more robust to utterances in various languages ​​and / or text formats. By augmenting the training data 445a with artificial utterances with a diverse set of any number of linguistic and / or text-format entities, the predictive model 425 can better recognize entities in various languages ​​and text formats. As will be explained in detail with respect to Figures 5A to 5C, augmentation generally involves several operations: (i) extracting templates from the original training dataset and ensuring that the augmented training data has a similar distribution of predictor variables to the original training dataset; and (ii) generating augmented training data that includes artificial utterances with a diverse set of entities in any number of languages ​​and formats using a slot mapping method based on a custom syntax. By introducing entity-level data augmentation techniques, the predictive model 425 can better recognize entities in any number of languages ​​and formats and therefore performs better to utterances in various languages ​​and / or text formats. Furthermore, entity-level data augmentation is applied automatically in a synthesis-independent manner, so customers or clients do not need to worry about manually adding utterances in various languages ​​and / or text formats to the training data.

[0105] The training process for Model 425 involves selecting hyperparameters for Model 425 and performing iterative operations to input utterances from a subset of the augmented training data 445a into Model 425 to find a set of model parameters (e.g., weights and / or biases) that maximizes or minimizes the objective function of Model 425, for example, minimizing the loss function. Hyperparameters are settings that can be tuned or optimized to control the behavior of Model 425. Most models explicitly define hyperparameters that control various aspects of the model, such as memory or execution cost. However, additional hyperparameters may be defined to adapt the model to a particular scenario. For example, hyperparameters may include the number of hidden units or hidden layers of the model, the learning rate of the model, the convolutional kernel width, or the number of parameters of the model. Each iteration of training may involve finding a set of model parameters for Model 425 (consisting of a defined set of hyperparameters) such that the value of the objective function using a set of model parameters is smaller than the value of the objective function using a different set of model parameters in a previous iteration. The objective function can be configured to measure the difference between the output inferred using model 425 and the ground truth annotated with labels 150 on a subset of the augmented training data 445a.

[0106] Once the model parameter set is identified, model 425 is trained and can be tested or validated using a subset of the test data 445b (test dataset or validation dataset). The testing or validation process involves iterative operations inputting utterances from a subset of test data 445b into model 425 using validation techniques such as K-Fold Cross-Validation, Leave-one-out Cross-Validation, Leave-one-group-out Cross-Validation, or Nested Cross-Validation to tune the hyperparameters and ultimately find the optimal set of hyperparameters. Once the optimal set of hyperparameters is obtained, a reserved test set from a subset of test data 445a can be input into model 425 to obtain an output (one or more recognized entities in this example), which is evaluated against ground truth entities using correlation techniques such as Bland-Altman and Spearman's rank correlation coefficient. Furthermore, performance metrics 450, such as error, accuracy, precision, repeatability, and receiver operating characteristic curve (ROC), may be calculated in the entity recognition evaluation stage 415. As will be explained in detail with respect to Figures 5A to 5C, metrics 450 may be used in the entity recognition evaluation stage 415 to analyze the performance of the model 425 for recognizing entities.

[0107] The model training stage 410 outputs a trained model that includes one or more trained entity recognition models 455. The one or more trained entity recognition models 455 may be deployed and used in the chatbot implementation stage 420 to implement one or more chatbots. For example, one or more chatbots may consist of one or more trained entity recognition models 455 to receive text data 460 from one or more users and recognize and extract entities 465 from the various utterances received by one or more chatbots. Entity 465 may be part of the extracted information obtained from the text data 460 (e.g., extracted information 205; 304 described in Figures 2 and 3, respectively) and may be used in downstream processing such as intent classification.

[0108] Entity Data Augmentation Technology Figure 5A is a flowchart of process 500 for overall tuning of a predictive model using entity data augmentation. The processes shown in Figure 5 may be implemented by software (e.g., code, instructions, programs) executed by one or more processing units (e.g., processors, cores) of each system, hardware, or combination thereof. The software may be stored in a non-temporary storage medium (e.g., a memory device). The methods shown in Figure 5 and described below are intended to be illustrative and non-limiting. Figure 5 shows various processing steps occurring in a particular sequence or order, but this is not intended to be limiting. In certain alternative embodiments, the steps may be executed in some different order, or some steps may be executed in parallel. In certain embodiments, such as the embodiments shown in Figures 1 to 4, the processes shown in Figure 5 may be executed by a preprocessing subsystem (e.g., preprocessing subsystem 210 or predictive model training stage 410) to create, train, update, and deploy one or more predictive models (implemented as name entity recognizers) using the augmented dataset.

[0109] In step 505, a data processing system (for example, the chatbot system 400 described in relation to Figure 4) acquires a dataset containing utterances. The dataset includes realistic task-oriented dialogue data (consisting of utterances), which may include question-and-answer datasets (e.g., Wikipedia articles, manually generated factual questions therefrom, and manually generated answers to these questions), customer support datasets (e.g., two-person conversations extracted from chat logs used to get technical support for various issues), dialogue datasets (e.g., a collection of fictional conversations with dialogue between pairs of characters in a movie, extracted from a raw movie script), multilingual datasets (e.g., customer feedback where the reasons for dissatisfaction with a given company, product, or service are described in various languages), and so on.

[0110] In step 510, the dataset is split into a training dataset and a test dataset and / or validation dataset (an evaluation dataset set as the target for one or more predictive models). While the following process describes a single training dataset and a single test dataset and / or validation dataset, it should be understood that a dataset can be split into one or more training datasets and one or more test datasets and / or validation datasets and processed in a similar manner. Splitting the data asset into training and validation sets may be done randomly (e.g., 90 / 10% or 70 / 30%), or this split may be performed according to more complex validation techniques such as K-Fold Cross-Validation, Leave-one-out Cross-Validation, Leave-one-group-out Cross-Validation, or Nested Cross-Validation to minimize sampling bias and overfitting. The training dataset may be acquired pre-labeled or labeled after acquisition. Labeling may involve pre-defining a list of entity classes, identifying the location of entities within each utterance, and labeling entities with a class from a default list of classes. For example, entity classes may be pre-defined as PRODCUT, COMPANY, CURRENCY, and TIME, and the index of the utterance in which the entities fall may be identified, and labels may be created for the entity's class within the indexed location (for example, if entities such as Oracle, which fall under the COMPANY category, are located between indices 9 and 17, and entities such as 10 / 25 / 20, which fall under the TIME category, are located between indices 35 and 45, the utterance may be labeled (entities (9,17, "COMPANY"), (35,45, "TIME")).

[0111] In step 515, the predictive model is trained on the training dataset to learn parameters for recognizing and classifying entities in utterances. The training process may include selecting hyperparameters for the model and performing iterative operations to input utterances from the training dataset into the model to find a set of model parameters (e.g., weights and / or biases) that maximizes or minimizes the model's objective function, for example, minimizing the loss function. Each iteration of training may include finding a set of model parameters (consisting of a defined set of hyperparameters) for the model such that the value of the objective function using a set of model parameters is less than the value of the objective function using a different set of model parameters in a previous iteration. The objective function may be configured to measure the difference between the output inferred using the model and the ground truth annotated with labels on the training dataset.

[0112] In step 520, one or more predictive models are tested and / or validated on a test dataset and / or validation dataset to recognize one or more entities in an utterance. The testing or validation process involves iterative operations of inputting utterances from the test dataset and / or validation dataset into the model using validation techniques such as K-Fold Cross-Validation, Leave-one-out Cross-Validation, Leave-one-group-out Cross-Validation, or Nested Cross-Validation to tune the hyperparameters and ultimately find the optimal set of hyperparameters. Once the optimal set of hyperparameters is obtained, the model can be input with a reserved test set from the test dataset and / or validation dataset to obtain an output (e.g., one or more recognized entities), which is evaluated against ground truth entities using correlation techniques such as Bland-Altman and Spearman's rank correlation coefficient. Furthermore, performance metrics such as error (e.g., mean absolute error or mean squared error), accuracy (e.g., classification accuracy), precision (e.g., F1 score), repeatability (e.g., F1 score), and receiver operating characteristic (ROC) curve (e.g., AUC (Area Under Curve)-ROC curve) may be calculated. These metrics may be used to analyze the model's performance in recognizing entities. For example, evaluation tools may be used to identify false positives and false negatives in a reserved test set to assess the model's behavior. Tests and / or validations are performed and analyzed to obtain a baseline assessment of the model, and to gain an understanding of additional information (e.g., additional nouns, text formats, currency types, URL types, etc.) that may need to be added or diversified in the training dataset to improve the model's performance in recognizing entities in utterances in various languages ​​and / or text formats.For example, testing and / or validation may show that the model does not perform well when attempting to recognize and classify entity CURRENCY (particularly in the case of utterances without the currency US dollar), and therefore, additional information (e.g., additional currency types) needs to be added or diversified in the training dataset to improve the model's performance with entity CURRENCY.

[0113] In cases where additional information needs to be added or diversified in the training dataset to improve the model's performance, in step 525, as detailed with respect to Figure 5B, one or more templates are created or extracted from the original corpus (the dataset obtained in step 505 before splitting) or a subset thereof, such as the split training dataset. In step 530, as detailed with respect to Figure 5B, the data processing system creates artificial utterances using one or more templates. In step 535, the dataset or a subset thereof is augmented with artificial utterances by the data processing system to obtain an augmented dataset. The percentage (augmentation rate) for augmenting the dataset or a subset thereof with artificial utterances may be set based on one or more factors. In some cases, the augmentation rate is selected based on the analysis of test and validation data. For example, the augmentation rate is selected based on the ability to be recognized / determined in order to improve the performance of the named entity recognition model. In other cases, the augmentation rate may be selected randomly and then optimized. In other cases, the augmentation rate is selected based on a default schema to prevent regression of the ability of one or more predictive models to predict entities from one or more prior observation classes. For example, the scaling rate may be limited by the schema(s), namely CURRENCY:3*|the number of currency utterances in the original set of training data| and / or TIME_DATE:0.75*|the number of time-date utterances in the original set of training data|.

[0114] In step 540, the augmented dataset is split into an augmented training dataset and an augmented test dataset and / or an augmented validation dataset (an evaluation dataset set as the target for one or more predictive models). While the following process describes a single training dataset and a single test dataset and / or validation dataset, it should be understood that an augmented dataset can be split into one or more augmented training datasets and one or more augmented test datasets and / or validation datasets and processed in a similar manner. Splitting the augmented data asset into training and validation sets may be done randomly (e.g., 90 / 10% or 70 / 30%), or this split may be performed according to more complex validation techniques such as K-Fold Cross-Validation, Leave-one-out Cross-Validation, Leave-one-group-out Cross-Validation, or Nested Cross-Validation to minimize sampling bias and overfitting. The augmented training dataset is labeled as described in step 510, based on a default list of entity classes.

[0115] In step 545, the model is trained using the augmented training dataset (or a new base model is trained). The training process may involve selecting the model's hyperparameters and performing iterative operations to input utterances from the augmented training dataset into the model to find a set of model parameters (e.g., weights and / or biases) that maximizes or minimizes the model's objective function, for example, minimizing the loss function. Each iteration of training may involve finding a set of model parameters (consisting of a defined set of hyperparameters) for the model such that the value of the objective function using that set of model parameters is less than the value of the objective function using a different set of model parameters in the previous iteration. The objective function can be constructed to measure the difference between the output inferred using the model and the ground truth annotated with labels on the augmented training dataset.

[0116] In step 550, the model is tested and / or validated on an augmented test dataset and / or an augmented validation dataset to recognize one or more entities in an utterance. The testing or validation process involves iterative operations of inputting utterances from the augmented test dataset and / or an augmented validation dataset into the model using validation techniques such as K-Fold Cross-Validation, Leave-one-out Cross-Validation, Leave-one-group-out Cross-Validation, or Nested Cross-Validation to tune the hyperparameters and ultimately find the optimal set of hyperparameters. Once the optimal set of hyperparameters is obtained, the reserved test set from the test dataset and / or validation dataset can be input into the model to obtain an output (e.g., one or more recognized entities), which is evaluated against ground truth entities using correlation techniques such as Bland-Altman and Spearman's rank correlation coefficient. Furthermore, performance metrics such as error (e.g., mean absolute error or mean squared error), accuracy (e.g., classification accuracy), precision (e.g., F1 score), repeatability (e.g., F1 score), and receiver operating characteristic (ROC) curve (e.g., AUC (Area Under Curve)-ROC curve) may be calculated. These metrics may be used to analyze the model's performance in recognizing entities. They may also be used to analyze the model's performance in comparison to the baseline evaluation of the model in step 520. For example, evaluation tools may be used to identify false positives and false negatives in a reserved test set to evaluate the model's behavior. Differentiation tools may be used to identify differences between the new model and previous versions of the model, and / or between the new model and the base model. Regression tools may be used to verify that there are no regressions between two model releases and / or between the new model and the base model.

[0117] Subsequently, based on the evaluation of model training and testing, steps 525–550 may be repeated until the evaluation demonstrates the optimal performance of one or more predictive models for recognizing one or more entities in utterances in various languages ​​and / or text formats. Optimal model performance may be defined by the user or system as achieving one or more performance metrics (e.g., x% accuracy, x error score achieved, x% increase in accuracy over baseline evaluation or previous versions of the model, at least x decrease in the error score for a given entity class, at least x improvement over baseline evaluation in recognizing and classifying a given entity, etc.). Once trained and / or optimal performance is achieved, the model is deployed in step 555 within a chatbot system for recognizing one or more entities in utterances in various languages ​​and / or text formats (e.g., as part of a skillbot). For example, an utterance may be received by a chatbot, and the utterance may be analyzed by one or more predictive models to determine whether the utterance contains one or more entities, and the recognized entities may be used in a downstream process by the chatbot, for example, to infer the intent of the utterance.

[0118] Figure 5B is a flowchart of process 560 for creating or extracting one or more templates from one or more sets of training data, and for creating artificial utterances based on one or more templates, as performed in process 500 described with respect to Figure 5A. The processes shown in Figure 5B may be implemented in software (e.g., code, instructions, programs) executed by one or more processing units (e.g., processors, cores) of each system, hardware, or combination thereof. The software may be stored in a non-temporary storage medium (e.g., a memory device). The methods shown in Figure 5B and described below are intended to be illustrative and non-limiting. Figure 5B shows various processing steps occurring in a particular sequence or order, but this is not intended to be limiting. In certain alternative embodiments, the steps may be performed in some different order, or some steps may be performed in parallel. In certain embodiments, such as the embodiments shown in Figures 1 to 4, the processes shown in Figure 5B may be performed by a data processing system including a template engine configured to automatically generate templates and utterances for use in training a predictive model. The template engine automatically generates a sufficient amount of augmented data, ensuring that this augmented data is not only contextually similar to the training data already used within the training system, but also that the model does not learn specific sentence patterns. This is advantageous over both crowdsourcing data (which provides quantity but lacks contextual similarity) and manually adding utterances (which provides similar context but cannot manually generate a sufficient number of utterances to ensure data diversity).

[0119] The following discussion of Process 560 describes the data augmentation process (fitting the entire Process 500) by giving CURRENCY as an example of an entity identified (e.g., through the analysis performed in steps 520 and 550) by the user or computing system (e.g., through the analysis performed in steps 520 and 550) that may benefit from data augmentation. While CURRENCY is used as an example entity, it should be understood that the processing described herein may also be performed on other classes of entities and combinations thereof. In step 565, a list of values ​​to be covered for a given entity is generated or identified. For example, a data processing system (e.g., the chatbot system 400 described with respect to Figure 4) generates or identifies a list of units of the CURRENCY entity to be covered. A geographical dictionary list (geographical dictionary or directory) may be used to generate a list of currency types categorized by country, language, or region, by well-known nouns, e.g., currency and its synonyms, large and small units (e.g., dollars and cents), and by standard text format, e.g., currency numeric format. The list would look like this:

[0120] List of currencies to cover SGD$ Singapore Dollar (singular) Singapore Dollars (plural) Sing SGD Rs. Rs Australian dollar (singular) Australian Dollars (plural) Thai Baht R$ EGP SDG LBP ARS COP PHP CLP DOP CUP UYU ALL In step 570, utterances are selected from a dataset that has context for a given entity (for example, the original corpus (the dataset obtained in step 505 before splitting) or a subset thereof such as the training dataset after splitting). For example, a number of utterances from a dataset with the correct context for the CURRENCY entity may be selected manually or by a data processing system. An example utterance that could be selected for the CURRENCY entity is "He paid $100.78 for dinner," but it should be understood that there are many other example utterances with the CURRENCY entity that can be selected alternatively or additionally.

[0121] In step 575, the selected utterances are converted into templates, each template containing slots that map to a list of values ​​for a given entity. For example, a data processing system executing a script templates all selected utterances. More specifically, the script converts the selected utterances into a list of templates. A template is essentially a sentence with slots. Templates are stored as a list associated with a given entity and optionally with the context of that given entity, and templates from the list are selected and populated with different values ​​for the entity by the template engine, as will be described in detail later with respect to Figure 5C. Templates can be defined with custom syntax to customize the model's behavior for recognizing defined classes of entities within an utterance. For example, a context-free grammar (a set of recursive rules used to generate patterns of strings) defined by the syntax of a programming language such as the metalanguage BNF (Backus-Naur Form) can be used to model complex entities such as currency, which has multiple components in it (e.g., symbols, codes, large units, small units, etc.) and supports nested slot mapping. An example template for the selected utterance "He paid $100.78 for dinner" is "He paid / <CURRENCY_slot _augmented> It could be {CURRENCY} / for dinner. As another example, the template for the selected utterance "An expense of 10 AUD$ was incurred on May 21" is Create an (expense|exp) for / (<digit_amount> [ ]<cur_type> |<cur_type> [ ]<digit_amount> ){CURRENCY} / [ (on|at) / <date>It could be {TIME_DATE} / ]. Each slot contains one or more elements that map to one or more values ​​(for example, a list of potential values ​​or operations to be performed). For example, continuing the above example, the slot / <CURRENCY_slot _augmented>{CURRENCY} / is an element<cur_type> and elements<digit_amount> Maps to the element.<cur_type> While it is a value list type element,<digit_amount> This is a functional element.<cur_type> and elements<digit_amount> Each of these maps to a different value. For example,<cur_type> This maps to the unit list of the CURRENCY entity,<digit_amount> This maps to a function for generating the decimal value of the CURRENCY entity.

[0122] In step 580, a template is selected from a list of templates, values ​​are selected from a list of values ​​based on the mapping between slots in the selected template and a given list of entity values, and an artificial utterance is created based on the selected template and selected values. Creating an artificial utterance involves inserting the selected values ​​into the slots of the selected template that map to the list of values ​​for a given entity. For example, the template engine is run with a list of templates (e.g., a list of templates containing the CURRECNY entity) and a list of values ​​for the entity to be covered (e.g., CURRENCY) (e.g., units) to generate an artificial utterance. There is a template engine configuration file where the user can specify which slots map to entity files, and the template engine will generate artificial utterances according to that mapping. Multiple templates and / or values ​​can be selected from a list, and multiple artificial utterances can be generated by repeatedly running the template engine with a list of templates and a list of entity values ​​to be covered (based on what expansion rate is selected or set).

[0123] The flowchart in Figure 5C is based on the template "He paid / <CURRENCY_slot _augmented> This figure shows a subprocess for generating artificial utterances (explained in 530;580 for Figures 5A and 5B, respectively) for the example "{CURRENCY} / for dinner". In step (A), based on the analysis of the test and validation data (for example, the analysis performed in steps 520;550 for Figure 5A), it is determined that the entity CURRENCY needs to be augmented to improve the performance of the named entity recognition model. Furthermore, an augmentation rate is set for the process of creating artificial utterances to augment the training data. In some cases, the augmentation rate is selected based on the analysis of the test and validation data. For example, the augmentation rate is selected based on the ability to recognize / determine in order to improve the performance of the named entity recognition model. In other cases, the augmentation rate is selected randomly and then may be optimized. In other cases, the augmentation rate is selected based on a default schema to prevent regression of the ability of one or more prediction models to predict entities from one or more prior observation classes.

[0124] In step (B), a template is selected from a list of templates associated with the determined entity CURRENCY and optionally associated with the context of a given entity (this list of templates may be generated according to step 575 as described with respect to Figure 5B). Once created, the list of templates is associated with an entity and optionally associated with the context of a given entity (for example, a mapping is created between the list and the entity in a file or database). For example, there may be a template engine configuration file where the user can specify which template maps to which entity file, and the template engine will identify the list of templates associated with the determined entity CURRENCY according to that mapping. Then the template (for example, He paid / <CURRENCY_slot _augmented> {CURRENCY} / for dinner) is selected in a controlled manner (i.e., based on a predefined selection protocol such as sequential selection or the use of a fitness function) or from a randomly identified list of templates. In some cases, template creation and extraction are performed to ensure that the augmented training dataset has the same distribution as the original dataset used to train the model (e.g., from the original corpus (the dataset obtained in step 505 before splitting) or a subset of it such as the training dataset after splitting). For example, heuristics may be used regarding how to select templates. Heuristics may include conditions such as (i) the choice between a template with multiple entity classes or a template with a single entity class, (ii) a trade-off between template diversity and template quality, (iii) categorization of templates depending on some context, such as those relating to account balances, (iv) whether the model requires a minimum number of occurrences per template / specific context that the user wants to teach the model, or (v) any combination of these.

[0125] In step (C), elements of one or more slots within the selected template are identified. Slots may also be identified by parsing the template and locating the text that defines the slot based on the custom syntax used to generate the template. For example, any text between / ****** / may be identified as a slot, any text containing an identifier such as _slot_ may be identified as a slot, or any combination thereof. Once created, slots are associated with one or more elements (for example, a mapping is created between slots and elements in a file or database). For example, there may be a configuration file for a template engine where the user can specify which elements map to which slots, and the template engine will identify the elements associated with the identified slots according to the mapping. As shown in the figure, slots / <CURRENCY_slot _augmented> {CURRENCY} / is template He paid / <CURRENCY_slot _augmented> {CURRENCY} / identified from for dinner, slot / <CURRENCY_slot _augmented> {CURRENCY} / is an element<cur_type> and elements<digit_amount> Maps to the element.<cur_type> While it is a value list type element,<digit_amount> This is a functional element.

[0126] In steps (D) and (E), the values ​​of the elements are identified. Once a slot element is created, it is associated with a value (for example, a mapping is created between elements and values ​​in a file or database). For example, there may be a configuration file for a template engine where the user can specify which values ​​map to which elements, and this mapping can be used by the template engine to identify the values ​​associated with the identified elements. As illustrated, the elements<cur_type> This maps to a list of various values ​​for the entity CURRENCY, and then maps to a function that generates a random decimal number as the value for the entity CURRENCY.

[0127] In step (F), one or more operations are performed according to the identified values, and the results of the operations are combined to generate a slot value for each identified slot. For example, for a value list type element, a value (e.g., SGD$) is selected from a list of various identified values ​​of entity CURRENCY (i.e., sequentially or based on a default selection protocol such as the use of a fitness function), or randomly. For a function type element, the function is performed to obtain the output of the function. For example, a random decimal number may be performed to obtain 20.45. The value of the value list type element SGD$ and the value of the function type element are combined to generate the slot value SGD$20.45. Naturally, the combination of values ​​can be more complex than a simple sequential arrangement of values. For example, in some cases, the values ​​may be combined using separate functions or random order generators.

[0128] In step (G), the slot values ​​for each identified slot are then entered into the associated slots within the identified template to generate artificial utterance. For example, SGD$20.45 is entered into the template He paid / <CURRENCY_slot _augmented> {CURRENCY} / for dinner slot / <CURRENCY_slot _augmented> Entering {CURRENCY} / generates the artificial utterance "He paid SGD$20.45 for dinner." Multiple templates and / or values ​​can be selected from a list, and multiple artificial utterances can be generated by iteratively running the template engine with the list of templates and the list of entity values ​​to cover (based on what expansion rate is selected or set).

[0129] While the template engine scales well, this is a simple example with one template; in reality, many different templates may be provided by scripts, potentially providing a longer hierarchy of yet another template, file, or function that the template engine can map to. Table 2 shows several examples where nested sets of slot mappings are used in templates to customize data extensions for specific entities, CURRENCY and TIME_DATE.

[0130] [Table 2-1]

[0131] [Table 2-2]

[0132] [Table 2-3]

[0133] [Table 2-4]

[0134] Exemplary System Figure 6 is a simplified diagram of a distributed system 600. In the illustrated example, the distributed system 600 includes one or more client computing devices 602, 604, 606, and 608, which are connected to a server 612 via one or more communication networks 610. The client computing devices 602, 604, 606, and 608 may be configured to run one or more applications.

[0135] In various examples, server 612 may be adapted to run one or more services or software applications that enable one or more embodiments described in this disclosure. In certain examples, server 612 may also provide other services or software applications that may include non-virtual and virtual environments. In some examples, these services may be provided to users of client computing devices 602, 604, 606 and / or 608 as web-based services or cloud services, such as under a Software as a Service (SaaS) model. Users operating client computing devices 602, 604, 606 and / or 608 can then interact with server 612 using one or more client applications to access the services provided by these components.

[0136] In the configuration shown in Figure 6, server 612 may include one or more components 618, 620, and 622 that implement the functions performed by server 612. These components may include software components, hardware components, or a combination thereof that can be executed by one or more processors. It should be understood that various different system configurations are possible, which may differ from the distributed system 600. Therefore, the example shown in Figure 6 is an example of a distributed system for implementing an exemplary system and is not intended to be limiting.

[0137] A user may use client computing devices 602, 604, 606, and / or 608 to run one or more applications, models, or chatbots that may generate one or more events or models that may be run or supplied in accordance with the teachings of this disclosure. A client device may provide an interface that allows a user of the client device to interact with the client device. The client device may also output information to the user through this interface. Although Figure 6 shows only four client computing devices, any number of client computing devices may be supported.

[0138] Client devices may include various types of computing systems, such as portable handheld devices, general-purpose computers like personal computers and laptops, workstation computers, wearable devices, game systems, thin clients, various messaging devices, and sensing devices like sensors. These computing devices may run various types and versions of software applications and operating systems (e.g., Microsoft Windows®, Apple Macintosh®, UNIX® or UNIX-like OSs, Linux® or Google Chrome® OS, etc.), including various mobile operating systems (e.g., Microsoft Windows®, iOS®, Windows Phone®, Android®, BlackBerry®, Palm OS, etc.). Mobile terminals may include mobile phones, smartphones (e.g., iPhone®), tablets (e.g., iPad®), and personal digital assistants (PDAs). Wearable devices may include Google Glass® head-mounted displays, etc. The game system may include various portable game devices, internet-enabled game devices (for example, Microsoft Xbox® game consoles with or without Kinect® gesture input, Sony PlayStation® systems, various game systems provided by Nintendo®, and others). The client device can run various different applications, such as various internet-related applications and communication applications (for example, email applications, short message service (SMS) applications), and may use various communication protocols.

[0139] Network(s) 610 may be any type of network known to those skilled in the art that can support data communication using any of the various available protocols, including but not limited to TCP / IP (Transmission Control Protocol / Internet Protocol), SNA (System Network Architecture), IPX (Internet Packet Switching), AppleTalk®, etc. For example, Network(s) 610 may be a local area network (LAN), Ethernet®, a Token Ring-based network, a wide area network (WAN), the Internet, a virtual network, a virtual private network (VPN), an intranet, an extranet, a public switched telephone network (PSTN), an infrared network, a wireless network (for example, a wireless network operating on any of the Institute of Electrical and Electronics (IEEE) 1002.11 suite of protocols, Bluetooth®, and / or other wireless protocols) and / or any combination of these and other networks.

[0140] Server 612 may consist of one or more general-purpose computers, dedicated server computers (including, for example, PC (personal computer) servers, UNIX® servers, midrange servers, mainframe computers, rack-mount servers), server farms, server clusters, or other appropriate configurations and / or combinations. Server 612 may include other computing architectures, including virtualization, such as one or more virtual machines running a virtual operating system, or one or more flexible pools of logical storage devices that can be virtualized to maintain virtual storage devices for the server. In various examples, Server 712 may be adapted to run one or more services or software applications that provide the functionality described in the foregoing disclosure.

[0141] The computing system of Server 612 may run one or more operating systems, including any of the above and any commercially available server operating systems. Server 612 may also run any of a variety of additional server applications and / or mid-tier applications, including HTTP (Hypertext Transfer Protocol) servers, FTP (File Transfer Protocol) servers, CGI (Common Gateway Interface) servers, Java® servers, database servers, etc. Exemplary database servers include, but are not limited to, those commercially available from Oracle®, Microsoft®, Sybase®, IBM® (International Business Machines), and others.

[0142] In some implementations, server 612 may include one or more applications for analyzing and integrating data feeds and / or event update information received from users of client computing devices 602, 604, 606, and 608. For example, data feeds and / or event update information may include, but are not limited to, real-time updates and continuous data streams received from Twitter® feeds, Facebook® updates, or one or more third-party sources, including real-time events related to sensor data applications, financial tickers, network performance measurement tools (e.g., network monitoring and traffic management applications), clickstream analysis tools, automotive traffic monitoring, etc. Server 612 may also include one or more applications for displaying data feeds and / or real-time events via one or more display devices of client computing devices 602, 604, 606, and 608.

[0143] The distributed system 600 may also include one or more data repositories 614,616. These data repositories may be used to store data and other information in certain examples. For example, one or more of the data repositories 614,616 may be used to store information such as information about chatbot performance or generated models for use by a chatbot used by server 612 when performing various functions according to different embodiments. The data repositories 614,616 may reside in different locations. For example, a data repository used by server 612 may be local to server 612, or it may be remote from server 612, communicating with server 612 via a network-based connection or a dedicated connection. The data repositories 614,616 may be of different types. In certain examples, a data repository used by server 612 may be a relational database, such as a database provided by Oracle Corporation® and other vendors. One or more of these databases may be adapted to allow the storage, updating, and retrieval of data to and from the database in response to SQL-format commands.

[0144] In a particular example, one or more of the data repositories 614,616 may also be used by an application to store application data. The data repositories used by the application may be of different types, such as a key-value store repository, an object store repository, or a general storage repository supported by the file system.

[0145] In a particular example, the functions described herein may be provided as services through a cloud environment. Figure 7 is a simplified block diagram of a cloud-based system environment in a particular example in which various services may be provided as cloud services. In the example shown in Figure 7, the cloud infrastructure system 702 may provide one or more cloud services that may be requested by a user using one or more client computing devices 704, 706, and 708. The cloud infrastructure system 702 may comprise one or more computers and / or servers, which may include the above-described server 712. The computers in the cloud infrastructure system 702 may be organized as general-purpose computers, dedicated server computers, server farms, server clusters, or other appropriate configurations and / or combinations.

[0146] Network 710 may facilitate the transmission and exchange of data between clients 704, 706, and 708 and the cloud infrastructure system 702. Network 710 may include one or more networks. These networks may be of the same type or different types. Network 710 may support one or more communication protocols, including wired and / or wireless protocols, to facilitate communication.

[0147] The example shown in Figure 7 is merely one example of a cloud infrastructure system and is not intended to be limiting. It should be understood that in several other examples, the cloud infrastructure system 702 may have more or fewer components than shown in Figure 7, may combine two or more components, or may have different configurations or arrangements of components. For example, while Figure 7 shows three client computing devices, any number of client computing devices may be supported in alternative examples.

[0148] The term "cloud service" is generally used to refer to services provided to users on demand via communication networks such as the internet, through a service provider's system (e.g., cloud infrastructure system 702). Generally, in a public cloud environment, the servers and systems that make up the cloud service provider's system are different from the customer's own on-premises servers or systems. The cloud service provider's system is managed by the cloud service provider. Therefore, customers can use the cloud services provided by the cloud service provider without separately purchasing licenses, support, hardware, and software resources. For example, the cloud service provider's system hosts applications, and users can order and use those applications as needed via the internet without purchasing the infrastructure resources to run them. Cloud services are designed to provide easy and scalable access to applications, resources, and services. Several providers offer cloud services. For example, some cloud services, such as middleware services, database services, and Java® cloud services, are provided by Oracle Corporation® in Redwood Shores, California.

[0149] In a particular example, the cloud infrastructure system 702 may provide one or more cloud services using different models, including a hybrid service model, a SaaS (Software as a Service) model, a PaaS (Platform as a Service) model, an IaaS (Infrastructure as a Service) model, and other models. The cloud infrastructure system 702 may include a set of applications, middleware, databases, and other resources that enable the delivery of various cloud services.

[0150] The SaaS model allows customers to receive applications or software as a service over a communication network such as the internet, without having to purchase hardware or software for the basic application. For example, the SaaS model can be used to provide customers with access to on-demand applications hosted by a cloud infrastructure system. Examples of SaaS services offered by Oracle Corporation® include, but are not limited to, a variety of services such as human resources / capital management, customer relationship management (CRM), enterprise resource planning (ERP), supply chain management (SCM), enterprise performance management (EPM), analytics services, and social applications.

[0151] The IaaS model is generally used to provide infrastructure resources (e.g., servers, storage, hardware, and network resources) to customers as a cloud service, offering resilient computing and storage capabilities. Various IaaS services are offered by Oracle Corporation®.

[0152] The PaaS model is generally used to provide a platform and environment resource as a service, enabling customers to develop, run, and manage applications and services without having to procure, build, or maintain such resources themselves. Examples of PaaS services offered by Oracle Corporation® include, but are not limited to, Oracle Java Cloud Service (JCS), Oracle Database Cloud Service (DBCS), data management cloud services, and various application development solution services.

[0153] Cloud services are generally provided in an on-demand self-service-based, subscription-based, elastically scalable, reliable, highly available, and secure manner. For example, a customer may order one or more services provided by the cloud infrastructure system 702 through a subscription order. The cloud infrastructure system 702 then performs processing to provide the services requested in the customer's subscription order. For example, a user may use utterances to request the cloud infrastructure system to take a specific action (e.g., an intent) as described above, and / or to provide services to a chatbot system as described herein. The cloud infrastructure system 702 may be configured to provide one or more cloud services.

[0154] The cloud infrastructure system 702 can provide cloud services through various deployment models. In the public cloud model, the cloud infrastructure system 702 may be owned by a third-party cloud service provider, and the cloud services are provided to customers who may be individuals, businesses, or the general public. In certain other examples, under the private cloud model, the cloud infrastructure system 702 may be operated within an organization (e.g., within a corporate organization), and services are provided to customers within that organization. For example, customers may be various departments within the company, such as the human resources department or the payroll department, or individuals within that company. In certain other examples, under the community cloud model, the cloud infrastructure system 702 and the services provided may be shared by several organizations within the relevant community. Various other models, such as hybrids of the above models, may also be used.

[0155] The client computing devices 704, 706, and 708 may be of different types (such as the client computing devices 602, 604, 606, and 608 shown in Figure 6) and may be capable of running one or more client applications. Users may use the client devices to interact with the cloud infrastructure system 702, such as by requesting services provided by the cloud infrastructure system 702. For example, a user may use the client devices to request information or actions from a chatbot as described in this disclosure.

[0156] In some cases, the processing performed by the cloud infrastructure system 702 to provide a service may include training and deploying models. This analysis may include using, analyzing, and manipulating a dataset to train and deploy one or more models. This analysis may be performed by one or more processors that process the data in parallel or perform simulations using the data. For example, big data analysis may be performed by the cloud infrastructure system 702 to generate and train one or more models for a chatbot system. The data used for this analysis may include structured data (e.g., data stored in a database or data structured according to a structured model) and / or unstructured data (e.g., data blobs (binary large objects)).

[0157] As illustrated in the example in Figure 7, the cloud infrastructure system 702 may include infrastructure resources 730 that are used to facilitate the provision of various cloud services offered by the cloud infrastructure system 702. Infrastructure resources 730 may include, for example, processing resources, storage or memory resources, networking resources, etc. In a particular example, a storage virtual machine available to supply storage requested by an application may be part of the cloud infrastructure system 702. In other examples, the storage virtual machine may be part of a different system.

[0158] In a particular example, to facilitate the efficient provisioning of these resources to support various cloud services provided to various customers by the cloud infrastructure system 702, resources may be bundled into sets of resources or resource modules (also referred to as “pods”). Each resource module or pod may contain a pre-integrated and optimized combination of one or more types of resources. In a particular example, different pods may be pre-provisioned for different types of cloud services. For example, a first set of pods may be provisioned for a database service, and a second set of pods, which may contain a different combination of resources than the pods in the first set of pods, may be provisioned for a Java service, and so on. For some services, the resources allocated to provisioning the service may be shared among the services.

[0159] The cloud infrastructure system 702 itself may internally use services 732 shared by various components of the cloud infrastructure system 702, thereby facilitating the provisioning of services by the cloud infrastructure system 702. These internally shared services may include, but are not limited to, security and identity services, integration services, enterprise repository services, enterprise manager services, virus scanning and whitelisting services, high availability, backup and recovery services, services to enable cloud support, email services, notification services, and file transfer services.

[0160] The cloud infrastructure system 702 may consist of multiple subsystems. These subsystems may be implemented in software, hardware, or a combination thereof. As shown in Figure 7, these subsystems may include a user interface subsystem 712 that enables users or customers of the cloud infrastructure system 702 to interact with the cloud infrastructure system 702. The user interface subsystem 712 may include a variety of different interfaces, such as a web interface 714, an online store interface 716 where cloud services provided by the cloud infrastructure system 702 are advertised and available for purchase by consumers, and other interfaces 718. For example, a customer may use a client device to request one or more services provided by the cloud infrastructure system 702 using one or more of interfaces 714, 716, and 718 (service request 734). For example, a customer may access the online store, browse the cloud services provided by the cloud infrastructure system 702, and place a subscription order for one or more services provided by the cloud infrastructure system 702 that the customer wishes to subscribe to. This service request may include information identifying the customer and the one or more services the customer wishes to subscribe to. For example, a customer may place a subscription order for services provided by the cloud infrastructure system 702. As part of the order, the customer may provide information identifying the chatbot system on which the service is provided, and optionally, one or more credentials for said chatbot system.

[0161] In certain examples, such as the one shown in Figure 7, the cloud infrastructure system 702 may include an order management subsystem (OMS) 720 configured to process new orders. As part of this process, the OMS 720 may be configured to create customer accounts if not already done, receive invoice issuance and / or billing information from the customer used to issue invoices to the customer in order to provide the requested services to the customer, verify customer information, confirm the customer order upon verification, orchestrate various workflows, and prepare the order for provisioning.

[0162] If properly validated, the OMS720 may then invoke the order provisioning subsystem (OPS)724, which is configured to provide resources for the order, including processing, memory, and networking resources. Provisioning may involve allocating resources for the order and configuring these resources to facilitate the services requested by the customer order. The manner in which resources are provisioned for the order and the types of resources provisioned may depend on the type of cloud service ordered by the customer. For example, according to one workflow, the OPS724 may be configured to determine that a particular cloud service is requested and identify the number of pods that may be pre-configured for that particular cloud service. The number of pods allocated to the order may depend on the size / volume / level / scope of the requested service. For example, the number of pods allocated may depend on the number of users supported by the service, the duration for which the service is requested, etc. The allocated pods may then be customized to suit the specific request-issuing customer to provide the requested service.

[0163] In a specific example, the setup phase processing described above may be performed by the cloud infrastructure system 702 as part of the provisioning process. The cloud infrastructure system 702 may generate an application ID and select a storage virtual machine for the application from among the storage virtual machines provided by the cloud infrastructure system 702 itself, or from storage virtual machines provided by other systems other than the cloud infrastructure system 702.

[0164] The cloud infrastructure system 702 may send a response or notification 744 to the requesting customer to indicate when the requested service will be available. In some cases, information (e.g., a link) that enables the customer to begin using and utilizing the benefits of the requested service may be sent to the customer. In a particular example, when a customer requests a service, the response may include a chatbot system ID generated by the cloud infrastructure system 702 and information identifying the chatbot system selected by the cloud infrastructure system 702 that corresponds to this chatbot system ID.

[0165] The cloud infrastructure system 702 may provide services to multiple customers. For each customer, the cloud infrastructure system 702 is responsible for managing information related to one or more subscription orders received from the customer, maintaining customer data related to these orders, and providing the requested services to the customer. The cloud infrastructure system 702 may also collect usage statistics about the customer's use of the subscribed services. For example, statistics may be collected on the amount of storage used, the amount of data transferred, the number of users, and system uptime and system downtime. This usage information may be used to issue invoices to customers. Invoice issuance may occur, for example, on a monthly cycle.

[0166] The cloud infrastructure system 702 may provide services to multiple customers in parallel. The cloud infrastructure system 702 may store information about these customers, which may include proprietary information. In a particular example, the cloud infrastructure system 702 includes an identity management subsystem (IMS) 728 configured to manage customer information and isolate managed information so that information related to one customer is inaccessible to another customer. The IMS 728 may be configured to provide a variety of security-related services, such as identity services, including information access management, authentication and authorization services, and services for managing customer identity and roles and related functions.

[0167] Figure 8 shows an example of a computer system 800. In some examples, the computer system 800 may be used to implement a digital assistant or chatbot system in a distributed environment, as well as any of the various servers and computer systems described above. As shown in Figure 8, the computer system 800 includes various subsystems, including a processing subsystem 804, which communicates with several other subsystems via a bus subsystem 802. These other subsystems may include a processing acceleration unit 806, an I / O subsystem 808, a storage subsystem 818, and a communication subsystem 824. The storage subsystem 818 may include a non-temporary computer-readable storage medium, which includes a storage medium 822 and system memory 910.

[0168] The bus subsystem 802 provides a mechanism for various components and subsystems of the computer system 800 to communicate with each other as intended. Although the bus subsystem 802 is schematically shown as a single bus, alternative examples of the bus subsystem may utilize multiple buses. The bus subsystem 802 may be any of several types of bus structures, which include memory buses or memory controllers, peripheral buses, local buses, etc., using any of the various bus architectures. For example, such architectures may include the Industry Standard Architecture (ISA) bus, the Micro Channel Architecture (MCA) bus, the Enhanced ISA (EISA) bus, the Video Electronics Standards Association (VESA) local bus, and the Peripheral Component Interconnect (PCI) bus, which can be implemented as a mezzanine bus manufactured according to the IEEE P1386.1 standard.

[0169] The processing subsystem 804 controls the operation of the computer system 800 and may consist of one or more processors, application-specific integrated circuits (ASICs), or field-programmable gate arrays (FPGAs). These processors may include single-core or multi-core processors. The processing resources of the computer system 800 may be organized into one or more processing units 832, 834, etc. A processing unit may include one or more processors, one or more cores from the same or different processors, a combination of cores and processors, or other combinations of cores and processors. In some examples, the processing subsystem 804 may include one or more special-purpose coprocessors, such as graphics processors or digital signal processors (DSPs). In some examples, some or all of the processing units of the processing subsystem 804 may be implemented using customized circuits such as application-specific integrated circuits (ASICs) or field-programmable gate arrays (FPGAs).

[0170] In some examples, processing units within the processing subsystem 804 may execute instructions stored in system memory 810 or on computer-readable storage medium 822. In various examples, processing units may execute various programs or code instructions and may maintain multiple concurrently running programs or processes. At any time, some or all of the program code to be executed may reside in system memory 810 and / or on computer-readable storage medium 822 (and possibly on one or more storage devices). Through suitable programming, the processing subsystem 804 may provide the various functions described above. In cases where the computer system 800 is running one or more virtual machines, one or more processing units may be assigned to each virtual machine.

[0171] In a particular example, the processing acceleration unit 806 may be optionally provided to accelerate the entire process performed by the computer system 800, to perform customized processing, or to offload a portion of the processing performed by the processing subsystem 804.

[0172] The input / output subsystem 808 may include devices and mechanisms for inputting information into and / or outputting information from or through the computer system 800. Generally, the use of the term input device is intended to include all possible types of devices and mechanisms for inputting information into the computer system 800. User interface input devices may include, for example, pointing devices such as keyboards, mice or trackballs, touchpads or touchscreens integrated into displays, scroll wheels, click wheels, dials, buttons, switches, keypads, voice input devices with voice command recognition systems, microphones, and other types of input devices. User interface input devices may also include motion sensing and / or gesture recognition devices such as Microsoft Kinect® motion sensors, Microsoft Xbox® 360 game controllers, and devices that provide interfaces for receiving input using gestures and voice commands, enabling users to control and interact with input devices. User interface input devices may also include eye gesture recognition devices, such as the Google Glass® blink detector, which detects eye activity from the user (e.g., blinking while taking a picture and / or selecting from a menu) and translates eye gestures into input to an input device (e.g., Google Glass®). Furthermore, user interface input devices may include voice recognition sensing devices that enable the user to interact with a voice recognition system (e.g., Siri Navigator) through voice commands.

[0173] Other examples of user interface input devices include, but are not limited to, three-dimensional (3D) mice, joysticks or pointing sticks, gamepads and graphic tablets, and audio / visual devices (such as speakers, digital cameras, digital camcorders, portable media players, webcams, image scanners, fingerprint scanners, barcode readers, 3D scanners, 3D printers, laser rangefinders, and eye-tracking devices). Furthermore, user interface input devices may include, for example, medical imaging input devices such as computed tomography, magnetic resonance imaging, positron emission tomography, and medical ultrasound devices. User interface input devices may also include, for example, audio input devices such as MIDI keyboards and digital musical instruments.

[0174] Generally, the use of the term “output device” is intended to include all possible types of devices and mechanisms for outputting information from the computer system 800 to a user or another computer. User interface output devices may include non-visual displays such as display subsystems, indicator lights, or audio output devices. Display subsystems may include flat panel devices such as those using cathode ray tubes (CRTs), liquid crystal displays (LCDs), or plasma displays, projection devices, and touchscreens. For example, user interface output devices may include, but are not limited to, a variety of display devices that visually convey text, graphics, and audio / video information, such as monitors, printers, speakers, headphones, car navigation systems, plotters, audio output devices, and modems.

[0175] The storage subsystem 818 provides a repository or datastore for storing information and data used by the computer system 800. The storage subsystem 818 provides a tangible, non-temporary, computer-readable storage medium for storing basic programming and data structures that provide some example functionalities. The storage subsystem 818 may store software (e.g., programs, code modules, instructions) that, when executed by the processing subsystem 804, provides the above functionalities. This software may be executed by one or more processing units of the processing subsystem 804. The storage subsystem 818 may also provide the authentication relating to the teachings of this disclosure.

[0176] The storage subsystem 818 may include one or more non-temporary memory devices, including volatile and non-volatile memory devices. As shown in Figure 8, the storage subsystem 818 includes system memory 810 and computer-readable storage medium 822. The system memory 810 may include several memories, which include volatile main random access memory (RAM) for storing instructions and data during program execution, and non-volatile read-only memory (ROM) or flash memory for storing fixed instructions. In some implementations, a basic input / output system (BIOS), which includes basic routines to help transfer information between elements within the computer system 800 during startup, etc., may generally be stored in ROM. The RAM generally includes data and / or program modules currently being operated and executed by the processing subsystem 804. In some implementations, the system memory 810 may include several different types of memory, such as static random access memory (SRAM) and dynamic random access memory (DRAM).

[0177] As an example, as shown in Figure 8, system memory 810 may, but is not limited to, load running application programs 812 (which may include various applications such as web browsers, middle-tier applications, and relational database management systems (RDBMS)), program data 814, and operating system 816. As an example, operating system 816 may include various versions of Microsoft Windows®, Apple Macintosh®, and / or Linux operating systems, various commercially available UNIX® or UNIX-like operating systems (including, but not limited to, various GNU / Linux operating systems and Google Chrome® OS), and mobile operating systems such as iOS, Windows® Phone, Android® OS, BlackBerry® OS, and Palm® OS.

[0178] The computer-readable storage medium 822 may store programming and data structures that provide several example functions. The computer-readable storage medium 822 may provide the computer system 800 with computer-readable instructions, data structures, program modules, and other data storage. Software (programs, code modules, instructions) that provides the above functions when executed by the processing subsystem 804 may be stored in the storage subsystem 818. As an example, the computer-readable storage medium 822 may include non-volatile memory such as hard disk drives, magnetic disk drives, optical disk drives such as CD-ROMs, DVDs, Blu-ray® discs, or other optical media. The computer-readable storage medium 822 may include, but is not limited to, Zip® drives, flash memory cards, universal serial bus (USB) flash drives, secure digital (SD) cards, DVD discs, digital videotapes, etc. Computer-readable storage media 822 may also include flash memory-based SSDs, enterprise flash drives, solid-state drives (SSDs) based on non-volatile memory such as solid-state ROM, SSDs based on volatile memory such as solid-state RAM, dynamic RAM, and static RAM, DRAM-based SSDs, magnetoresistive RAM (MRAM) SSDs, and hybrid SSDs that use a combination of DRAM-based SSDs and flash memory-based SSDs.

[0179] In a particular example, the storage subsystem 818 may include a computer-readable storage medium reader 820 that can be further connected to the computer-readable storage medium 822. The reader 820 may be configured to receive and read data from memory devices such as disks and flash drives.

[0180] In certain cases, computer system 800 may support virtualization technologies, including but not limited to the virtualization of processing and memory resources. For example, computer system 800 may provide support for running one or more virtual machines. In certain cases, computer system 800 may run programs such as hypervisors that facilitate the configuration and management of virtual machines. Each virtual machine may also consist of allocated memory, computing resources (e.g., processors, cores), input / output, and networking resources. Each virtual machine generally operates independently of other virtual machines. A virtual machine generally runs its own operating system, which may be the same as or different from the operating systems run by other virtual machines run by computer system 800. Thus, in some cases, multiple operating systems may run simultaneously by computer system 800.

[0181] The communication subsystem 824 provides interfaces to other computer systems and networks. It functions as an interface for sending and receiving data between other systems and the computer system 800. For example, the communication subsystem 824 may enable the computer system 800 to establish a communication channel with one or more client devices via the Internet for sending and receiving information. For instance, if the computer system 800 is used to implement the bot system 120 shown in Figure 1, the communication subsystem may be used to communicate with a chatbot system selected to suit the application.

[0182] The communication subsystem 824 may support both wired and / or wireless communication protocols. In certain examples, the communication subsystem 824 may include radio frequency (RF) transceiver components for accessing wireless voice and / or data networks (e.g., cellular telephone technology, advanced data network technologies such as 3G, 4G, or EDGE (enhanced data rates for global evolution), WiFi® (IEEE 802.XX family standards, or other mobile communication technologies, or any combination thereof), global positioning system (GPS) receiving components, and / or other components. In some examples, the communication subsystem 824 may provide wired network connectivity (e.g., Ethernet) in addition to, or instead of, a wireless interface.

[0183] The communication subsystem 824 can send and receive data in various formats. In some examples, in addition to other formats, the communication subsystem 824 can receive input communications in the form of structured and / or unstructured data feeds 826, event streams 828, event update information 830, etc. For example, the communication subsystem 824 may be configured to receive (or send) data feeds 826 in real time from users of social media networks and / or other communication services, such as web feeds like Twitter® feeds, Facebook® updates, Rich Site Summary (RSS) feeds, and / or real-time update information from one or more third-party sources.

[0184] In certain examples, the communication subsystem 824 may be configured to receive data in the form of a continuous data stream, which may include an event stream 828 of real-time events and / or event update information 830 of which are inherently continuous or infinite and do not have a clear termination. Examples of applications that generate continuous data include, for example, sensor data applications, financial tickers, network performance measurement tools (e.g., network monitoring and traffic management applications), clickstream analysis tools, and automotive traffic monitoring.

[0185] Furthermore, the communication subsystem 824 may be configured to transmit data from the computer system 800 to other computer systems or networks. This data may be transmitted to one or more databases in various different formats, such as structured and / or unstructured data feeds 826, event streams 828, and event update information 830, and these one or more databases may communicate with one or more streaming data source computers coupled to the computer system 800.

[0186] Computer system 800 may be one of many types, such as a handheld portable device (e.g., iPhone® mobile phone, iPad® calculating tablet, PDA), a wearable device (e.g., Google Glass® head-mounted display), a personal computer, a workstation, a mainframe, a kiosk, a server rack, or any other data processing system. Because the nature of computers and networks is constantly changing, the description of computer system 800 shown in Figure 8 is intended only as a specific example. Many other configurations are possible with more or fewer components than the system shown in Figure 8. It should be understood that there are other ways and / or methods of carrying out various examples based on the disclosures and teachings provided herein.

[0187] While specific examples have been described, various variations, modifications, alternative configurations, and equivalents are possible. The examples are not limited to operation within a specific data processing environment, but can freely operate within multiple data processing environments. Furthermore, while specific examples have been described using a particular set of transactions and steps, it will be apparent to those skilled in the art that this is not intended to be limiting. Some flowcharts describe operations as sequential processes, but many of these operations may be performed in parallel or concurrently. The order of operations may also be rearranged. Processes may have additional steps not included in the diagrams. The various features and aspects of the above examples may be used individually or together.

[0188] Furthermore, while specific examples have been described using particular combinations of hardware and software, it should be recognized that other combinations of hardware and software are also possible. A particular example may be implemented using hardware alone, software alone, or a combination thereof. The various processes described herein may be implemented on the same processor in any combination, or on different processors in any combination.

[0189] A device, system, component, or module is described as being configured to perform a particular operation or function, but such configuration may be achieved, for example, by designing electronic circuits to perform an operation, by programming programmable electronic circuits (such as a microprocessor) to perform an operation, such as by executing computer instructions or code programmed to execute code or instructions stored in a non-temporary memory medium, or by executing a processor or core, or by any combination thereof. Processes may communicate using a variety of techniques, including but not limited to conventional techniques for inter-process communication, different pairs of processes may use different techniques, and the same pair of processes may use different techniques at different times.

[0190] To fully understand the examples, specific details are provided in this disclosure. However, the examples can be implemented without these specific details. For example, well-known circuits, processes, algorithms, structures, and techniques are shown without unnecessary details to avoid obscuring the examples. This description is merely illustrative and is not intended to limit the scope, applicability, or configuration of other examples. Rather, the above description of the examples will provide a practical description for implementing various examples for those skilled in the art. Various modifications may be made in terms of the function and arrangement of the elements.

[0191] Therefore, the specification and drawings should be understood as illustrative rather than restrictive. However, it will be clear that additions, subtractions, deletions, and other variations and modifications may be made without departing from the broader spirit and scope set forth in the claims. Thus, while specific examples have been described, they are not intended to be restrictive. Various variations and equivalents are within the scope of the following claims.

[0192] While the above specification illustrates aspects of the disclosure with reference to specific examples, those skilled in the art will recognize that the disclosure is not limited thereto. The various features and aspects of the above disclosure may be used individually or together. Furthermore, the examples may be used in many more environments and applications beyond those described herein without departing from the broader spirit and scope of the specification. Accordingly, the specification and drawings should be considered illustrative rather than restrictive.

[0193] In the above description, the methods are presented in a specific order for illustrative purposes. It should be understood that in alternative examples, these methods may be executed in a different order than described. It should also be understood that the above methods may be executed by hardware components, or embodied in a sequence of machine-executable instructions that can be used to cause a machine, such as a general-purpose or special-purpose processor or logic circuit programmed with instructions, to execute the above methods. These machine-executable instructions may be stored in one or more machine-readable media, such as a CD-ROM or other type of optical disk, a floppy diskette, ROM, RAM, EPROM, EEPROM, magnetic or optical card, flash memory, or other type of machine-readable media suitable for storing electronic instructions. Alternatively, these methods may be executed by a combination of hardware and software.

[0194] When a component is described as being configured to perform a particular operation, such a configuration can be achieved, for example, by designing electronic circuits or other hardware to perform the operation, by programming programmable electronic circuits (e.g., a microprocessor or other suitable electronic circuit) to perform the operation, or by any combination thereof.

[0195] While this specification has described in detail the exemplary examples of the present application, the concepts of the present invention may be embodied and adopted in various ways, and it should be understood that the appended claims are intended to be interpreted to encompass such modifications, unless limited by the prior art.< / date>

Claims

1. 1. A method comprising: a data processing system generating a list of values ​​to cover the entity; the data processing system selecting, from a dataset, an utterance having a context for the entity; and the data processing system converting the utterance into a plurality of templates, each of the plurality of templates including a slot for the entity that maps to the list of values, the method further comprising: the data processing system selecting a template from the plurality of templates; the data processing system selecting a value from the list of values ​​based on a mapping between the slot in the selected template and the list of values ​​for the entity; and generating artificial speech using the selected template and the selected value, wherein generating the artificial speech includes inserting the selected value into the slot of the selected template.

2. The method further comprises, before selecting an utterance from the dataset, the data processing system training a model using a training dataset, the training dataset being the dataset or a subset thereof, the method further comprising: the data processing system testing the model using a test data set or a validation data set; the data processing system evaluating performance of the model based on testing the model; The method of claim 1 , further comprising the data processing system identifying the entity based on the evaluating.

3. the data processing system augmenting the data set or the training data set with the artificial utterances to generate an augmented training data set; the data processing system training the model using the augmented training data set; the data processing system retesting the model using the test data set or the validation data set, or another test data set or another validation data set; the data processing system evaluating the performance of the model based on retesting the model; determining by the data processing system that the performance of the model meets predetermined criteria; 3. The method of claim 2, further comprising: in response to the model satisfying the predetermined criteria, deploying the model as part of a chatbot system.

4. the data processing system determining that the slot in the selected template includes a first element that maps to the list of values ​​and a second element that maps to a function; the data processing system processes the function to generate a function value based on a mapping between the slot in the selected template and the function; the data processing system further comprising combining the selected value with the function value to obtain a slot value; The method of claim 1 , wherein generating the artificial utterance comprises inserting the slot value into the slot of the selected template.

5. the data processing system determining that the selected template maps to a nested set of slot mappings that includes the slot and other slots; the data processing system determining that the other slots in the nested set of slot mappings map to other lists of values; the data processing system further comprising: selecting another value from the list of other values ​​based on a mapping between the other slot in the selected template and the list of other values; 2. The method of claim 1 , wherein generating the artificial utterance comprises inserting the selected value into the slot of the selected template and inserting the selected other value into the other slot of the selected template.

6. 10. The method of claim 1, further comprising the data processing system augmenting the data set or a training data set with the artificial utterances to generate an augmented training data set for training a model to recognize and classify multiple entities including the entity.

7. 10. The method of claim 1, wherein the templates are selected based on heuristics including: (i) a choice between templates with multiple entity classes and templates with a single entity class; (ii) a trade-off between template diversity and template quality; (iii) a classification of templates according to context; (iv) the model requires a minimum number of occurrences per template / specific context that a user wants to teach the model; or (v) any combination thereof.

8. 1. A system comprising: one or more processors; a memory coupled to the one or more processors, the memory storing a plurality of instructions executable by the one or more processors, the plurality of instructions, when executed by the one or more processors, causing the one or more processors to perform an operation, the operation comprising: generating a list of values ​​to cover the entity; selecting an utterance from a dataset that has a context of the entity; and converting the utterance into a plurality of templates, each of the plurality of templates including a slot for the entity that maps to the list of values, the process further comprising: selecting a template from the plurality of templates; selecting a value from the list of values ​​based on a mapping between the slot in the selected template and the list of values ​​for the entity; generating an artificial utterance using the selected template and the selected value, wherein generating the artificial utterance includes inserting the selected value into the slot of the selected template.

9. The process further comprises: and prior to selecting utterances from the dataset, training a model using a training dataset, the training dataset being the dataset or a subset thereof, the process further comprising: testing the model using a test or validation data set; evaluating the performance of the model based on testing the model; and and identifying the entity based on the evaluating.

10. The process further comprises: augmenting the dataset or the training dataset with the artificial utterances to generate an augmented training dataset; training the model using the augmented training data set; retesting the model using the test or validation data set or another test or validation data set; assessing the performance of the model based on retesting the model; and determining that the performance of the model meets predetermined criteria; and deploying the model as part of a chatbot system in response to the model satisfying the predetermined criteria.

11. The process further comprises: determining that the slot in the selected template includes a first element that maps to the list of values ​​and a second element that maps to a function; processing the function to generate a function value based on a mapping between the slots in the selected template and the function; combining the selected value with the function value to obtain a slot value; The system of claim 8 , wherein generating the artificial utterance comprises inserting the slot value into the slot of the selected template.

12. The process further comprises: determining that the selected template maps to a nested set of slot mappings that includes the slot and other slots; determining that the other slots in the nested set of slot mappings map to other lists of values; selecting another value from the list of other values ​​based on a mapping between the other slot in the selected template and the list of other values; 9. The system of claim 8, wherein generating the artificial utterance includes inserting the selected value into the slot of the selected template and inserting the selected other value into the other slot of the selected template.

13. 10. The system of claim 8, wherein the processing further comprises augmenting the data set or a training data set with the artificial utterances to generate an augmented training data set for training a model to recognize and classify multiple entities including the entity.

14. 10. The system of claim 8, wherein the templates are selected based on heuristics including: (i) a choice between templates with multiple entity classes or templates with a single entity class; (ii) a trade-off between template diversity and template quality; (iii) a classification of templates according to context; (iv) the model requires a minimum number of occurrences per template / specific context that a user wants to teach the model; or (v) any combination thereof.

15. 1. A non-transitory computer-readable memory storing a plurality of instructions executable by one or more processors, the plurality of instructions including instructions that, when executed by the one or more processors, cause the one or more processors to perform a process, the process comprising: generating a list of values ​​to cover the entity; selecting an utterance from a dataset that has a context of the entity; and converting the utterance into a plurality of templates, each of the plurality of templates including a slot for the entity that maps to the list of values, the process further comprising: selecting a template from the plurality of templates; selecting a value from the list of values ​​based on a mapping between the slot in the selected template and the list of values ​​for the entity; generating an artificial utterance using the selected template and the selected value, wherein generating the artificial utterance includes inserting the selected value into the slot of the selected template.

16. The process further comprises: and prior to selecting utterances from the dataset, training a model using a training dataset, the training dataset being the dataset or a subset thereof, the process further comprising: testing the model using a test or validation data set; evaluating the performance of the model based on testing the model; and and identifying the entity based on the evaluating.

17. The process further comprises: augmenting the dataset or the training dataset with the artificial utterances to generate an augmented training dataset; training the model using the augmented training data set; retesting the model using the test or validation data set or another test or validation data set; assessing the performance of the model based on retesting the model; and determining that the performance of the model meets predetermined criteria; and in response to the model satisfying the predetermined criteria, deploying the model as part of a chatbot system.

18. The process further comprises: determining that the slot in the selected template includes a first element that maps to the list of values ​​and a second element that maps to a function; processing the function to generate a function value based on a mapping between the slots in the selected template and the function; combining the selected value with the function value to obtain a slot value; 16. The non-transitory computer-readable memory of claim 15, wherein generating the artificial utterance comprises inserting the slot value into the slot of the selected template.

19. The process further comprises: determining that the selected template maps to a nested set of slot mappings that includes the slot and other slots; determining that the other slots in the nested set of slot mappings map to other lists of values; selecting another value from the list of other values ​​based on a mapping between the other slot in the selected template and the list of other values; 16. The non-transitory computer-readable memory of claim 15, wherein generating the artificial utterance includes inserting the selected value into the slot of the selected template and inserting the selected other value into the other slot of the selected template.

20. 16. The non-transitory computer-readable memory of claim 15, wherein the processing further comprises augmenting the data set or training data set with the artificial utterances to generate an augmented training data set for training a model to recognize and classify multiple entities including the entity.

Images