Entity level data augmentation in chatbot for robust named entity recognition

Abstract

To provide a method, a system, and a non-transitory computer-readable memory for augmenting data for training chatbot systems in natural language processing.SOLUTION: A method includes generating a list of values to cover an entity, selecting utterances having context of the entity from a data set, and converting the utterances into a plurality of templates. Each template in the plurality of templates includes a slot for mapping to the list of values for the entity. The method further includes selecting a template from the plurality of templates, selecting a value from the list of values on the basis of the mapping between the slot within the selected template and the list of values for the entity, and generating an artificial utterance on the basis of the selected template and the selected value. Generating the artificial utterance includes inserting the selected value into the slot of the selected template for mapping to the list of values for the entity.SELECTED DRAWING: Figure 5C

Description

[Technical Field]

[0001] Priority claim This application is a nonprovisional application of and claims the benefit of U.S. Provisional Application No. 63 / 038,337, filed June 12, 2020, the entire disclosure of which is incorporated herein by reference.

[0002] FIELD OF THE INVENTION The present disclosure relates generally to chatbot systems, and more particularly to data augmentation techniques for training chatbot systems in natural language processing. [Background technology]

[0003] background Many users around the world use instant messaging or chat platforms to get instant responses. Organizations often use these instant messaging or chat platforms to communicate live with their customers (or end users). However, it can be very expensive for organizations to employ service staff to communicate live with customers or end users. Chatbots or bots have begun to be developed to simulate conversations with end users, especially over the internet. End users can converse with the bots through messaging apps they already have installed and are using. Generally, AI-powered intelligence Because intelligent bots are capable of more intelligent and contextual communication in live conversation, they can potentially improve the conversational experience by enabling a more natural conversation between the bot and the end user. Instead of the end user having to learn a fixed set of keywords or commands that the bot knows how to respond to, an intelligent bot can understand the end user's intent based on the user's utterances in natural language and respond accordingly.

[0004] However, building chatbots is challenging because these automated solutions require specific knowledge in a particular domain and the application of specific techniques that may only be within the capabilities of an expert developer. As part of building such a chatbot, a developer may first understand the needs of the enterprise and end users. The developer may then perform analysis and decision-making related to, for example, selecting a dataset to use for analysis, preparing the input dataset for analysis (e.g., cleansing the data, extracting, formatting, and / or transforming the data before analysis, performing data feature engineering, etc.), identifying appropriate machine learning (ML) techniques or model(s) to perform the analysis, and improving the techniques or models to improve results / outcomes based on feedback. The task of identifying an appropriate model may involve developing multiple models, possibly in parallel, and repeatedly testing and experimenting with these models before identifying one or more specific models to use. Furthermore, supervised learning-based solutions typically include a training phase, followed by an application (i.e., inference) phase, and an iterative loop between the training and application phases. The developer may be responsible for carefully implementing and monitoring these phases to achieve an optimal solution. For example, to train ML techniques or models, one might apply specific patterns or features (e.g., in the case of a chatbot, this involves not only raw language processing but also intent extraction and careful syntactic parsing) that the ML techniques or models will use to predict a desired outcome (e.g., inferring intent from an utterance). Accurate training data is necessary so that algorithms can understand and learn. To ensure that the ML technique(s) or model(s) properly learn these patterns and features, developers may be responsible for selecting, enhancing, and optimizing the training datasets for the ML technique(s) or model(s). Summary of the Invention [Means for solving the problem]

[0005] A brief summary The present disclosure relates to data augmentation techniques for training chatbot systems in natural language processing.

[0006] In various embodiments, a method is provided, the method comprising: a data processing system generating a list of values to cover an entity; the data processing system selecting an utterance having a context of the entity from a dataset; and the data processing system converting the utterance into a plurality of templates, each of the plurality of templates including a slot for the entity mapping to a list of values; the method further comprising: the data processing system selecting a template from the plurality of templates; the data processing system selecting a value from the list of values based on the mapping between the slot in the selected template and the list of values for the entity; and the data processing system generating an artificial utterance using the selected template and the selected value, wherein generating the artificial utterance comprises inserting the selected value into the slot of the selected template.

[0007] In some embodiments, the method further comprises, prior to selecting utterances from the dataset, the data processing system training the model using a training dataset, where the training dataset is the dataset or a subset thereof, and the method further comprises the data processing system testing the model using a test dataset or a validation dataset, the data processing system evaluating performance of the model based on testing the model, and the data processing system identifying entities based on the evaluation.

[0008] In some embodiments, the method further comprises: augmenting the dataset or the training dataset with artificial speech to generate an augmented training dataset; training a model using the augmented training dataset; retesting the model using the test dataset or the validation dataset or other test dataset or other validation dataset; evaluating performance of the model based on retesting the model; determining that the performance of the model meets predetermined criteria; and deploying the model as part of a chatbot system in response to the model meeting the predetermined criteria.

[0009] In some embodiments, the method further comprises the data processing system determining that a slot in the selected template includes a first element that maps to a list of values and a second element that maps to a function; the data processing system processing the function to generate a function value based on the mapping between the slot in the selected template and the function; and the data processing system combining the selected value and the function value to obtain a slot value, wherein generating the artificial utterance includes inserting the slot value into the slot of the selected template.

[0010] In some embodiments, the method further comprises: the data processing system The method comprises determining that the template maps to a nested set of slot mappings including the slot and another slot; the data processing system determining that the other slot in the nested set of slot mappings maps to a list of other values; and the data processing system selecting another value from the list of other values based on the mapping between the other slot in the selected template and the list of other values, and generating the artificial utterance includes inserting the selected value into the slot of the selected template and inserting the selected other value into the other slot of the selected template.

[0011] In some embodiments, the method further comprises the data processing system augmenting the dataset or the training dataset with artificial utterances to generate an augmented training dataset for training the model to recognize and classify a plurality of entities including the entity.

[0012] In some embodiments, templates are selected based on heuristics including: (i) a choice between templates with multiple entity classes versus templates with a single entity class; (ii) a trade-off between template diversity and template quality; (iii) categorization of templates according to context; (iv) whether the model requires a minimum number of occurrences per template / specific context that the user wants to teach the model; or (v) any combination of these.

[0013] In various embodiments, a system is provided that includes one or more data processors and a non-transitory computer-readable storage medium that includes instructions that, when executed on the one or more data processors, cause the one or more data processors to perform some or all of one or more methods disclosed herein.

[0014] In various embodiments, a computer program product is provided, the computer program product being tangibly embodied in a non-transitory machine-readable storage medium and including instructions configured to cause one or more data processors to perform some or all of one or more of the methods disclosed herein.

[0015] The techniques described above and below can be implemented in many aspects and in many contexts. As described in more detail below, some example implementations and contexts are provided with reference to the following figures. However, the following implementations and contexts are only a few of many. [Brief explanation of the drawings]

[0016] [Figure 1] FIG. 1 is a simplified block diagram illustrating a distributed environment incorporating an illustrative embodiment. [Figure 2] FIG. 1 is a simplified block diagram illustrating a computing system implementing a Masterbot, according to certain embodiments. [Figure 3] FIG. 1 is a simplified block diagram illustrating a computing system for implementing a skillbot, according to certain embodiments. [Figure 4] FIG. 1 is a simplified block diagram illustrating a chatbot training and deployment system according to various embodiments. [Figure 5A] FIG. 1 illustrates a process flow for training a predictive model with augmented training data, according to various embodiments. [Figure 5B] FIG. 1 illustrates a process flow for generating augmented training data, according to various embodiments. [Figure 5C] FIG. 1 illustrates a process flow for generating artificial utterances from templates, according to various embodiments. [Figure 6] 1 is a simplified diagram illustrating a distributed system for implementing various embodiments. [Figure 7] FIG. 1 is a simplified block diagram illustrating one or more components of a system environment in which services provided by one or more components of an embodiment system may be provided as cloud services, according to various embodiments. [Figure 8] FIG. 1 illustrates an example of a computer system that may be used to implement various embodiments. DETAILED DESCRIPTION OF THE INVENTION

[0017] Detailed Description In the following description, for purposes of explanation, specific details are set forth in order to provide a thorough understanding of particular embodiments. It will be apparent, however, that various embodiments may be practiced without these specific details. The figures and descriptions are not intended to be limiting. The word "exemplary" is used herein to mean "serving as an example, instance, or illustration." Any embodiment or design described herein as "exemplary" is not necessarily to be construed as preferred or advantageous over other embodiments or designs.

[0018] Introduction A digital assistant is an artificial intelligence-driven interface that helps users accomplish various tasks through natural language conversation. With each digital assistant, customers can pair one or more skills. Skills (also referred to herein as chatbots, bots, or skillbots) are individual bots focused on specific types of tasks, such as tracking inventory, submitting time cards, and creating expense reports. When an end user engages with a digital assistant, the digital assistant evaluates the end user's input and routes the conversation to or from the appropriate chatbot. Digital assistants can be made available to end users through various channels, such as Facebook Messenger, SKYPE MOBILE messenger, or short message service (SMS). Channels carry chats back and forth from end users on various messaging platforms to the digital assistant and its various chatbots. Channels may also support user-agent escalation, event-initiated conversations, and testing.

[0019] Intents enable a chatbot to understand what a user wants from it. Intents consist of a sequence of typical user requests and statements, also called utterances (e.g., get account balance, make a purchase, etc.). As used herein, utterance or message may refer to a set of words (e.g., one or more sentences) exchanged during a conversation with a chatbot. An intent may be created by providing a name that exemplifies a certain user action (e.g., order a pizza) and compiling a set of real-world user statements, or utterances, commonly associated with triggering the action. Because chatbot cognition is derived from these intents, each intent can be created from a robust (1 to 20+ utterances) and diverse dataset to enable the chatbot to interpret ambiguous user input. A rich set of utterances allows the chatbot to understand the user's wishes when it receives messages with the same meaning but different expressions, such as "Forget this order!" or "Cancel delivery!" These intents and their associated utterances together constitute the training corpus for the chatbot. By training a model with this corpus, customers essentially have a reference tool for resolving end-user input to a single intent. Customers can improve their chatbot's cognitive capabilities through repeated intent testing and intent training.

[0020] However, building a chatbot that can determine an end user's intent based on the user's utterances is a challenging task, in part due to the subtleties and ambiguities of natural language, as well as the dimensionality of the input space (e.g., possible user utterances) and the size of the output space (number of intents). Thus, chatbots may need to be trained, monitored, debugged, and retrained to improve their performance and the user experience using them. In conventional systems, training systems are provided for training and retraining machine learning models of digital assistants or chatbots in spoken language understanding (SLU) and natural language processing (NLP). Traditionally, models used in chatbot systems are trained in NLP with "crafted" utterances for any intent. For example, an utterance of "Would you like to change the price?" may be used to train a classifier model for the chatbot system to classify this type of utterance into the intent of "Would you like to price match?" Training a model with crafted utterances helps initially train the chatbot system to provide a service, and can then be retrained once the chatbot system is deployed and receives actual utterances from users.

[0021] As part of the NLP processing of speech, digital assistants are trained to understand the meaning of the utterance, which includes identifying one or more intents and one or more entities that correspond to the utterance. Entity extraction in digital assistants involves two stages: named entity recognition via a named entity recognizer and entity resolution. The specific training problem addressed here is related to named entity recognition. Entities can be understood as nouns and are sometimes called slots. Entities are typically things like dates, times, cities, names, brands, etc. (Some common entities used as examples herein are domain-independent and are referred to as system entities, such as PERSON, NUMBER, CURRENCY, DATE_TIME, etc. However, it should be understood that this disclosure is not limited to these system entities and is applicable to any entity type, such as composite entities, entity roles, and entity lists.) For example, with respect to a travel bot, capturing origin, destination, travel mode, price, date, and time is the foundation of the interface. However, entity capture is a challenging task for digital assistants because users input data randomly and out of order, in different languages and in different contexts. Table 1 below shows a few example inputs (utterances) and outputs (entities) for the entity extraction problem.

[0022] [Table 1]

[0023] However, capturing these entities is still important for digital assistants to take action based on user intent.

[0024] Traditionally, training a model for entity recognition starts with pre-labeled data. In a supervised machine learning setting, especially for this problem, it is necessary to provide pre-labels for the model to learn. The main challenge is the lack of sufficient training data with labeled data. More specific challenges in building robust named entity recognition models for digital assistants include: (i) the lack of coverage of a wider variety of CURRENCY symbols in the training data (a condition for handling different currencies); (ii) the lack of coverage of different CURRENCY amount notations in the training data (a condition for handling different monetary notations); (iii) the lack of unique CURRENCY unit and amount combination utterances in the training data (a condition for handling utterances of large and small units per currency and utterances of large, small, and monetary forms); (iv) the lack of utterances of TIME_DATE contractions (a condition for handling abbreviations such as "Sep 5" for September 5); and (v) the lack of utterances of different PERSON and NUMBER formats (a condition for handling all formats of names and numbers). Any machine learning model can only achieve good quality by training with pre-labeled training data. Therefore, the quality of the training data determines the model's behavior.

[0025] The system entities described herein are very general, and therefore, there are many public datasets to use. However, the problem with publicly available pre-labeled data is that it is not diverse enough to train a model to detect entities with all kinds of variations (such as currency types, numeric formats, and data time formats). One option for obtaining diverse labeled data is to manually add labeled utterances to the training data. Another option is to outsource the addition of labeled utterances to freelancers or specialized data labeling companies. Another option is to use crowdsourcing, which essentially scales up the manual work by using crowdworkers. However, these approaches can be difficult to implement for enterprise systems that employ many chatbot systems trained for many different tasks in multiple languages and receive a wide variety of utterances (often very short utterances) for each task. Systems employing these chatbots need to automatically capture entity diversity in a synthetic-agnostic manner to quickly and efficiently generate large corpora of multilingual training data for many different chatbots.

[0026] Therefore, a different approach is needed to address these issues. The developed approach uses data augmentation techniques to generate more utterances with all the characteristics necessary to obtain entity diversity and train a model to detect entities with all kinds of variations. In various embodiments, a method is provided, the method comprising: a data processing system generating a list of values covering the entity; selecting an utterance from a dataset having the entity's context; and converting the utterance into a plurality of templates, each of the plurality of templates including a slot for the entity that maps to a list of values. The method further comprises: the data processing system selecting a template from the plurality of templates; selecting a value from the list of values based on the mapping between the slot in the selected template and the list of values for the entity; and generating an artificial utterance using the selected template and the selected value, wherein generating the artificial utterance includes inserting the selected value into the slot of the selected template.

[0027] Bots and analytics systems A bot (also known as a skill, chatbot, chatterbot, or talkbot) is a computer program that can conduct a conversation with an end user. Bots can respond to natural language messages (e.g., questions or comments), typically through messaging applications that use natural language messages. Enterprises may use one or more bot systems to interact with end users through messaging applications. The messaging application, sometimes referred to as a channel, may be the end user's preferred messaging application that the end user already has installed and is familiar with. Thus, the end user does not need to download and install a new application to chat with the bot system. Messaging applications may include, for example, over-the-top (OTT) messaging channels (such as Facebook Messenger, Facebook WhatsApp, WeChat, Line, Kik, Talk, Skype, Slack, or SMS), virtual private assistants (such as Amazon Dot, Echo, or Show, Google Home, or Apple HomePod), mobile and web application extensions that extend native or hybrid / responsive mobile or web applications with chat capabilities, or voice-based input (such as Siri®, Cortana, Google Voice, or devices or applications with interfaces that use other speech input for interaction).

[0028] In some cases, the bot system uses a Uniform Resource Identifier (URI) The URI may be used to identify a bot system using a string. The URI may be used as a webhook for one or more messaging application systems. The URI may include, for example, a Uniform Resource Locator (URL) or a Uniform Resource Name (URN). The bot system The system transfers messages (for example, hypertext transfer protocol (HTTP)) from a messaging application system. The bot system may be designed to receive a HTTP postcall message. The HTTP postcall message may be directed to a URI from the messaging application system. In some embodiments, the message may be different from the HTTP postcall message. For example, the bot system may be designed to receive a Short Message Service (SMS) message. The bot system may receive a message from a remote server (MS). While the description herein refers to communications received by the bot system as a message, it should be understood that the message may be an HTTP post call message, an SMS message, or any other type of message between the two systems.

[0029] End users can interact with bot systems through conversational interactions (sometimes called conversational user interfaces (UIs)), similar to interactions between people. In some cases, this interaction involves the end user saying "hello" to the bot, and the bot responding with "hi" and asking the end user what it can do for them. In some cases, this interaction can be a transactional interaction with a banking bot, such as transferring money from one account to another, an informational interaction with an HR bot, such as checking vacation balances, or an interaction with a retail bot, such as returning a purchase or requesting technical support.

[0030] In some embodiments, the bot system can intelligently handle interactions with end users without interaction with an administrator or developer of the bot system. For example, an end user can send one or more messages to the bot system to achieve a desired goal. The messages can include specific content, such as text, emojis, audio, images, video, or other message delivery methods. In some embodiments, the bot system can then convert this content into a standard form, such as a representational state transfer (REST) query to an enterprise service with appropriate parameters. The bot system can then convert the call into a natural language response and prompt the end user for additional input parameters or request other additional information. In some embodiments, the bot system may initiate a dialogue with the end user rather than passively responding to the end user's utterances. Described herein are various techniques for identifying explicit invocations of a bot system and determining input for the bot system being invoked. In certain embodiments, analysis of the explicit invocation is performed by a master bot based on detecting an invocation name in the utterance. In response to detecting the invocation name, the utterance may be refined for input to a skill bot associated with the invocation name.

[0031] A conversation with a bot may follow a specific conversational flow that includes multiple states. This flow can define what happens next based on input. In some embodiments, a bot system can be implemented using a state machine that includes user-defined states (e.g., end-user intents) and actions to perform in or for each state. A conversation can take various paths based on end-user input, which can influence the decisions the bot makes about this flow. For example, at each state, the bot can determine the end-user's intent to determine the appropriate action to take next based on the end-user's input. In the context of utterances, the term "intent" as used herein refers to the intent of the user who provided the utterance. For example, a user may intend to have a conversation with a bot to order a pizza, so the user's intent can be expressed by the utterance "Order me a pizza." User intent can be directed to a specific task the user wants the chatbot to perform on their behalf. Thus, utterances can be expressed as questions, commands, requests, etc. that reflect the user's intent. An intent can include a goal the end user wants to achieve.

[0032] In the context of configuring a chatbot, the term "intent" is used herein to refer to configuration information for mapping a user's utterance to a specific task / action or category of task / action that the chatbot can perform. To distinguish between the intent of an utterance (i.e., a user intent) and the intent of a chatbot, the latter is sometimes referred to herein as a "bot intent." A bot intent may consist of a set of one or more utterances associated with the intent. For example, an intent to order a pizza may have various permutations of utterances expressing the desire to order a pizza. These associated utterances can be used to train the chatbot's intent classifier so that it can later determine whether an input utterance from a user matches the pizza ordering intent. A bot intent may be associated with one or more dialog flows for initiating a conversation with a user in a particular state. For example, the first message for a pizza ordering intent may be the question, "What kind of pizza do you like?" In addition to the associated utterance, a bot intent may further comprise a named entity associated with the intent. For example, a pizza ordering intent may include variables or parameters used to perform the task of ordering a pizza, such as topping 1, topping 2, type of pizza, size of pizza, quantity of pizza, etc. The values of the entities are generally obtained through conversation with the user.

[0033] 1 is a simplified block diagram of an environment 100 incorporating a chatbot system according to certain embodiments. The environment 100 includes a digital assistant builder platform (DABP) 102 that enables users of the DABP 102 to create and deploy digital assistant or chatbot systems. The DABP 102 can be used to create one or more digital assistant (DA) or chatbot systems. For example, as shown in FIG. 1, a user 104 representing a particular company may DABP 102 can be used to create and deploy digital assistants 106 for users of a particular business. For example, DABP 102 can be used by a bank to create one or more digital assistants for use by the bank's customers. The same DABP 102 platform may be used by multiple businesses to create digital assistants. As another example, a restaurant (e.g., a pizzeria) owner may use DABP 102 to create and deploy a digital assistant that allows customers of the restaurant to order food (e.g., order pizza).

[0034] For purposes of this disclosure, a "digital assistant" is an entity that assists a user of the digital assistant in accomplishing various tasks through natural language conversation. A digital assistant can be implemented using software alone (e.g., a digital assistant is a digital entity implemented using programs, codes, or instructions executable by one or more processors), using hardware, or using a combination of hardware and software. A digital assistant can be embodied or implemented in a variety of physical systems or devices, such as computers, mobile phones, watches, home appliances, and vehicles. A digital assistant may also be referred to as a chatbot system. Thus, for purposes of this disclosure, the terms digital assistant and chatbot system are interchangeable.

[0035] A digital assistant, such as a digital assistant 106 built using DABP 102, can be used to perform various tasks through natural language-based conversations between the digital assistant and its user 108. As part of the conversation, the user can provide one or more user inputs 110 to the digital assistant 106 and obtain responses 112 from the digital assistant 106. The conversations can include one or more of the inputs 110 and responses 112. Through these conversations, the user can request one or more tasks to be performed by the digital assistant, and in response, the digital assistant is configured to perform the user-requested tasks and respond to the user with appropriate responses.

[0036] User input 110 is generally in the form of natural language and is referred to as an utterance. User utterance 110 may be in text form, such as when a user types a sentence, a question, a text fragment, or a single word and provides it as input to the digital assistant 106. In some embodiments, user utterance 110 may be in the form of voice input or speech, such as when a user says or speaks something that is provided as input to the digital assistant 106. The utterance is typically in the language that the user 108 speaks. For example, the utterance may be in English or another language. If the utterance is in speech form, the speech input is converted into textual speech in that particular language, and the textual speech is then processed by the digital assistant 106. Various speech-to-text processing techniques may be used to convert the speech or voice input into textual speech, which is then processed by the digital assistant 106. In some embodiments, the speech-to-text conversion may be performed by the digital assistant 106 itself.

[0037] The utterance, which may be a text utterance or a voice utterance, may be a fragment, a sentence, multiple sentences, one or more words, one or more questions, a combination of the aforementioned types, etc. The digital assistant 106 is configured to apply natural language understanding (NLU) techniques to the utterance to understand the meaning of the user input. As part of the NLU processing of the utterance, the digital assistant 106 performs processing to understand the meaning of the utterance. The digital assistant 106 is configured to execute a text utterance, which processing includes identifying one or more intents and one or more entities corresponding to the utterance. Upon understanding the meaning of the utterance, the digital assistant 106 can perform one or more actions or operations in response to the understood meaning or intent. For purposes of this disclosure, the utterance is assumed to be a text utterance provided directly by a user 108 of the digital assistant 106 or the result of converting an input voice utterance into text format. However, this is not intended to be limiting or restrictive in any way.

[0038] For example, user 108 input may request a pizza order by providing an utterance such as, "I'd like to order a pizza." Upon receiving such an utterance, digital assistant 106 is configured to understand the meaning of the utterance and take appropriate action. The appropriate action may include, for example, responding to the user with a question requesting user input regarding the type of pizza the user would like to order, the size of the pizza, the pizza toppings, etc. The responses provided by digital assistant 106 may also be in natural language format, typically in the same language as the input utterance. As part of generating these responses, digital assistant 106 may perform natural language generation (NLG). For a user ordering a pizza, throughout a conversation between the user and digital assistant 106, the digital assistant may provide all the necessary information for the pizza order and then guide the user to order the pizza at the end of the conversation. Digital assistant 106 may end the conversation by outputting information to the user indicating that the pizza has been ordered.

[0039] At a conceptual level, digital assistant 106 performs various processes in response to utterances received from a user. In some embodiments, this processing includes a series or pipeline of processing steps, including, for example, understanding the meaning of the input utterance (sometimes referred to as natural language understanding (NLU)), determining an action to perform in response to the utterance, executing the action if necessary, generating a response to be output to the user in response to the user's utterance, outputting the response to the user, etc. NLU processing may include parsing the received input utterance to understand the structure and meaning of the utterance, refining and refining the utterance to develop a more understandable form (e.g., logical form) or structure for the utterance, etc. Generating a response may include using NLG techniques.

[0040] NLU processing performed by a digital assistant such as digital assistant 106 may include various NLP-related processes, such as parsing a sentence (e.g., tokenizing, lemmatizing, identifying part-of-speech tags for a sentence, identifying named entities in a sentence, generating a dependency tree to represent the sentence structure, splitting a sentence into clauses, parsing the individual clauses, resolving anaphora, performing chunking, etc.). In particular embodiments, the NLU processing, or portions thereof, are performed by digital assistant 106 itself. In some other embodiments, digital assistant 106 may use other resources to perform portions of the NLU processing. For example, the syntax and structure of an input spoken sentence may be identified by processing the sentence using a parser, a part-of-speech tagger, and / or a named entity recognizer. In one implementation, for English, a parser, a part-of-speech tagger, and a named entity recognizer, such as those provided by the Stanford Natural Language Processing (NLP) Group, are used to parse the structure and syntax of the sentence. These are provided as part of the Stanford CoreNLP Toolkit.

[0041] Although various examples provided in this disclosure show utterances in English, this is intended as an example only. In certain embodiments, the digital assistant 106 is also capable of processing utterances in languages other than English. The digital assistant 106 may include subsystems (e.g., implementing NLU functionality) configured to perform processing for different languages. The NLU core server may provide a list of subsystems that can be provided by the NLU core server. These subsystems may be implemented as pluggable units that can be invoked using service calls from the NLU core server. This allows for flexible and scalable NLU processing for each language, such as changing the order of processing. Language packs may be provided for individual languages, and language packs can register a list of subsystems that can be provided by the NLU core server.

[0042] A digital assistant, such as the digital assistant 106 shown in FIG. 1, can be available or accessible to its user 108 through a variety of different channels, including, but not limited to, through a particular application, through social media platforms, through various messaging services and applications, and other applications or channels. A single digital assistant may have multiple channels configured for it to run and be accessible on different services simultaneously.

[0043] Digital assistant or chatbot systems typically include or are associated with one or more skills. In particular embodiments, these skills are individual chatbots (called skillbots) configured to interact with users and perform specific types of tasks, such as tracking inventory, submitting timecards, creating expense reports, ordering food, checking bank accounts, making reservations, purchasing widgets, etc. For example, in the embodiment shown in FIG. 1 , digital assistant or chatbot system 106 includes skills 116-1, 116-2, etc. For purposes of this disclosure, the terms “skill” and “skills” are used interchangeably with the terms “skillbot” and “skillbots,” respectively.

[0044] Each skill associated with a digital assistant helps a user of the digital assistant complete a task through a conversation with the user, which may include a combination of text or voice input provided by the user and responses provided by the skill bot. These responses may be in the form of text or voice messages to the user and / or using simple user interface elements (e.g., list selections) presented to the user for the user to make a selection.

[0045] There are various ways in which skills or skill bots can be associated with or added to a digital assistant. In some cases, skill bots may be developed by a company and then added to a digital assistant using DABP 102. In other cases, skill bots may be developed and created using DABP 102 and then added to a digital assistant created using DABP 102. In yet another case, DABP 102 provides an online digital store (referred to as a "skill store") that offers multiple skills aimed at a wide range of tasks. Skills offered through the skill store may also expose various cloud services. To add a skill to a digital assistant being created using DABP 102, a user of DABP 102 can access the skill store via DABP 102, select the desired skill, and indicate that the selected skill should be added to the digital assistant created using DABP 102. Skills from the skill store can be added to a digital assistant as is or in a modified form (e.g., a user of DABP102 may select and clone a particular skillbot provided by the skill store, customize or modify the selected skillbot, and then add the modified skillbot to a digital assistant created using DABP102).

[0046] A variety of different architectures may be used to implement a digital assistant or chatbot system. For example, in certain embodiments, a digital assistant created and deployed using DABP 102 may be implemented using a masterbot / child (or sub)bot paradigm or architecture. According to this paradigm, a digital assistant is implemented as a masterbot that interacts with one or more child bots, which are skillbots. For example, in the embodiment shown in FIG. 1 , digital assistant 106 is composed of masterbot 114 and skillbots 116-1, 116-2, etc., that are child bots of masterbot 114. In certain embodiments, digital assistant 106 may itself function as a masterbot.

[0047] A digital assistant implemented according to the masterbot-childbot architecture allows users of the digital assistant to interact with multiple skills through a unified user interface, i.e., via the masterbot. When a user engages with the digital assistant, user input is received by the masterbot. The masterbot then performs processing to determine the meaning of the user input utterance. The masterbot then determines whether the masterbot itself can handle the task requested by the user in the utterance. Otherwise, the masterbot selects a suitable skillbot to handle the user's request and routes the conversation to the selected skillbot. This allows users to interact with the digital assistant through a common, single interface while still providing the ability to use multiple skillbots configured to perform specific tasks. For example, in a digital assistant developed for an enterprise, the digital assistant's masterbot can interface with skillbots with specific functions, such as a CRM bot for performing functions related to customer relationship management (CRM), an ERP bot for performing functions related to enterprise resource planning (ERP), and an HCM bot for performing functions related to human capital management (HCM). In this way, the end user or consumer of the digital assistant only needs to know how to access the digital assistant through a common master bot interface, and behind the scenes, multiple skill bots are provided to handle user requests.

[0048] In certain embodiments, in a masterbot / childbot infrastructure, the masterbot is configured to be aware of an available list of skillbots. The masterbot has access to metadata identifying various available skillbots and their capabilities, including tasks that can be performed by the skillbots for each skill. Upon receiving a user request in the form of an utterance, the masterbot is configured to identify or predict a specific skillbot from multiple available skillbots that can best accommodate and process the user request. The masterbot then routes the utterance (or a portion of the utterance) to the specific skillbot for further processing. In this manner, control flows from the masterbot to the skillbot. A masterbot can support multiple input and output channels.

[0049] 1 illustrates a digital assistant 106 consisting of a masterbot 114 and skillbots 116-1, 116-2, and 116-3, but this is not intended to be limiting. The digital assistant may include various other components (e.g., other systems and subsystems) that provide the functionality of the digital assistant. These systems and subsystems may be implemented solely in software (e.g., code, instructions stored on a computer-readable medium and executable by one or more processors), solely in hardware, or in an implementation using a combination of software and hardware.

[0050] DABP 102 provides infrastructure and various services and features that enable users of DABP 102 to create digital assistants, including one or more skillbots associated with the digital assistant. In some cases, a skillbot can be created by cloning an existing skillbot, for example, a skillbot provided by a skill store. As previously indicated, DABP 102 provides a skill store or skill catalog that offers multiple skillbots for performing various tasks. Users of DABP 102 can clone a skillbot from the skill store. They can modify or customize the cloned skillbot as needed. In some other cases, users of DABP 102 create a skillbot from scratch using tools and services provided by DABP 102. As previously indicated, a skill store or skill catalog provided by DABP 102 can offer multiple skillbots for performing various tasks.

[0051] In certain embodiments, creating or customizing a skillbot generally involves the following steps:

[0052] (1) Steps to set up a new skill bot (2) Configure one or more intents for your skill bot. (3) configuring one or more entities for one or more intents; (4) Steps to train the skill bot (5) Steps to create a dialogue flow for the skill bot (6) Add custom components to your skill bot as needed (7) Test and deploy your skill bot Each of the above steps is briefly described below.

[0053] (1) Configuring a new skill bot—Various settings can be configured for a skill bot. For example, a skill bot designer can specify one or more invocation names for the skill bot being created. A user of the digital assistant can then explicitly invoke the skill bot using the invocation names. For example, a user can enter the invocation name in a user utterance to explicitly invoke the corresponding skill bot.

[0054] (2) Configuring One or More Intents and Associated Example Utterances for a Skillbot—A skillbot designer specifies one or more intents (also referred to as bot intents) for the skillbot being created. The skillbot is then trained based on these specified intents. These intents represent categories or classes of input utterances that the skillbot is trained to infer. Upon receiving an utterance, the trained skillbot infers the intent of the utterance, and the inferred intent is selected from a predefined set of intents used to train the skillbot. The skillbot then takes an appropriate action in response to the utterance based on the intent inferred for the utterance. In some cases, the intents of a skillbot represent tasks that the skillbot can perform for a user of the digital assistant. Each intent is given an intent identifier or intent name. For example, in a skillbot trained for banking, the intents specified for the skillbot may include "CheckBalance," "TransferMoney," "DepositCheck," etc.

[0055] For each intent defined for a skill bot, the skill bot designer must A skill bot may also provide one or more example utterances that represent and explain the intent. These example utterances are intended to represent utterances that a user might input to the skill bot for that intent. For example, for the CheckBalance intent, example utterances might include, "What's the balance in my savings account?", "How much is in my checking account?", "How much do I have in my account?", etc. Thus, various permutations of typical user utterances may be specified as example utterances for the intent.

[0056] The intents and their associated example utterances are used as training data to train the skill bot. A variety of different training techniques may be used. This training results in a predictive model that is configured to take an utterance as input and output an intent for the utterance that is inferred by the predictive model. In some cases, the input utterance is provided to an intent analysis engine that is configured to predict or infer the intent of the input utterance using the trained model. The skill bot may then take one or more actions based on the inferred intent.

[0057] (3) Configuring entities for one or more intents for the skill bot—In some cases, additional context may be required to enable the skill bot to respond appropriately to a user utterance. For example, there may be situations in which a user input utterance resolves to the same intent in the skill bot. For example, in the example above, the utterances "What's the balance in my savings account?" and "How much do I have in my checking account?" both resolve to the same CheckBalance intent, but these utterances are different requests that ask for different things. To clarify such requests, one or more entities are added to the intent. Using the banking skill bot example, an entity called AccountType that defines values called "checking" and "regular" may enable the skill bot to parse the user request and respond appropriately. In the example above, the utterances resolve to the same intent, but the values associated with the AccountType entity are different for these two utterances. This allows the skill bot to potentially perform different actions for two utterances, even though they resolve to the same intent. One or more entities may be specified for a particular intent configured for the skill bot. Therefore, entities are used to add context to the intent itself, helping to more fully describe the intent and enabling the skill bot to complete the user request.

[0058] In certain embodiments, there are two types of entities: (a) built-in entities provided by DABP 102, and (2) custom entities that can be specified by the skill bot designer. Built-in entities are general-purpose entities that can be used with a wide variety of bots. Examples of built-in entities include, but are not limited to, time, date, address, number, email address, duration, recurring period, currency, phone number, URL, etc. Custom entities are used for more customized applications. For example, in a banking skill, the skill bot designer may define an AccountType entity that enables various banking transactions by checking user input for keywords such as checking, regular, and credit card.

[0059] (4) Training the Skillbot—The skillbot is configured to receive user input in the form of utterances and parse or process the received input to identify or select an intent associated with the received user input. As noted above, the skillbot must be trained for this. In certain embodiments, the skillbot trains the skillbot by providing it with the intents and example utterances associated with the intents (collectively, training The skillbot is trained based on the training data (data) so that it can resolve user input utterances into one of its configured intents. In certain embodiments, the skillbot uses a predictive model that is trained with training data and enables the skillbot to discern what the user is saying (or, in some cases, what they are trying to say). DABP 102 provides a variety of different training techniques that skillbot designers can use to train skillbots, including various machine learning-based training techniques, rule-based training techniques, and / or combinations thereof. In certain embodiments, a portion of the training data (e.g., 80%) is used to train the skillbot model, and another portion (e.g., the remaining 20%) is used to test or validate the model. Once trained, the trained model (sometimes referred to as a trained skillbot) can be used to process and respond to user utterances. In certain cases, the user utterance may be a question that requires only one answer and does not require further conversation. To address such situations, a Q&A (question and answer) intent may be defined for the skillbot. This allows the skillbot to output answers to user requests without having to update the dialog definition. Q&A intents are created in the same manner as regular intents. The dialog flow for a Q&A intent may differ from that of a regular intent.

[0060] (5) Creating a Dialog Flow for the Skill Bot—The dialog flow specified for the skill bot describes how the skill bot will react as its various intents are resolved in response to received user input. This dialog flow defines the behavior or actions of the skill bot, for example, how the skill bot responds to user utterances, how the skill bot prompts the user for input, and how the skill bot returns data. The dialog flow is similar to a flowchart that the skill bot follows. Skill bot designers specify the dialog flow using a language such as Markdown. In certain embodiments, a version of YAML called OBotML may be used to specify the dialog flow for a skill bot. The dialog flow definition for a skill bot serves as a model of the conversation itself, i.e., a model that allows the skill bot designer to compose the interaction between the skill bot and the user it serves.

[0061] In particular embodiments, the dialog flow definition for a skill bot includes the following three sections:

[0062] (a) Context Section (b) Default transition section (c) Status section Context Section - Skill bot designers can define variables used in the conversation flow in the context section. Other variables that can be named in the context section include, but are not limited to, variables for error handling, variables for built-in or custom entities, user variables that allow the skill bot to recognize and persist user preferences, etc.

[0063] Default Transition Section - Transitions for a skill bot can be defined in the dialog flow state section or the default transition section. Transitions defined in the default transition section act as fallbacks and are triggered when there are no applicable transitions defined in a state or when the conditions required to trigger a state transition cannot be met. The default transition section can be used to define routing that allows the skill bot to gracefully handle unexpected user actions.

[0064] State Section - A dialog flow and its associated behavior are defined as a series of temporary states that govern the logic within the dialog flow. Each state node in a dialog flow definition names a component that provides the functionality needed at that point in the dialog. Thus, states are built around components. A state contains characteristics specific to the component and defines transitions to other states that are triggered after the component executes.

[0065] Special case scenarios may be handled using the state section. For example, you may want to offer the user the option to temporarily leave a first skill they are involved in and do something in a second skill within the digital assistant. For example, if a user is interacting with a shopping skill (e.g., making some purchasing selections), they may want to jump to a banking skill (e.g., to ensure they have enough money for the purchase), and then return to the shopping skill to complete the user's order. To address this, an action in a first skill can be configured to return to the original flow after initiating an interaction with a second, different skill within the same digital assistant.

[0066] (6) Adding Custom Components to the Skillbot—As noted above, states specified in a skillbot's dialog flow name components that provide the necessary functionality corresponding to those states. The components enable the skillbot to perform its functions. In certain embodiments, DABP 102 provides a set of pre-configured components for performing a wide range of functions. A skillbot designer can select one or more of these pre-configured components and associate them with states in the skillbot's dialog flow. A skillbot designer can also create custom or new components using tools provided by DABP 102 and associate these custom components with one or more states in the skillbot's dialog flow.

[0067] (7) Test and Deploy Skillbots - DABP 102 provides several features that allow skillbot designers to test the skillbots as they are developed. The skillbots can then be deployed and included in a digital assistant.

[0068] While the above describes how to create a skillbot, similar techniques may be used to create a digital assistant (or masterbot). At the masterbot or digital assistant level, built-in system intents may be configured for the digital assistant. These built-in system intents are used to identify common tasks that the digital assistant itself (i.e., the masterbot) can handle without invoking the skillbot associated with the digital assistant. Examples of system intents defined for a masterbot include (1) Exit, which applies when a user signals a desire to end the current conversation or context in the digital assistant; (2) Help, which applies when a user requests help or orientation; and (3) UnresolvedIntent, which applies to user input that does not neatly match the exit and help intents. The digital assistant also stores information about one or more skillbots associated with the digital assistant. This information enables the masterbot to select a specific skillbot to process an utterance.

[0069] At the master bot or digital assistant level, users can When a phrase or utterance is input into the digital assistant, the digital assistant is configured to perform processing to determine how to route the utterance and associated conversation. The digital assistant makes this determination using a routing model. This routing model may be rule-based, AI-based, or a combination thereof. The digital assistant uses this routing model to determine whether the conversation corresponding to the user input utterance is routed to a specific skill for processing, handled by the digital assistant or master bot itself per built-in system intent, or handled as a different state in the current conversation flow.

[0070] In certain embodiments, as part of this processing, the digital assistant determines whether the user input utterance explicitly identifies a skill bot using its invocation name. If an invocation name is present in the user input, it is treated as an explicit invocation of the skill bot corresponding to the invocation name. In such a scenario, the digital assistant may route the user input to the explicitly invoked skill bot for further processing. In the absence of a specific or explicit invocation, in certain embodiments, the digital assistant evaluates the received user input utterance to calculate confidence scores for the system intent and the skill bot associated with the digital assistant. The calculated score for the skill bot or system intent indicates how likely the user input represents a task that the skill bot is configured to perform, or represents a system intent. Any system intent or skill bot whose associated calculated confidence score exceeds a threshold (e.g., a confidence threshold routing parameter) is selected as a candidate for further evaluation. The digital assistant then selects a specific system intent or skill bot from the identified candidates for further processing of the user input utterance. In certain embodiments, after one or more skill bots are identified as candidates, the intents associated with those candidate skills are evaluated (according to the intent model for each skill) and a confidence score is determined for each intent. Generally, any intent with a confidence score above a threshold (e.g., 70%) is treated as a candidate intent. Once a particular skill bot is selected, the user utterance is routed to that skill bot for further processing. Once a system intent is selected, one or more actions are performed by the master bot itself according to the selected system intent.

[0071] FIG. 2 is a simplified block diagram illustrating a master bot (MB) system 200 according to a specific embodiment. The MB system 200 can be implemented in software only, hardware only, or a combination of hardware and software. The MB system 200 includes a pre-processing subsystem 210, a multiple intent subsystem 220, and a multi-intent subsystem 230. 2 includes a masterbot intent subsystem (MIS) 220, an explicit invocation subsystem (EIS) 230, a skillbot invoker 240, and a data store 250. The MB system 200 shown in FIG. 2 is only one example of an arrangement of components in a masterbot. Those skilled in the art will recognize many possible variations, alternatives, and modifications. For example, in some implementations, the MB system 200 may have more or fewer systems or components than those shown in FIG. 2, may combine two or more subsystems, or may have a different configuration or arrangement of subsystems.

[0072] The pre-processing subsystem 210 receives an utterance "A" 202 from a user and processes the utterance through a language detector 212, a language parser 214, and a named entity recognizer 216. As discussed above, the utterance can be provided in a variety of ways, including audio or text. The utterance 202 can be a sentence fragment, a complete sentence, multiple sentences, etc. The utterance 202 can include punctuation. For example, if the utterance 202 is provided as audio, the pre-processing subsystem 210 may convert speech to text using a speech-to-text converter (not shown) that inserts punctuation marks, such as commas, semicolons, periods, etc., into the resulting text.

[0073] The language detector 212 detects the language of the utterance 202 based on the text of the utterance 202. Each language has its own grammar and semantics, so the way in which the utterance 202 is processed is language dependent. Differences between languages are taken into account when analyzing the syntax and structure of the utterance.

[0074] The linguistic parser 214 parses the utterance 202 to extract part of speech (POS) tags for individual linguistic units (e.g., words) within the utterance 202. POS tags include, for example, nouns (NN), pronouns (PN), verbs (VB), and so on. The language parser 214 may tokenize linguistic units of the utterance 202 (e.g., convert each word into a separate token) and lemmatize the words. A lemma is the primary form of a collection of words as represented in a dictionary (e.g., "run" is the lemma for run, runs, ran, running, etc.). Other types of preprocessing that the language parser 214 may perform include chunking of compound expressions, e.g., combining "credit" and "card" into a single expression "credit_card." The language parser 214 may also identify relationships between words in the utterance 202. For example, in some embodiments, the language parser 214 generates a dependency tree that indicates which parts of the utterance (e.g., a particular noun) are direct objects, which parts of the utterance are prepositions, etc.

[0075] The named entity recognizer 216 identifies named entities in text (e.g., utterances) and classifies them into predefined categories, such as people, organizations, places, expressions of time, currencies, and universal resource language addresses. Identity identification involves detecting words or word strings that form predefined entities. For example, each word represents a token, and "Pacific Ocean" is a string of three tokens that represents one entity. In some cases, inside-outside-beginning tagging, which indicates the beginning and end of an entity, may be used to facilitate entity identification. Each identified entity may then be classified into a predefined category. For example, a named entity recognition machine learning model may detect the word "oracle" in a text string, identify the word as an entity, and classify it as a "company." To learn what are related and unrelated entities and how to make the classification, the named entity recognition machine learning model implemented by the named entity recognizer 216 is trained on a corpus of training data. The more relevant the training data is to the task or skill, the more accurately each model will complete named entity recognition for the task or skill. Once defined, the entities and categories can be used to label data and create a training dataset (in some cases, the named entity recognition data program can do this automatically). This training dataset can then be used to train one or more algorithms to predictively label text for entities and categories in newly presented utterances.

[0076] The results of the processing performed by language detector 212, language parser 214, and named entity recognizer 216 form extracted information 205, which, along with utterance 202 itself, is provided as input to MIS 220.

[0077] As mentioned above, an utterance 202 may contain more than one sentence. For purposes of multiple intent and explicit invocation detection, an utterance 202 may be treated as a single unit even if it contains multiple sentences. However, in particular embodiments, preprocessing may be performed, for example, by preprocessing subsystem 210, to identify a single sentence within multiple sentences for multiple intent and explicit invocation analysis. Generally, an utterance 202 The results produced by the MIS 220 and the EIS 230 are substantially the same whether the sentence is processed at the individual sentence level or as a single unit containing multiple sentences.

[0078] The MIS 220 determines whether the utterance 202 expresses multiple intents. While the MIS 220 can detect the presence of multiple intents in the utterance 202, the processing performed by the MIS 220 does not include determining whether the intent of the utterance 202 matches any intent configured for the bot. Instead, the processing for determining whether the intent of the utterance 202 matches a bot intent can be performed by the intent classifier 242 of the MB system 200 or the intent classifier of the skill bot (shown in the embodiment of FIG. 3). The processing performed by the MIS 220 assumes that a bot (e.g., a particular skill bot or the master bot itself) exists that can process the utterance 202. Thus, the processing performed by the MIS 220 does not require knowledge of what bots are in the chatbot system (e.g., the identities of the skill bots registered with the master bot) or what intents have been configured for a particular bot.

[0079] To determine that utterance 202 contains multiple intents, MIS 220 applies one or more rules from rule set 252 in data store 250. The rules applied to utterance 202 depend on the language of utterance 202 and may include a sentence pattern that indicates the presence of multiple intents. For example, the sentence pattern may include a coordinating conjunction (e.g., a conjunction) that joins two parts of a sentence, both parts corresponding to separate intents. If utterance 202 matches a sentence pattern, it can be inferred that utterance 202 expresses multiple intents. Note that an utterance with multiple intents does not necessarily have different intents (e.g., intents directed to different bots or different intents within the same bot). Instead, the utterance may have different instances of the same intent, such as "order pizza using payment account X, then order pizza using payment account Y."

[0080] As part of determining that utterance 202 represents multiple intents, MIS 220 also determines which portions of utterance 202 are associated with each intent. For each intent represented in the multiple-intent utterance, MIS 220 constructs a new utterance for separate processing to replace the original utterance, e.g., utterance “B” 206 and utterance “C” 208 shown in FIG. 2 . Thus, original utterance 202 may be split into two or more separate utterances that are processed one at a time. MIS 220 determines which of the two or more utterances should be processed first using extracted information 205 and / or from an analysis of utterance 202 itself. For example, MIS 220 may determine that utterance 202 includes a marker word indicating that a particular intent should be processed first. The newly formed utterance corresponding to this particular intent (e.g., one of utterance 206 or utterance 208) is sent first for further processing by EIS 230. After the conversation triggered by the first utterance has ended (or is temporarily suspended), the next highest priority utterance (e.g., the other of utterance 206 or utterance 208) may be sent to EIS 230 for processing.

[0081] EIS 230 determines whether the utterance it receives (e.g., utterance 206 or utterance 208) includes an invocation name for the skillbot. In certain embodiments, each skillbot in a chatbot system is assigned a unique invocation name that distinguishes the skillbot from other skillbots in the chatbot system. A list of invocation names can be maintained in data store 250 as part of skillbot information 254. If the utterance contains words that match an invocation name, the utterance is an explicit invocation. If a bot is not explicitly invoked, the utterance received by the EIS 230 is considered to be an implicit invocation utterance 234 and is input to an intent classifier (e.g., intent classifier 242) of the master bot to determine which bot to use to process the utterance. In some cases, the intent classifier 242 determines that the master bot should process the implicit invocation utterance. In other cases, the intent classifier 242 determines which skill bot the utterance should be routed to for processing.

[0082] The explicit invocation feature provided by EIS 230 has several advantages. It can reduce the amount of processing that a masterbot must perform. For example, when there is an explicit invocation, the masterbot may not need to perform intent classification analysis (e.g., using intent classifier 242) or may be able to reduce the intent classification analysis that it must perform to select a skillbot. Thus, explicit invocation analysis can enable the selection of a specific skillbot without relying on intent classification analysis.

[0083] There may also be overlapping functionality between multiple skill bots. This can occur, for example, when the intents handled by two skill bots overlap or are very close to each other. In such a situation, it becomes difficult for the master bot to identify which of the multiple skill bots to select based solely on intent classification analysis. In such a scenario, an explicit invocation will clarify the specific skill bot to use.

[0084] In addition to determining that an utterance is an explicit invocation, EIS 230 is responsible for determining whether any portion of the utterance should be used as input to the explicitly invoked skillbot. In particular, EIS 230 can determine whether any portion of the utterance is not associated with the invocation. EIS 230 can make this determination by analyzing the utterance and / or by analyzing extracted information 205. Instead of sending the entire utterance received by EIS 230, EIS 230 can send the portion of the utterance that is not associated with the invocation to the invoked skillbot. In some cases, the input to the invoked skillbot is formed by simply removing any portion of the utterance that is associated with the invocation. For example, "I would like to order a pizza using PizzaBot" can be shortened to "I would like to order a pizza" because "using PizzaBot" is related to the invocation of PizzaBot but is irrelevant to the processing performed by PizzaBot. In some cases, EIS 230 may reformat the portion sent to the invoked bot, for example, to form a complete sentence. Thus, the EIS 230 determines not only that there is an explicit call, but also what to send to the skillbot when there is an explicit call. In some cases, there may be no text to input to the bot being called. For example, if the utterance is "pizzabot," the EIS 230 may determine that the pizzabot is being called, but there is no text to be processed by the pizzabot. In such a scenario, the EIS 230 may inform the skillbot invoker 240 that there is nothing to send.

[0085] The skillbot invoker 240 invokes a skillbot in a variety of ways. For example, the skillbot invoker 240 can invoke a bot in response to receiving an indication 235 that a particular skillbot has been selected as a result of an explicit invoke. The indication 235 can be sent by the EIS 230 along with input regarding the explicitly invoked skillbot. In this scenario, the skillbot invoker 240 will relinquish control of the conversation to the explicitly invoked skillbot. The explicitly invoked skillbot determines an appropriate response to the input from the EIS 230 by treating the input as a standalone utterance. For example, the response can include performing a particular action. Alternatively, the new conversation may be started in a particular state where the initial state of the new conversation is determined by input sent from EIS 230.

[0086] Another way that the skillbot invoker 240 can invoke a skillbot is through implicit invocation using the intent classifier 242. The intent classifier 242 can be trained using machine learning and / or rule-based training techniques to determine the likelihood that an utterance represents a task that a particular skillbot is configured to perform. The intent classifier 242 is trained on different classes, i.e., one class for each skillbot. For example, whenever a new skillbot is registered with a masterbot, the intent classifier 242 can be trained using a list of example utterances associated with the new skillbot to determine the likelihood that a particular utterance represents a task that the new skillbot can perform. Parameters generated as a result of this training (e.g., a set of values for the parameters of a machine learning model) can be stored as part of the skillbot information 254.

[0087] In particular embodiments, the intent classifier 242 is implemented using a machine learning model, as described in further detail herein. Training the machine learning model may include inputting at least a subset of utterances from example utterances associated with various skill bots and generating, as an output of the machine learning model, an inference about which bot is the correct bot for processing any particular training utterance. For each training utterance, an indication of the correct bot to use for that training utterance may be provided as ground truth information. The behavior of the machine learning model may then be adapted (e.g., via backpropagation) to minimize the difference between the generated inference and the ground truth information.

[0088] In particular embodiments, the intent classifier 242 determines a confidence score for each skillbot registered with the master bot, indicating the skillbot's likelihood of processing the utterance (e.g., the implicit invocation utterance 234 received from the EIS 230). The intent classifier 242 may also determine a confidence score for each configured system-level intent (e.g., help, exit). If a particular confidence score satisfies one or more conditions, the skillbot invoker 240 invokes the bot associated with this particular confidence score. For example, a threshold confidence score value may need to be met. Thus, the output 245 of the intent classifier 242 is either an identification of a system intent or an identification of a particular skillbot. In some cases, in addition to meeting the threshold confidence score value, the confidence score must exceed the next-highest confidence score by a particular winning margin. Imposing such a condition enables routing to a particular skillbot when the confidence scores of multiple skillbots each exceed the threshold confidence score value.

[0089] After identifying the bot based on the evaluation of the confidence score, the skillbot invoker 240 hands over processing to the identified bot. In the case of a system intent, the identified bot is a masterbot. In other cases, the identified bot is a skillbot. Furthermore, the skillbot invoker 240 determines what to provide as input 247 to the identified bot. As described above, in the case of an explicit invoke, the input 247 can be based on a portion of the utterance that is not associated with this invoke, or the input 247 can be nothing (e.g., an empty string). In the case of an implicit invoke, the input 247 can be the entire utterance.

[0090] The data store 250 includes one or more computing devices that store data used by the various subsystems of the masterbot system 200. As noted above, the data store 250 includes rules 252 and skillbot information 254. The rules 252 include, for example, rules for the MIS 220 to determine when an utterance expresses multiple intents and how to split an utterance expressing multiple intents. The rules 252 further include rules for the EIS 230 to determine which portions of an utterance that explicitly invokes a skillbot to send to the skillbot. The skillbot information 254 includes invocation names of skillbots in the chatbot system, for example, a list of invocation names of all skillbots registered to a particular masterbot. The skillbot information 254 may also include a confidence score for each skillbot in the chatbot system, for example, information used by the intent classifier 242 to determine parameters of a machine learning model.

[0091] 3 is a simplified block diagram of a Skillbot system 300, according to certain embodiments. Skillbot system 300 is a computing system that can be implemented solely in software, solely in hardware, or a combination of hardware and software. In certain embodiments, such as the embodiment shown in FIG. 1, Skillbot system 300 can be used to implement one or more Skillbots within a digital assistant.

[0092] Skillbot system 300 includes MIS 310, intent classifier 320, and conversation manager 330. MIS 310 is similar to MIS 220 in FIG. 2 and provides similar functionality, including being operable to use rules 352 in data store 350 to determine (1) whether an utterance expresses multiple intents, and if so, (2) how to split the utterance into separate utterances for each of the multiple intents. In particular embodiments, the rules applied by MIS 310 to detect multiple intents and to split the utterance are the same as the rules applied by MIS 220. MIS 310 receives utterance 302 and extracted information 304. Extracted information 304 is similar to extracted information 205 in FIG. 2 and can be generated using language detector 212, language parser 214, and / or named entity recognizer 216, or language detectors, language parsers, and / or named entity recognizers local to Skillbot system 300.

[0093] The intent classifier 320 may be trained in a manner similar to the intent classifier 242 described above in connection with the embodiment of FIG. 2 and as described in more detail herein. For example, in particular embodiments, the intent classifier 320 is implemented using a machine learning model. The machine learning model for the intent classifier 320 is trained for a particular skill bot using as training utterances at least a subset of example utterances associated with the particular skill bot. The ground truth for each training utterance will be the particular bot intent associated with the training utterance.

[0094] The utterance 302 can be received directly from a user or provided via a masterbot. If the utterance 302 is provided via a masterbot, for example, as a result of processing via the MIS 220 and the EIS 230 in the embodiment shown in FIG. 2 , the MIS 310 can be bypassed to avoid repeating processing already performed by the MIS 220. However, if the utterance 302 is received directly from a user, for example, during a conversation that occurs after routing to a skillbot, the MIS 310 can process the utterance 302 to determine whether the utterance 302 represents multiple intents. If so, the MIS 310 applies one or more rules to split the utterance 302 into separate utterances for each intent, e.g., utterance “D” 306 and utterance “E” 308. If the utterance 302 does not represent multiple intents, the MIS 310 forwards the utterance 302 to the intent classifier 320 for intent classification without splitting the utterance 302.

[0095] The intent classifier 320 is configured to match a received utterance (e.g., utterance 306 or 308) with an intent associated with the skillbot system 300. As described above, a skillbot can be configured with one or more intents, each of which includes at least one example utterance associated with the intent and used to train the classifier. In the embodiment of FIG. 2, the intent classifier 242 of the masterbot system 200 is trained to determine confidence scores for individual skillbots and system intents. Similarly, the intent classifier 320 can be trained to determine confidence scores for each intent associated with the skillbot system 300. The classification performed by the intent classifier 242 is at the bot level, while the classification performed by the intent classifier 320 is at the intent level and is therefore more fine-grained. The intent classifier 320 has access to intent information 354. The intent information 354 includes a list of utterances for each intent associated with the skillbot system 300, which represent and explain the meaning of the intent and are generally associated with the tasks that can be performed by that intent. The intent information 354 may also include parameters generated as a result of training on this list of utterances.

[0096] The conversation manager 330 receives as an output of the intent classifier 320 an indication 322 that the particular intent identified by the intent classifier 320 most closely matches the utterance input to the intent classifier 320. In some cases, the intent classifier 320 is unable to determine any match. For example, if the utterance is directed to a system intent or an intent of a different skill bot, the confidence score calculated by the intent classifier 320 may fall below a threshold confidence score value. When this occurs, the skill bot system 300 may refer the utterance to the master bot for processing, e.g., routing to a different skill bot. However, if the intent classifier 320 successfully identifies the intent within the skill bot, the conversation manager 330 will begin a conversation with the user.

[0097] The conversation initiated by the conversation manager 330 is specific to the intent identified by the intent classifier 320. For example, the conversation manager 330 may be implemented using a state machine configured to execute a dialog flow for the identified intent. This state machine may include a default starting state (e.g., the intent is invoked without additional input) and one or more further states, each associated with an action to be performed by the skill bot (e.g., perform a purchase transaction) and / or a dialog to be presented to the user (e.g., question, response). Thus, the conversation manager 330 can determine an action / dialog 335 upon receiving an indication 322 identifying the intent and can determine further actions or dialog in response to subsequent utterances received during the conversation.

[0098] Data store 350 includes one or more computing devices that store data used by various subsystems of skillbot system 300. As shown in Figure 3, data store 350 includes rules 352 and intent information 354. In particular embodiments, data store 350 can be integrated into a masterbot or digital assistant data store, such as data store 250 in Figure 2.

[0099] Entity Data Extensions Models used for entity recognition and extraction have been found to provide poor results for texts containing entities that can be expressed in a wide variety of languages and formats. To overcome this problem, various embodiments are directed to techniques for augmenting training data with a diverse set of entities in any number of languages and formats (e.g., adding different texts and text formats used to describe one or more entities) to enable a model to detect entities with all kinds of variations (e.g., currency types, numeric formats, date-time formats, etc.). Augmenting training data with a diverse set of entities in different languages and formats enables better entity recognition and extraction for the model. As described with respect to FIGS. 1, 2, and 3, models trained with augmented entity data can be implemented in chatbot systems. Advantageously, these models and chatbots perform better for utterances in different languages and / or text formats because the models are better able to recognize and extract entities within utterances. Furthermore, because the augmentation is applied automatically in a synthesis-agnostic manner, customers or clients do not need to worry about manually adding utterances in different languages and / or text formats to the training data.

[0100] FIG. 4 is a block diagram illustrating an embodiment of a chatbot system 400 configured to train one or more models, e.g., one or more models implemented as named entity recognizers, based on text data. As shown in FIG. 4, entity recognition performed by the chatbot system 400 in this example includes various stages: a predictive model training stage 410 for building and training models; an evaluation stage 415 for evaluating the performance of the trained models; and a chatbot implementation stage 420 for implementing one or more chatbots. The predictive model training stage 410 builds and trains one or more predictive models 425a-425n (where "n" represents any natural number) for use in the other stages (referred to herein individually and collectively as predictive models 425). For example, the predictive models 425 may include a model for recognizing one or more entities in an utterance, another model for determining the likelihood that an utterance represents a task that a particular skillbot is configured to perform, another model for predicting intent from an utterance for a first type of skillbot, and another model for predicting intent from an utterance for a second type of skillbot. Still other types of predictive models may be implemented in other examples according to this disclosure.

[0101] The predictive model 425 may be a machine learning (“ML”) model such as a convolutional neural network (“CNN”) (e.g., an Inception Neural Network, a Residual Neural Network (“Resnet”)), or a recurrent neural network (e.g., a long short-term memory (“LSTM”) model or a gated recurrent unit (“GRU”) model), or other variants of a deep neural network (“DNN”) (e.g., a multi-label n-binary DNN classifier or a multi-class DNN classifier for single-intent classification). The predictive model 425 may also be any other suitable ML model trained for natural language processing, such as a naive Bayes classifier, a linear classifier, a support vector machine, a bagging model such as a random forest model, a boosting model, a shallow neural network, or one or more combinations of such techniques (e.g., a CNN-HMM or a multi-scale convolutional neural network (MCNN)). The chatbot system 400 may utilize the same or different types of predictive models to recognize one or more entities in an utterance, determine the likelihood that the utterance represents a task that a particular skill bot is configured to perform, predict an intent from the utterance for a first type of skill bot, and predict an intent from the utterance for a second type of skill bot. Still other types of predictive models may be implemented in other examples of the present disclosure.

[0102] To train various predictive models 425, the training stage 410 consists of three main components: dataset preparation 430, template engineering 435, and model training 440. Dataset preparation 430 involves the process of loading data assets 445, splitting the data assets 445 into training and validation sets 445a-n, and performing basic preprocessing so that the system can train and test predictive models 425. The splitting of data assets 445 into training and validation sets 445a-n may be performed randomly (e.g., 90 / 10% or 70 / 30%), or this splitting may be performed according to more complex validation techniques such as K-fold cross-validation, leave-one-out cross-validation, leave-one-group-out cross-validation, or nested cross-validation to minimize sampling bias and overfitting.

[0103] The training data 445a may include at least one subset of utterances from example utterances associated with one or more skill bots. As indicated above, the utterances can be provided in various forms, including audio or text. The utterances can be sentence fragments, complete sentences, multiple sentences, and the like. For example, if the utterances are provided as audio, the data preparation 430 may convert the audio to text using a speech-to-text converter (not shown) that inserts punctuation, such as commas, semicolons, periods, etc., into the resulting text. In some cases, the example utterances are provided by a client or customer. In other cases, the example utterances are automatically generated from a prior library of utterances (e.g., identifying utterances from the library that are specific to the skill the chatbot is specified to learn). The training data 445a for the predictive model 425 may include input text or audio (or input features of text or audio frames) and labels 450 corresponding to the input text or audio (or input features) as a matrix or table of values. For example, for each training utterance, an indication of the correct entity and its classification to be inferred by the predictive model 425 may be provided as ground truth information in the labels 450. The behavior of the predictive model 425 may then be adapted (e.g., by backpropagation) to minimize the difference between the inferences generated for various entities and the ground truth information.

[0104] Template engineering 435 involves a process of augmenting training data 445a to include artificial utterances with a diverse set of entities in any number of languages and formats to make predictive model 425 more robust to utterances in various languages and / or text formats. By augmenting training data 445a with artificial utterances with a diverse set of entities in any number of languages and formats, predictive model 425 better recognizes entities in various languages and text formats. As described in detail with respect to FIGS. 5A-5C , augmentation generally involves the following operations: (i) extracting templates from the original training dataset and ensuring that the augmented training data has a similar distribution of predictor variables as the original training dataset; and (ii) using a custom syntax-based slot mapping method to generate augmented training data including artificial utterances with a diverse set of entities in any number of languages and formats. The introduction of entity-level data augmentation techniques enables predictive model 425 to better recognize entities in any number of languages and formats, thereby providing better performance for utterances in various languages and / or text formats. Furthermore, because entity-level data augmentation is applied automatically in a synthesis-agnostic manner, customers or clients do not need to worry about manually adding utterances in different languages and / or text formats to their training data.

[0105] The training process for the model 425 involves selecting hyperparameters for the model 425 and iteratively inputting utterances from a subset of the augmented training data 445a into the model 425. The training may involve running the model 425 to find a set of model parameters (e.g., weights and / or biases) that maximizes or minimizes an objective function of the model 425, e.g., minimizes a loss function. Hyperparameters are settings that can be adjusted or optimized to control the behavior of the model 425. Most models explicitly define hyperparameters that control various aspects of the model, such as memory or execution cost. However, additional hyperparameters may be defined to adapt the model to specific scenarios. For example, hyperparameters may include the number of hidden units or layers of the model, the model's learning rate, the convolution kernel width, or the number of parameters of the model. Each training iteration may involve finding a model parameter set (consisting of a defined set of hyperparameters) for the model 425 such that the value of the objective function using the model parameter set is smaller than the value of the objective function using a different set of model parameters in the previous iteration. The objective function may be configured to measure the difference between the output inferred using the model 425 and ground truth, which is annotated with labels 150 on a subset of the augmented training data 445a.

[0106] Once a set of model parameters has been identified, the model 425 is trained and can be tested or validated using a subset of the test data 445b (a test or validation data set). The testing or validation process involves iteratively inputting utterances from a subset of the test data 445b into the model 425 using a validation technique such as K-fold cross-validation, leave-one-out cross-validation, leave-one-group-out cross-validation, or nested cross-validation to adjust the hyperparameters and ultimately find an optimal set of hyperparameters. Once an optimal set of hyperparameters is obtained, a reserved test set from a subset of the test data 445a can be input into the model 425 to obtain an output (in this example, one or more recognized entities), which can be evaluated against ground truth entities using correlation techniques such as the Bland-Altman method and Spearman's rank correlation coefficient. Additionally, error, accuracy, precision, recall, and receiver operating characteristic curves (RECCs) can be calculated. Performance metrics such as receiver-operated characteristic curve (ROC) are used to evaluate entity recognition. Metrics 450 may be calculated in stage 415. As described in more detail with respect to Figures 5A-5C, metrics 450 may be used in entity recognition evaluation stage 415 to analyze the performance of model 425 for recognizing entities.

[0107] The model training stage 410 outputs trained models, including one or more trained entity recognition models 455. The one or more trained entity recognition models 455 may be deployed and used in a chatbot implementation stage 420 to implement one or more chatbots. For example, one or more chatbots may receive text data 460 from one or more users and be configured with the one or more trained entity recognition models 455 to recognize and extract entities 465 from various utterances received by the one or more chatbots. The entities 465 may be part of extracted information (e.g., extracted information 205; 304 described in FIGS. 2 and 3, respectively) obtained from the text data 460 and may be used in downstream processing, such as intent classification.

[0108] Entity Data Augmentation Technology 5A is a flowchart illustrating a process 500 for global tuning of a predictive model with entity data augmentation. The process illustrated in FIG. 5 may be implemented in software (e.g., code, instructions, programs) executed by one or more processing units (e.g., processors, cores) of a respective system, hardware, or combination thereof. The software may be stored in a non-transitory storage medium (e.g., a memory device). The method illustrated in FIG. 5 and described below is an exemplary and non-transitory method. 5 illustrates various processing steps occurring in a particular sequence or order, but this is not intended to be limiting. In certain alternative embodiments, the steps may be performed in some different order, or some steps may be performed in parallel. In certain embodiments, such as those illustrated in FIGS. 1-4, the process illustrated in FIG. 5 may be performed by a preprocessing subsystem (e.g., preprocessing subsystem 210 or predictive model training stage 410) to create, train, update, and deploy one or more predictive models (implemented as name entity recognizers) using the augmented data set.

[0109] In step 505, a data processing system (e.g., the chatbot system 400 described with respect to FIG. 4) obtains a dataset containing utterances. The dataset includes realistic task-oriented dialogue data (consisting of utterances), such as question-and-answer datasets (e.g., including Wikipedia articles, manually generated fact-like questions therefrom, and manually generated answers to these questions), customer support datasets (e.g., conversations between two people extracted from chat logs used to receive technical support for various issues), dialogue datasets (e.g., fictional conversation books with conversational exchanges between pairs of movie characters extracted from raw movie scripts), multilingual datasets (e.g., customer feedback describing reasons for dissatisfaction with a given company, product, or service in various languages), etc.

[0110] In step 510, the dataset is split into a training dataset and a test and / or validation dataset (an evaluation dataset set as a target for one or more predictive models). While the following process is described with respect to a single training dataset and a single test and / or validation dataset, it should be understood that a dataset can be split into one or more training datasets and one or more test and / or validation datasets and processed in a similar manner. Splitting the data asset into training and validation sets can be done randomly (e.g., 90 / 10% or 70 / 30%), or the split can be performed using K-fold cross-validation, leave-one-out cross-validation, leave-one-group-out cross-validation, or other methods to minimize sampling bias and overfitting. or more complex validation techniques such as nested cross-validation. The dataset may be pre-labeled and acquired, or may be labeled after acquisition. Labeling may involve pre-defining a list of entity classes, identifying the location of the entities in each utterance, and labeling the entities with a class from the pre-defined list of classes. For example, the classes of entities may be pre-defined as PRODCUT, COMPANY, CURRENCY, and TIME, the index of the utterance between which the entities fall may be identified, and a label may be created for the class of entity within the indexed location (e.g., if an entity such as Oracle, which falls under the COMPANY category, is located between indexes 9 and 17, and an entity such as 10 / 25 / 20, which falls under the TIME category, is located between indexes 35 and 45, the utterance may be labeled as (entities (9, 17, "COMPANY"), (35, 45, "TIME")).

[0111] In step 515, the predictive model is trained on the training dataset to learn parameters for recognizing and classifying entities in utterances. The training process may include selecting hyperparameters for the model and performing iterations of inputting utterances from the training dataset into the model to find a set of model parameters (e.g., weights and / or biases) that maximizes or minimizes an objective function of the model, e.g., minimizes a loss function. Each iteration of training involves selecting hyperparameters for the model using a set of model parameters. The objective function may include finding a set of model parameters (consisting of a defined set of hyperparameters) for the model such that the value of an objective function is less than the value of the objective function using a different set of model parameters in a previous iteration. The objective function may be configured to measure the difference between outputs inferred using the model and ground truth annotated with labels on a training dataset.

[0112] In step 520, the one or more predictive models are tested and / or validated on a test dataset and / or a validation dataset to recognize one or more entities in the utterance. The testing or validation process may involve K-fold cross-validation, leave-one-out cross-validation, leave-one-group-out cross-validation, or other methods to tune the hyperparameters and ultimately find an optimal set of hyperparameters. Use validation techniques such as nested cross-validation to validate the test dataset and / or The method involves iterative input of utterances from a test dataset or a validation dataset into the model. Once an optimal set of hyperparameters is obtained, a reserved test set from the test dataset and / or validation dataset can be input into the model to obtain an output (e.g., one or more recognized entities), which can be evaluated against the ground truth entities using correlation techniques such as the Bland-Altman method and Spearman's rank correlation coefficient. Furthermore, performance metrics such as error (e.g., mean absolute error or mean squared error), accuracy (e.g., classification accuracy), precision (e.g., F1 score), recall (e.g., F1 score), and receiver operating characteristic (ROC) curves (e.g., AUC (Area Under Curve)-ROC curves) can be calculated. The metrics can be used to analyze the model's performance for recognizing entities. For example, evaluation tools can be used to understand false positives and false negatives on the reserved test set to evaluate the model's behavior. Testing and / or validation are performed and analyzed to obtain a baseline evaluation of the model and to obtain an understanding of what, if any, additional information (e.g., additional nouns, text formats, currency types, URL types, etc.) needs to be added or diversified in the training dataset to improve the model's performance for recognizing entities within utterances in different languages and / or text formats. For example, testing and / or validation may indicate that the model performs poorly when attempting to recognize and classify the entity CURRENCY (particularly for utterances lacking the currency of U.S. dollars), thereby providing an understanding that additional information (e.g., additional currency types) needs to be added or diversified in the training dataset to improve the model's performance for the entity CURRENCY.

[0113] In cases where additional information needs to be added or diversified in the training dataset to improve model performance, one or more templates are created or extracted from the original corpus (the dataset obtained in step 505 before splitting) or a subset thereof, such as the training dataset after splitting, in step 525, as described in detail with reference to FIG. 5B. In step 530, artificial utterances are created by a data processing system using one or more templates, as described in detail with reference to FIG. 5B. In step 535, the dataset or a subset thereof is augmented with artificial utterances by the data processing system to obtain an augmented dataset. The rate at which the dataset or a subset thereof is augmented with artificial utterances (augmentation rate) may be set based on one or more factors. In some cases, the expansion rate is selected based on an analysis of test data and validation data. For example, the expansion rate is selected based on a recognized / determined ability to improve the performance of a named entity recognition model. In other cases, the expansion rate may be randomly selected and then optimized. In other cases, the expansion rate is selected based on a predetermined schema to prevent regression in the ability of one or more predictive models to predict entities from one or more previously observed classes. For example, the expansion rate can be calculated based on the schema(s), i.e., CURRENCY:3*| It may be limited by the number of time-date utterances in the original set of training data | and / or TIME_DATE:0.75 * |number of time-date utterances in the original set of training data|.

[0114] In step 540, the augmented data set is split into an augmented training data set and an augmented test data set and / or an augmented validation data set (an evaluation data set set as a target for one or more predictive models). While the following process is described with respect to a single training data set and a single test data set and / or validation data set, it should be understood that the augmented data set can be split into one or more augmented training data sets and one or more augmented test data sets and / or validation data sets and processed in a similar manner. Splitting the augmented data asset into training and validation sets may be performed randomly (e.g., 90 / 10% or 70 / 30%), or this split may be performed according to more complex validation techniques such as K-fold cross-validation, leave-one-out cross-validation, leave-one-group-out cross-validation, or nested cross-validation to minimize sampling bias and overfitting. The augmented training data set is labeled as described with respect to step 510 based on a list of predefined entity classes.

[0115] In step 545, the model is trained (or a new base model is trained) using the augmented training dataset. The training process may include selecting hyperparameters for the model and performing iterations of inputting utterances from the augmented training dataset into the model to find a set of model parameters (e.g., weights and / or biases) that maximizes or minimizes an objective function of the model, e.g., minimizes a loss function. Each iteration of training may include finding a model parameter set (consisting of a defined set of hyperparameters) for the model such that the value of the objective function using the model parameter set is smaller than the value of the objective function using a different set of model parameters in the previous iteration. The objective function may be constructed to measure the difference between the output inferred using the model and ground truth annotated with labels in the augmented training dataset.

[0116] In step 550, the model is tested and / or validated with an augmented test data set and / or an augmented validation data set to recognize one or more entities in the utterance. The testing or validation process involves iterative operations of inputting utterances from the augmented test data set and / or the augmented validation data set into the model using validation techniques such as K-fold cross-validation, leave-one-out cross-validation, leave-one-group-out cross-validation, or nested cross-validation to adjust hyperparameters and ultimately find an optimal set of hyperparameters. Once an optimal set of hyperparameters is obtained, a reserved test set from the test data set and / or the validation data set can be input into the model to obtain an output (e.g., one or more recognized entities), which is evaluated against the ground truth entities using correlation techniques such as the Bland-Altman method and Spearman's rank correlation coefficient. Additionally, performance metrics such as error (e.g., mean absolute error or mean squared error), accuracy (e.g., classification accuracy), precision (e.g., F1 score), recall (e.g., F1 score), and receiver operating characteristic (ROC) curves (e.g., AUC (Area Under Curve)-ROC curves) may be calculated. The metrics may be used to analyze the model's performance for recognizing entities. The metrics may also be used to analyze the model's performance in comparison with the model's baseline evaluation in step 520. For example, to evaluate the model's behavior, an evaluation tool may be used to identify false positives and false negatives on a reserved test set. The difference between the new model and a previous version of the model and / or the difference between the new model and the base model may be identified. A differentiation tool may be used to ensure there are no regressions between two model releases and / or between the new model and the base model.

[0117] Thereafter, based on evaluation of the training and testing of the models, steps 525-550 may be repeated iteratively until the evaluation demonstrates optimal performance of the one or more predictive models for recognizing one or more entities within utterances in various languages and / or text formats. Optimal performance of the model may be defined by the user or the system as achieving one or more metrics of performance (e.g., x% accuracy, achieving an error score of x, x% increase in accuracy over a baseline evaluation or previous version of the model, a decrease in error score for a given entity class by at least x, an improvement of at least x over the baseline evaluation in recognizing and classifying a given entity, etc.). Once the model has been trained and / or optimal performance has been achieved, it is deployed in step 555 within a chatbot system (e.g., as part of a skill bot) for recognizing one or more entities within utterances in various languages and / or text formats. For example, an utterance may be received by a chatbot, the utterance may be analyzed by one or more predictive models to determine whether the utterance includes one or more entities, and the recognized entities may be used in downstream processes by the chatbot, for example, to infer the intent of the utterance.

[0118] FIG. 5B is a flowchart illustrating a process 560 for creating or extracting one or more templates from one or more sets of training data and creating artificial utterances based on the one or more templates, as implemented in the process 500 described with respect to FIG. 5A. The process illustrated in FIG. 5B may be implemented in software (e.g., code, instructions, programs) executed by one or more processing units (e.g., processors, cores) of a respective system, hardware, or combination thereof. The software may be stored in a non-transitory storage medium (e.g., a memory device). The method illustrated in FIG. 5B and described below is intended to be exemplary and non-limiting. While FIG. 5B depicts various processing steps occurring in a particular sequence or order, this is not intended to be limiting. In certain alternative embodiments, steps may be performed in some different order, or some steps may be performed in parallel. In certain embodiments, such as those illustrated in FIGS. 1-4, the process illustrated in FIG. 5B may be performed by a data processing system including a template engine configured to automatically generate templates and utterances for use in training a predictive model. The template engine automatically generates a sufficient amount of augmented data that is not only contextually similar to the training data already used in the training system, but also ensures that the model does not learn specific sentence patterns. This is advantageous over both crowdsourcing data (which provides volume but lacks contextual similarity) or manually adding utterances (which provides contextual similarity but does not allow for the manual generation of a sufficient number of utterances to ensure data diversity).

[0119] In the following discussion of process 560, the data augmentation process (which applies to process 500 as a whole) is described by taking CURRENCY as an example of an entity that a user or computing system has identified (e.g., through the analysis performed in steps 520 and 550) as potentially benefiting from data augmentation (e.g., the training data lacks sufficient coverage for some currency units). While CURRENCY is used as an example entity, it should be understood that the processing described herein may also be performed for other classes of entities and combinations thereof. In step 565, a list of values to cover for a given entity is generated. For example, a data processing system (e.g., chatbot system 400 described with respect to FIG. 4) generates or identifies a list of units of the CURRENCY entity to cover. A gazetteer list (geographical dictionary or directory) may be used to generate a list of well-known nouns, e.g., currency and its synonyms, major and minor units (e.g., dollars and cents), and currency types categorized by country, language, or region by standard text format, e.g., monetary numeric format. The list might look like this:

[0120] List of currencies covered SGD$ Singapore Dollar (singular) Singapore Dollars (multiple) Sing SGD Rs. Rs Australian Dollar (singular) Australian Dollars (multiple) Thai Baht R$ EGP SDG LBP ARS COP PHP CLP DOP CUP UYU ALL In step 570, utterances are selected from a dataset having the context of a given entity (e.g., the original corpus (the dataset obtained in step 505 before splitting) or a subset thereof, such as the training dataset after splitting). For example, a number of utterances from the dataset having the correct context for the CURRENCY entity may be selected manually or by a data processing system. An example utterance that may be selected for the CURRENCY entity is "He paid $100.78 for dinner," but it should be understood that there are many other example utterances with the CURRENCY entity that could be selected alternatively or additionally.

[0121] In step 575, the selected utterance is converted into templates, each of which contains slots that map to a list of values for a given entity. For example, a data processing system running a script may templatize all selected utterances. More specifically, the script converts the selected utterance into a list of templates. A template is essentially a sentence with slots. The templates are stored as a list associated with a given entity and, optionally, the context of the given entity, and templates from the list are selected and filled with different values of the entity by a template engine, as will be described in more detail later with respect to FIG. 5C. Templates may be defined in a custom syntax to customize the behavior of the model for recognizing defined classes of entities within an utterance. For example, a data processing system running a script may templatize all selected utterances. More specifically, the script converts the selected utterance into a list of templates. A template is essentially a sentence with slots. The templates are stored as a list associated with a given entity and, optionally, the context of the given entity, and templates from the list are selected and filled with different values of the entity by a template engine, as will be described in more detail with respect to FIG. 5C. Templates may be defined in a custom syntax to customize the behavior of the model for recognizing defined classes of entities within an utterance. For example, a template may be defined in a metalanguage such as BNF ( Context-free grammars (a context-free grammar is a set of recursive rules used to generate patterns of strings) defined by the syntax of programming languages such as Backus-Naur Form can be used to model complex entities, such as currency, which has multiple components within it (e.g., symbols, codes, larger units, smaller units, etc.) and supports nested slot mappings. An example template for the selected utterance "He paid $100.78 for dinner" is "He paid / <CURRENCY_slot _augmented> {CURRENCY} / for dinner." Another example is the selected utterance "May 21st Create an (expense|exp) for / (<digit_amount> [ ]<cur_type> |<cur_type> [ ]<digit_amount> ){CURRENCY} / [ (on|at) / <date>Each slot can be one or more values (e.g., potential values or contains one or more elements that map to a list of operations to be performed. For example, continuing the example above, the slot / <CURRENCY_slot _augmented>{CURRENCY} / maps to the element<cur_type> and elements<digit_amount> element<cur_type> is a value list type element, while the element<digit_amount> is a function element.<cur_type> and elements<digit_amount> Each of the maps to a different value. For example,<cur_type> maps to a list of units for the CURRENCY entity,<digit_amount> maps to a function for generating the decimal equivalent of the CURRENCY entity.

[0122] In step 580, a template is selected from the list of templates, a value is selected from the list of values based on a mapping between the slot in the selected template and the list of values for the given entity, and an artificial utterance is created based on the selected template and the selected value. Creating the artificial utterance includes inserting the selected value into the slot of the selected template that maps to the list of values for the given entity. For example, a template engine is run using a list of templates (e.g., a list of templates including the CURRECNY entity) and a list of values (e.g., units) for the entity (e.g., CURRENCY) to be covered to generate the artificial utterance. A configuration file for the template engine allows a user to specify which slots map to entity files, and the template engine will generate the artificial utterance according to the mapping. Multiple templates and / or values can be selected from the list, and multiple artificial utterances can be generated by repeatedly running the template engine using the list of templates and the list of values for the entity to be covered (based on what expansion rate is selected or configured).

[0123] The flowchart in Figure 5C shows the template "He paid / <CURRENCY_slot _augmented> Regarding the example of "{CURRENCY} / for dinner," the artificial utterances (Figures 5A and 5B) are explained below. 5A illustrates a sub-process for generating a named entity recognition model (530; 580). In step (A), based on an analysis of the test data and validation data (e.g., the analysis performed in steps 520; 550 described with respect to FIG. 5A), it is determined that the entity CURRENCY needs to be expanded to improve the performance of the named entity recognition model. Furthermore, an expansion rate is set for the process of creating artificial utterances to expand the training data. In some cases, the expansion rate is selected based on an analysis of the test data and validation data. For example, the expansion rate is selected based on the recognized / determined ability to improve the performance of the named entity recognition model. In other cases, the expansion rate may be randomly selected and then optimized. In other cases, the expansion rate is selected based on a predetermined schema to prevent regression in the ability of one or more predictive models to predict entities from one or more prior observed classes.

[0124] In step (B), a list of templates associated with the determined entity CURRENCY and, optionally, associated with the context of the given entity is generated. From the list, a template is selected (this list of templates may be generated according to step 575 described with respect to FIG. 5B). Once the list of templates is created, it is associated with the entity and, optionally, with the context of the given entity (e.g., a mapping is created between the list and the entity in a file or database). For example, there may be a configuration file for the template engine that allows the user to specify which templates map to which entity files, and the template engine will follow that mapping to identify the list of templates associated with the determined entity CURRENCY. Then, the template (e.g., He paid / <CURRENCY_slot _augmented> {CURRENCY} / for dinner) in a controlled manner (i.e., sequentially or by a predetermined method such as using a fitness function) The templates are selected based on a selection protocol (based on the selection protocol in step 505) or from a list of randomly identified templates. In some cases, template creation and extraction is performed to ensure that the augmented training dataset has the same distribution as the original dataset used to train the model (e.g., from the original corpus (the dataset obtained in step 505 before splitting) or a subset thereof, such as the training dataset after splitting). For example, heuristics may be used for how to select templates. Heuristics may include (i) choosing between templates with multiple entity classes versus templates with a single entity class; (ii) a trade-off between template diversity and template quality; (iii) categorizing templates according to some context, such as with respect to account balances; (iv) whether the model requires a minimum number of occurrences per template / specific context that the user wants to teach the model; or (v) any combination of these.

[0125] In step (C), elements of one or more slots in the selected template are identified. A slot may be identified by parsing the template and locating the text that defines the slot based on the custom syntax used to generate the template. For example, any text between / ****** / may be identified as a slot, any text containing an identifier such as _slot_ may be identified as a slot, or any combination thereof. Once a slot is created, it is associated with one or more elements (e.g., a mapping is created between the slot and the elements in a file or database). For example, there may be a configuration file for the template engine that allows the user to specify which elements map to which slots, and the template engine will identify the elements associated with the identified slots according to the mapping. As shown in the figure, the slot / <CURRENCY_slot _augmented> {CURRENCY} / is the template He paid / <CURRENCY_slot _augmented> {CURRENCY} / for dinner slot / <CURRENCY_slot _augmented> {CURRENCY} / is the element<cur_type> and elements<digit_amount> element<cur_type> is a value list type element, while the element<digit_amount> is a function element.

[0126] In steps (D) and (E), values for elements are identified. When an element of a slot is created, it is associated with a value (e.g., a mapping is created between elements and values in a file or database). For example, there may be a configuration file for the template engine that allows the user to specify which values map to which elements, and this mapping can be used by the template engine to identify the value associated with the identified element. As shown, the element<cur_type> maps to a list of different values for the entity CURRENCY and to a function to generate a random decimal number as the value of the entity CURRENCY.

[0127] In step (F), one or more operations are performed according to the identified values, and the results of the operations are combined to generate a slot value for each identified slot. ,For value list type elements, a value (e.g., SGD$) is selected from a list of different identified values of the entity CURRENCY (i.e., sequentially or based on a predefined selection protocol such as using a fitness function), or randomly. For function type elements, a function is executed to obtain the function's output. For example, a random decimal may be executed to obtain 20.45. The value of the value list type element SGD$ and the value of the function type element are combined to generate the slot value SGD$20.45. Of course, the combination of values can be more complex than a simple sequential ordering of values. For example, in some cases the values may be combined using separate functions or a random order generator.

[0128] In step (G), the slot value for each identified slot is then input into the associated slot in the identified template to generate the artificial utterance. For example, SGD$20.45 is input into the template He paid / <CURRENCY_slot _augmented> {CURRENCY} / for dinner slots / <CURRENCY_slot _augmented> {CURRENCY} / is entered and paid SGD$20.45 for dinner." Multiple templates and / or values can be selected from a list, and multiple artificial utterances can be generated by repeatedly running the template engine with the list of templates and the list of entity values to cover (based on what expansion rate is selected or set).

[0129] The template engine scales well, but this is a simple example with one template; in reality many different templates may be provided by the script, and even a longer hierarchy of further templates or files or functions that the template engine can map. Table 2 shows some examples where nested sets of slot mappings are used in templates to customize data extensions for a specific entity, CURRENCY, and a specific entity, TIME_DATE.

[0130] [Table 2-1]

[0131] [Table 2-2]

[0132] [Table 2-3]

[0133] [Table 2-4]

[0134] Exemplary System 6 is a simplified diagram illustrating a distributed system 600. In the illustrated example, the distributed system 600 includes one or more client computing devices 602, 604, 606, and 608, which are coupled to a server 612 via one or more communication networks 610. The client computing devices 602, 604, 606, and 608 may be configured to execute one or more applications.

[0135] In various examples, server 612 may be adapted to run one or more services or software applications that enable one or more embodiments described in this disclosure. In certain examples, server 612 may also provide other services or software applications, which may include non-virtualized and virtualized environments. In some examples, these services may be provided under a software as a service (SaaS) model or other similar model. These components may be provided as any web-based or cloud services to users of client computing devices 602, 604, 606, and / or 608. Users operating client computing devices 602, 604, 606, and / or 608 may then utilize one or more client applications to interact with server 612 to utilize the services provided by these components.

[0136] 6, server 612 may include one or more components 618, 620, and 622 that implement the functions performed by server 612. These components may include software components, hardware components, or a combination thereof that may be executed by one or more processors. A variety of different system configurations are possible that may differ from distributed system 600. It should be understood that the example shown in Figure 6 is therefore an example of a distributed system for implementing the exemplary system and is not intended to be limiting.

[0137] A user may use client computing devices 602, 604, 606, and / or 608 to run one or more applications, models, or chatbots, which may generate one or more events or models that may be executed or provided in accordance with the teachings of this disclosure. The client devices may provide an interface that allows a user of the client device to interact with the client device. The client devices may also output information to the user through this interface. While FIG. 6 shows only four client computing devices, any number of client computing devices may be supported.

[0138] Client devices may include various types of computing systems, such as portable handheld devices, general-purpose computers such as personal computers and laptops, workstation computers, wearable devices, gaming systems, thin clients, various messaging devices, and sensing devices such as sensors. These computing devices may run various types and versions of software applications and operating systems (e.g., Microsoft Windows®, Apple Macintosh®, UNIX® or UNIX-like OS, Linux® or Linux®), including various mobile operating systems (e.g., Microsoft Windows®, iOS®, Windows Phone®, Android®, BlackBerry®, Palm OS®, etc.). The client device may run a Linux-based operating system such as the Google Chrome® OS. The mobile terminal may include a mobile phone, a smartphone (e.g., an iPhone®), a tablet (e.g., an iPad®), a personal digital assistant (PDA), etc. The wearable device may include a Google Glass® head-mounted display, etc. The gaming system may include various handheld gaming devices, Internet-enabled gaming devices (e.g., Microsoft Xbox® gaming consoles with or without Kinect® gesture input devices, Sony PlayStation® systems, various gaming systems offered by Nintendo®, etc.), etc. The client device may run a variety of different apps, such as various Internet-related apps, communication apps (e.g., email apps, short message service (SMS) apps), etc., and may use a variety of communication protocols.

[0139] The network(s) 610 may be any type of network known to those skilled in the art that can support data communications using any of a variety of available protocols, including, but not limited to, TCP / IP (Transmission Control Protocol / Internet Protocol), SNA (Systems Network Architecture), IPX (Internet Packet Exchange), AppleTalk®, etc. By way of example only, the network(s) 610 may be a local area network (LAN), Ethernet®, a token ring-based network, a wide area network (WAN), the Internet, a virtual network, a virtual private network (VPN), an intranet, an extranet, a public switched telephone network (PSTN), an infrared network, a wireless network (e.g., wireless networks (e.g., Institute of Electrical and Electronics (IEEE) 1002.11 suite of protocols, Bluetooth®, and the like). and / or other wireless protocols) and / or any combination of these and other networks.

[0140] The servers 612 may be comprised of one or more general-purpose computers, dedicated server computers (including, by way of example, PC (personal computer) servers, UNIX servers, mid-range servers, mainframe computers, rack-mounted servers, etc.), server farms, server clusters, or other suitable configurations and / or combinations. The servers 612 may include other computing architectures that involve virtualization, such as one or more virtual machines running a virtual operating system, or one or more flexible pools of logical storage devices that may be virtualized to maintain virtual storage devices for the servers. In various examples, the servers 712 may be adapted to run one or more services or software applications that provide the functionality described in the foregoing disclosure.

[0141] The computing system of server 612 may run one or more operating systems, including any of those described above, as well as any commercially available server operating system. Server 612 may also run any of a variety of additional server and / or mid-tier applications, including an HTTP (Hypertext Transfer Protocol) server, an FTP (File Transfer Protocol) server, a CGI (Common Gateway Interface) server, a JAVA server, a database server, etc. Exemplary database servers include, but are not limited to, those commercially available from Oracle®, Microsoft®, Sybase®, IBM® (International Business Machines), etc.

[0142] In some implementations, server 612 may include one or more applications for parsing and consolidating data feeds and / or event updates received from users of client computing devices 602, 604, 606, and 608. By way of example, the data feeds and / or event updates may include, but are not limited to, Twitter® feeds, Facebook® updates, or real-time updates and continuous data streams received from one or more third-party sources, including real-time events associated with sensor data applications, financial tickers, network performance measurement tools (e.g., network monitoring and traffic management applications), clickstream analysis tools, automotive traffic monitoring, etc. Server 612 may also include one or more applications for displaying the data feeds and / or real-time events via one or more display devices of client computing devices 602, 604, 606, and 608.

[0143] The distributed system 600 may also include one or more data repositories 614, 616. These data repositories may, in certain examples, be used to store data and other information. For example, one or more of the data repositories 614, 616 may be used to store information, such as information about chatbot performance or generated models for use by a chatbot used by the server 612 in performing various functions according to various embodiments. The data repositories 614, 616 may reside in various locations. For example, the data repository used by the server 612 may be local to the server 612 or may be remote from the server 612 and in communication with the server 612 via a network-based or dedicated connection. The data repositories 614, 616 may be of different types. In certain examples, the data repository used by the server 612 may be, for example, a relational database, such as databases provided by Oracle Corporation® and other vendors. One or more of these databases ... The numbers may be adapted to allow the storage, updating and retrieval of data to and from the database in response to SQL format commands.

[0144] In particular examples, one or more of the data repositories 614, 616 may also be used by an application to store application data. The data repositories used by an application may be of different types, such as, for example, a key-value store repository, an object store repository, or a general storage repository supported by a file system.

[0145] In certain examples, the functionality described in this disclosure may be provided as services via a cloud environment. FIG. 7 is a simplified block diagram of a cloud-based system environment in which various services may be provided as cloud services, according to certain examples. In the example shown in FIG. 7, cloud infrastructure system 702 may provide one or more cloud services that may be requested by users using one or more client computing devices 704, 706, and 708. Cloud infrastructure system 702 may comprise one or more computers and / or servers, which may include those described above for server 712. The computers in cloud infrastructure system 702 may be organized as general-purpose computers, dedicated server computers, server farms, server clusters, or any other suitable configuration and / or combination.

[0146] Network(s) 710 may facilitate the communication and exchange of data between clients 704, 706, and 708 and cloud infrastructure system 702. Network 710 may include one or more networks. These networks may be of the same type or different types. Network 710 may support one or more communication protocols, including wired and / or wireless protocols, to facilitate communication.

[0147] The example shown in Figure 7 is merely one example of a cloud infrastructure system and is not intended to be limiting. It should be understood that in some other examples, cloud infrastructure system 702 may have more or fewer components than those shown in Figure 7, may combine two or more components, or may have a different configuration or arrangement of components. For example, while Figure 7 shows three client computing devices, in alternative examples, any number of client computing devices may be supported.

[0148] The term cloud service is generally used to refer to services provided on demand to users via a communication network such as the Internet by a service provider's system (e.g., cloud infrastructure system 702). Generally, in a public cloud environment, the servers and systems that make up the cloud service provider's system are different from the customer's own on-premise servers and systems. The cloud service provider's system is managed by the cloud service provider. As a result, customers can use the cloud services provided by the cloud service provider without having to purchase licenses, support, or hardware and software resources separately. For example, the cloud service provider's system hosts applications, and users can order and use the applications as needed via the Internet without having to purchase infrastructure resources to run the applications. Cloud services are designed to provide easy and scalable access to applications, resources, and services. Several providers offer cloud services. For example, middleware Some cloud services, such as software services, database services, and Java® cloud services, are offered by Oracle Corporation®, located in Redwood Shores, California.

[0149] In certain examples, the cloud infrastructure system 702 may support a Software as a Service (SaaS) model, a Platform as a Service (PaaS) model, an Infrastructure as a Service (IaaS) model, and other models, including hybrid service models. Cloud infrastructure system 702 may provide one or more cloud services using different models, such as under the cloud infrastructure model. Cloud infrastructure system 702 may include a set of applications, middleware, databases, and other resources that enable the provision of various cloud services.

[0150] The SaaS model allows applications or software to be delivered to customers as a service over a communications network such as the Internet, without the customer having to purchase hardware or software for the underlying application. For example, the SaaS model may be used to provide customers with access to on-demand applications hosted by cloud infrastructure system 702. Examples of SaaS services offered by Oracle Corporation® include, but are not limited to, various services such as human resources / capital management, customer relationship management (CRM), enterprise resource planning (ERP), supply chain management (SCM), enterprise performance management (EPM), analytics services, and social applications.

[0151] The IaaS model is commonly used to provide infrastructure resources (e.g., servers, storage, hardware, and network resources) as cloud services to customers, providing elastic compute and storage capabilities. Oracle Corporation (registered trademark) offers a variety of IaaS services.

[0152] The PaaS model is generally used to provide platform and environment resources as a service that enable customers to develop, run, and manage applications and services without having to procure, build, or maintain such resources. Examples of PaaS services offered by Oracle Corporation (registered trademark) include, but are not limited to, Oracle Java Cloud Service (JCS), Oracle Database Cloud Service (DBCS), data management cloud services, and various application development solution services.

[0153] Cloud services are generally provided in an on-demand self-service, subscription-based, elastically scalable, reliable, highly available, and secure manner. For example, a customer may order one or more services provided by cloud infrastructure system 702 via a subscription order. Cloud infrastructure system 702 then executes processing to provide the services requested in the customer's subscription order. For example, a user may use utterances to request the cloud infrastructure system to take a particular action (e.g., intent), as described above, and / or to provide a service to a chatbot system, as described herein. Cloud infrastructure system 702 may be configured to provide one or more cloud services.

[0154] Cloud infrastructure system 702 may provide cloud services through a variety of deployment models. In a public cloud model, cloud infrastructure system 702 may be owned by a third-party cloud service provider, and cloud services are offered to customers, who may be individuals, businesses, or any member of the general public. In certain other examples, under a private cloud model, cloud infrastructure system 702 may be operated within an organization (e.g., within a corporate organization), and services are offered to customers within the organization. For example, customers may be various departments of a company, such as a human resources department, a payroll department, or individuals within the company. In certain other examples, under a community cloud model, cloud infrastructure system 702 and the services offered may be shared by several organizations within an associated community. Various other models, such as hybrids of the above models, may also be used.

[0155] Client computing devices 704, 706, and 708 may be of different types (such as client computing devices 602, 604, 606, and 608 shown in FIG. 6 ) and may be capable of running one or more client applications. Users may use the client devices to interact with cloud infrastructure system 702, such as to request services provided by cloud infrastructure system 702. For example, users may use client devices to request information or actions from a chatbot as described in this disclosure.

[0156] In some examples, the processing performed by cloud infrastructure system 702 to provide services may include model training and deployment. This analysis may include using, analyzing, and manipulating datasets to train and deploy one or more models. This analysis may be performed by one or more processors, possibly processing the data in parallel or performing simulations using the data. For example, big data analysis may be performed by cloud infrastructure system 702 to generate and train one or more models for a chatbot system. The data used in this analysis may include structured data (e.g., data stored in a database or structured according to a structured model) and / or unstructured data (e.g., data blobs (binary large objects)).

[0157] 7 illustrates, cloud infrastructure system 702 may include infrastructure resources 730 utilized to facilitate the provision of various cloud services offered by cloud infrastructure system 702. Infrastructure resources 730 may include, for example, processing resources, storage or memory resources, networking resources, etc. In particular examples, storage virtual machines available to provide storage requested by applications may be part of cloud infrastructure system 702. In other examples, the storage virtual machines may be part of a different system.

[0158] In particular examples, to facilitate efficient provisioning of these resources to support various cloud services offered by cloud infrastructure system 702 to various customers, resources may be bundled into sets of resources or resource modules (also referred to as "pods"). Each resource module or pod may comprise a pre-integrated and optimized combination of one or more types of resources. In particular examples, different pods may be pre-provisioned for different types of cloud services. For example, a first set of pods may include: A second set of pods, which may include a different combination of resources than the pods in the first set of pods, may be provisioned for a database service, a second set of pods, which may include a different combination of resources than the pods in the first set of pods, may be provisioned for a Java service, etc. For some services, resources allocated for provisioning of a service may be shared between services.

[0159] Cloud infrastructure system 702 itself may use services 732 internally that are shared by various components of cloud infrastructure system 702, which facilitates service provisioning by cloud infrastructure system 702. These internal shared services may include, but are not limited to, security and identity services, integration services, enterprise repository services, enterprise manager services, virus scanning and whitelisting services, high availability, backup and recovery services, services to enable cloud support, email services, notification services, file transfer services, etc.

[0160] Cloud infrastructure system 702 may be comprised of multiple subsystems. These subsystems may be implemented in software, hardware, or a combination thereof. As shown in FIG. 7 , these subsystems may include a user interface subsystem 712 that allows users or customers of cloud infrastructure system 702 to interact with cloud infrastructure system 702. User interface subsystem 712 may include a variety of different interfaces, such as a web interface 714, an online store interface 716 through which cloud services offered by cloud infrastructure system 702 are advertised and available for purchase by consumers, and other interfaces 718. For example, a customer may use a client device to request one or more services offered by cloud infrastructure system 702 using one or more of interfaces 714, 716, and 718 (service request 734). For example, a customer may access an online store to browse cloud services offered by cloud infrastructure system 702 and place a subscription order for one or more services offered by cloud infrastructure system 702 for which the customer wishes to subscribe. The service request may include information identifying the customer and the one or more services for which the customer wishes to subscribe. For example, a customer may place a subscription order for services provided by cloud infrastructure system 702. As part of the order, the customer may provide information identifying the chatbot system for which the services are to be provided, and optionally, one or more credentials for that chatbot system.

[0161] 7, cloud infrastructure system 702 may include an order management subsystem (OMS) 720 configured to process new orders. As part of this processing, OMS 720 may be configured to create an account for the customer if not already done so, receive billing and / or billing information from the customer that is used to issue a bill to the customer for providing the customer with the requested services, verify the customer information, and upon verification, finalize the customer's order, and orchestrate various workflows to prepare the order for provisioning.

[0162] Upon proper validation, OMS 720 may then invoke an order provisioning subsystem (OPS) 724 configured to provide resources for the order, including processing, memory, and networking resources. Provisioning allocates resources for the order to facilitate the service requested by the customer order. The manner in which resources are provisioned for an order and the type of resources provisioned may depend on the type of cloud service ordered by the customer. For example, according to one workflow, OPS 724 may be configured to determine that a particular cloud service is requested and identify the number of pods that may be pre-configured for that particular cloud service. The number of pods allocated to an order may depend on the size / amount / level / scope of the requested service. For example, the number of pods allocated may be determined based on the number of users supported by the service, the duration for which the service is requested, etc. The allocated pods may then be customized to the particular requesting customer to provide the requested service.

[0163] In certain examples, the setup phase processing as described above may be performed by cloud infrastructure system 702 as part of a provisioning process. Cloud infrastructure system 702 may generate an application ID and select a storage virtual machine for the application from among storage virtual machines provided by cloud infrastructure system 702 itself or from storage virtual machines provided by other systems other than cloud infrastructure system 702.

[0164] Cloud infrastructure system 702 may send a response or notification 744 to the requesting issuing customer to indicate when the requested service will be ready for use. In some cases, information (e.g., a link) may be sent to the customer that enables the customer to begin using and utilizing the benefits of the requested service. In a particular example, when a customer requests a service, the response may include a chatbot system ID generated by cloud infrastructure system 702 and information identifying the chatbot system selected by cloud infrastructure system 702 that corresponds to the chatbot system ID.

[0165] Cloud infrastructure system 702 may provide services to multiple customers. For each customer, cloud infrastructure system 702 is responsible for managing information related to one or more subscription orders received from the customer, maintaining customer data related to these orders, and providing the requested services to the customer. Cloud infrastructure system 702 may also collect usage statistics regarding the customer's use of the subscribed services. For example, statistics may be collected on the amount of storage used, the amount of data transferred, the number of users, and system up and down time. This usage information may be used to bill the customer. Billing may occur, for example, on a monthly cycle.

[0166] Cloud infrastructure system 702 may provide services to multiple customers concurrently. Cloud infrastructure system 702 may store information for these customers, possibly including proprietary information. In particular examples, cloud infrastructure system 702 includes an identity management subsystem (IMS) 728 configured to manage customer information and segregate the managed information so that information associated with one customer is inaccessible by another customer. The MS 728 may be configured to provide various security-related services, such as information access management, authentication and authorization services, identity services, such as services for managing customer identities and roles and related functions.

[0167] 8 illustrates an example of a computer system 800. In some examples, the computer system 800 may be a digital assistant or chatbot system in a distributed environment. 8, computer system 800 includes various subsystems, including a processing subsystem 804, which communicates with several other subsystems via a bus subsystem 802. These other subsystems may include a processing acceleration unit 806, an I / O subsystem 808, a storage subsystem 818, and a communication subsystem 824. Storage subsystem 818 may include a non-transitory computer-readable storage medium, including a storage medium 822 and a system memory 910.

[0168] Bus subsystem 802 provides a mechanism for allowing the various components and subsystems of computer system 800 to communicate with each other as intended. While bus subsystem 802 is shown schematically as a single bus, alternative embodiments of the bus subsystem may utilize multiple buses. Bus subsystem 802 may be any of several types of bus structures, including a memory bus or memory controller, a peripheral bus, a local bus, etc., using any of a variety of bus architectures. For example, such architectures include the Industry Standard Architecture (ISA) bus, the Micro Channel Architecture (MCA) bus, the Enhanced ISA (EISA) bus, the Video Electronics Standards Association (VESA) local bus, and IEEE P1386. Such buses may include a Peripheral Component Interconnect (PCI) bus, which may be implemented as a mezzanine bus manufactured in accordance with the PCI standard.

[0169] The processing subsystem 804 controls the operation of the computer system 800 and may include one or more processors, application specific integrated circuits (ASICs), and / or other hardware components. These processors may be implemented as single-core processors or FPGAs (Application Specific Integrated Circuits). The processing resources of computer system 800 may include a processor or multi-core processor. The processing resources of computer system 800 may be organized into one or more processing units 832, 834, etc. A processing unit may include one or more processors, one or more cores from the same or different processors, a combination of cores and processors, or other combinations of cores and processors. In some examples, processing subsystem 804 may include one or more special-purpose co-processors, such as a graphics processor, a digital signal processor (DSP), etc. In some examples, some or all of the processing units of processing subsystem 804 may be implemented using customized circuitry, such as an application-specific integrated circuit (ASIC) or a field-programmable gate array (FPGA).

[0170] In some examples, processing units within processing subsystem 804 may execute instructions stored in system memory 810 or on computer-readable storage medium 822. In various examples, the processing units may execute various program or code instructions and may maintain multiple concurrently executing programs or processes. At any given time, some or all of the program code being executed may reside in system memory 810 and / or on computer-readable storage medium 822 (including, in some cases, on one or more storage devices). Through suitable programming, processing subsystem 804 may provide the various functions described above. In instances where computer system 800 is running one or more virtual machines, one or more processing units may be assigned to each virtual machine.

[0171] In a particular example, the processing acceleration unit 806 may be configured by the computer system 800 to It may optionally be provided to accelerate the overall processing performed, to perform customized processing, or to offload part of the processing performed by the processing subsystem 804.

[0172] Input / output subsystem 808 may include devices and mechanisms for inputting information into computer system 800 and / or outputting information from or through computer system 800. In general, use of the term input device is intended to include all possible types of devices and mechanisms for inputting information into computer system 800. User interface input devices may include, for example, keyboards, pointing devices such as mice or trackballs, touchpads or touchscreens integrated into displays, scroll wheels, click wheels, dials, buttons, switches, keypads, voice input devices with voice command recognition systems, microphones, and other types of input devices. User interface input devices may also include motion-sensing and / or gesture-recognition devices such as Microsoft Kinect® motion sensors, Microsoft Xbox® 360 game controllers, and devices that provide an interface for receiving input using gestures and voice commands, allowing a user to control and interact with the input device. The user interface input devices may also include eye gesture recognition devices, such as a Google Glass® blink detector, that detects eye activity from the user (e.g., "blinking" while taking a picture and / or making a menu selection) and translates the eye gesture as input to the input device (e.g., Google Glass®). Additionally, the user interface input devices may include a voice recognition sensing device that allows the user to interact with a voice recognition system (e.g., Siri Navigator) through voice commands.

[0173] Other examples of user interface input devices include, but are not limited to, three-dimensional (3D) mice, joysticks or pointing sticks, gamepads and graphic tablets, as well as audio / visual devices (such as speakers, digital cameras, digital camcorders, portable media players, webcams, image scanners, fingerprint scanners, barcode reader 3D scanners, 3D printers, laser range finders, and eye-tracking devices). Additionally, user interface input devices may include medical imaging input devices, such as, for example, computed tomography, magnetic resonance imaging, positron emission tomography, and medical ultrasound devices. User interface input devices may also include audio input devices, such as, for example, MIDI keyboards, digital musical instruments, and the like.

[0174] In general, use of the term "output device" is intended to include all possible types of devices and mechanisms for outputting information from computer system 800 to a user or to another computer. User interface output devices may include non-visual displays such as a display subsystem, indicator lights, or audio output devices. Display subsystems may be flat-panel devices such as those using cathode ray tubes (CRTs), liquid crystal displays (LCDs), or plasma displays, projection devices, touchscreens, etc. For example, user interface output devices may include, but are not limited to, various display devices that visually convey text, graphics, and audio / visual information, such as monitors, printers, speakers, headphones, automobile navigation systems, plotters, audio output devices, and modems.

[0175] The storage subsystem 818 stores information used by the computer system 800. The storage subsystem 818 provides a repository or data store for storing information and data. The storage subsystem 818 provides a tangible, non-transitory computer-readable storage medium for storing the basic programming and data structures that provide some example functionality. The storage subsystem 818 may store software (e.g., programs, code modules, instructions) that, when executed by the processing subsystem 804, provide the functionality described above. This software may be executed by one or more processing units of the processing subsystem 804. The storage subsystem 818 may also provide authentication according to the teachings of the present disclosure.

[0176] The storage subsystem 818 may include one or more non-transitory memory devices, including volatile and non-volatile memory devices. As shown in FIG. 8, the storage subsystem 818 includes a system memory 810 and a computer-readable storage medium 822. The system memory 810 may include several memories, including a volatile main random access memory (RAM) for storing instructions and data during program execution, and a non-volatile read-only memory (ROM) or flash memory in which fixed instructions are stored. In some implementations, a basic input / output system (BIS) containing basic routines that help to transfer information between elements within the computer system 800, such as during start-up. The system memory 810 may be stored in ROM, RAM, or other memory devices. ... The memory may include multiple different types of memory, such as dynamic random access memory (DRAM).

[0177] 8, system memory 810 may load running application programs 812 (which may include various applications such as a web browser, a middle-tier application, a relational database management system (RDBMS), etc.), program data 814, and operating system 816. By way of example, and not limitation, operating system 816 may include various versions of Microsoft Windows®, Apple Macintosh®, and / or Linux operating systems, various commercially available UNIX® or UNIX-like operating systems (including, but not limited to, various GNU / Linux operating systems, Google Chrome® OS, etc.), and mobile operating systems such as iOS, Windows® Phone, Android® OS, BlackBerry® OS, Palm® OS, etc.

[0178] The computer-readable storage medium 822 may store programming and data structures that provide some example functionality. The computer-readable medium 822 may provide storage of computer-readable instructions, data structures, program modules, and other data for the computer system 800. Software (programs, code modules, instructions) that, when executed by the processing subsystem 804, provide the above-described functionality may be stored in the storage subsystem 818. As an example, the computer-readable storage medium 822 may include non-volatile memory such as a hard disk drive, a magnetic disk drive, a CD-ROM, a DVD, an optical disk drive such as a Blu-ray disc, or other optical media. The computer-readable storage medium 822 may also include a Zip drive, a flash memory card, a universal serial bus (USB) flash drive, a secure digital (SD) card, a DVD, a DVD drive ... The computer-readable storage medium 822 may include, but is not limited to, a flash memory-based SSD, an enterprise flash drive, a solid-state drive (SSD) based on non-volatile memory such as solid-state ROM, solid-state RAM, dynamic RAM, etc. M, SSDs based on volatile memory such as static RAM, DRAM-based SSDs, magnetoresistive RAM (MRAM) SSDs, and hybrid SSDs that use a combination of DRAM-based SSDs and flash memory-based SSDs.

[0179] In particular examples, storage subsystem 818 may include a computer-readable storage medium reader 820 that may be further connected to a computer-readable storage medium 822. Reader 820 may be configured to receive and read data from a memory device such as a disk, flash drive, or the like.

[0180] In certain examples, computer system 800 may support virtualization techniques, including but not limited to virtualization of processing and memory resources. For example, computer system 800 may provide support for running one or more virtual machines. In certain examples, computer system 800 may execute a program such as a hypervisor that facilitates configuration and management of virtual machines. Each virtual machine may be assigned memory, computational (e.g., processors, cores), input / output, and networking resources. Each virtual machine generally operates independently from other virtual machines. A virtual machine generally runs its own operating system, which may be the same or different from the operating systems run by other virtual machines executed by computer system 800. Thus, in some cases, multiple operating systems may be executed simultaneously by computer system 800.

[0181] The communications subsystem 824 provides an interface to other computer systems and networks. The communications subsystem 824 serves as an interface for sending and receiving data between other systems and the computer system 800. For example, the communications subsystem 824 may enable the computer system 800 to establish a communications channel with one or more client devices over the Internet to send and receive information to and from the client devices. For example, if the computer system 800 is used to implement the bot system 120 shown in FIG. 1, the communications subsystem may be used to communicate with a chatbot system selected for the application.

[0182] The communications subsystem 824 may support both wired and / or wireless communications protocols. In particular examples, the communications subsystem 824 may include a radio frequency (RF) transceiver component for accessing wireless voice and / or data networks (e.g., cellular telephone technology, advanced data network technologies such as 3G, 4G, or EDGE (enhanced data rates for global evolution), Wi-Fi (IEEE 802.XX family of standards, or other mobile communications technologies, or any combination thereof), a global positioning system (GPS), or other wireless communications technologies. S) receiving components and / or other components. In some examples, the communications subsystem 824 may provide wired network connectivity (e.g., Ethernet) in addition to or instead of a wireless interface.

[0183] The communications subsystem 824 may transmit and receive data in a variety of formats. In some examples, among other formats, the communications subsystem 824 may transmit and receive data in structured and / or unstructured formats. The incoming communications may be received in the form of unstructured data feeds 826, event streams 828, event updates 830, etc. For example, the communications subsystem 824 may receive web feeds such as Twitter® feeds, Facebook® updates, Rich Site Summary (RSS) feeds, and / or other data sources. Alternatively, it may be configured to receive (or send) data feeds 826, such as real-time updates from one or more third-party sources, in real time from users of social media networks and / or other communication services.

[0184] In particular examples, the communications subsystem 824 may be configured to receive data in the form of a continuous data stream, which may include an event stream 828 of real-time events and / or event updates 830, which may be continuous or infinite in nature with no clear termination. Examples of applications that generate continuous data may include, for example, sensor data applications, financial tickers, network performance measurement tools (e.g., network monitoring and traffic management applications), clickstream analysis tools, automobile traffic monitoring, etc.

[0185] Communications subsystem 824 may also be configured to communicate data from computer system 800 to other computer systems or networks. This data may be communicated in a variety of different formats, such as structured and / or unstructured data feeds 826, event streams 828, event updates 830, etc., to one or more databases, which may communicate with one or more streaming data source computers coupled to computer system 800.

[0186] Computer system 800 may be one of a variety of types, such as a handheld portable device (e.g., an iPhone® mobile phone, an iPad® computing tablet, a PDA), a wearable device (e.g., a Google Glass® head-mounted display), a personal computer, a workstation, a mainframe, a kiosk, a server rack, or any other data processing system. Because the nature of computers and networks is constantly changing, the description of computer system 800 shown in FIG. 8 is intended only as a specific example. Many other configurations are possible, having more or fewer components than the system shown in FIG. 8. It should be understood that there are other ways and / or methods of implementing the various examples, based on the disclosure and teachings provided herein.

[0187] While specific examples have been described, various modifications, variations, alternative constructions, and equivalents are possible. The examples are not limited to operation in any particular data processing environment, but can freely operate in multiple data processing environments. Furthermore, while specific examples have been described using a particular sequence of transactions and steps, it will be apparent to those skilled in the art that this is not intended to be limiting. While some flowcharts describe operations as a sequential process, many of these operations may be performed in parallel or simultaneously. The order of operations may also be rearranged. A process may have additional steps not included in the figures. Various features and aspects of the above examples may be used individually or together.

[0188] Additionally, while particular examples have been described using particular combinations of hardware and software, it should be recognized that other combinations of hardware and software are possible. Particular examples may be implemented solely in hardware, or may be implemented solely in software. The various processes described herein may be implemented on the same processor in any combination, or on different processors in any combination.

[0189] Although a device, system, component, or module is described as being configured to perform particular operations or functions, such configuration may be achieved, for example, by designing an electronic circuit to perform the operations, by programming a programmable electronic circuit (such as a microprocessor) to perform the operations, such as executing computer instructions or code that is programmed to execute code or instructions stored on a non-transitory memory medium, or a processor or core, or any combination thereof. Processes may communicate using a variety of techniques, including but not limited to conventional techniques for inter-process communication, and different process pairs may use different techniques, and the same process pair may use different techniques at different times.

[0190] Specific details are provided in this disclosure to provide a thorough understanding of the examples. However, the examples can be practiced without these specific details. For example, well-known circuits, processes, algorithms, structures, and techniques are shown without unnecessary detail to avoid obscuring the examples. This description provides only illustrative examples and is not intended to limit the scope, applicability, or configuration of other examples. Rather, the above description of the examples will provide one skilled in the art with an enabling description for implementing various examples. Various changes may be made in the function and arrangement of elements.

[0191] Accordingly, the specification and drawings are to be regarded in an illustrative rather than a restrictive sense. It will be apparent, however, that additions, subtractions, deletions, and other modifications and alterations may be made without departing from the broader spirit and scope of the appended claims. Accordingly, although specific examples have been described, these are not intended to be limiting. Various modifications and equivalents are within the scope of the following claims.

[0192] While the foregoing specification describes aspects of the disclosure with reference to specific examples thereof, those skilled in the art will recognize that the disclosure is not limited thereto. Various features and aspects of the above disclosure may be used individually or together. Moreover, the examples may be utilized in many environments and applications beyond those described herein without departing from the broader spirit and scope of the specification. Accordingly, the specification and drawings should be regarded in an illustrative rather than a restrictive sense.

[0193] In the above description, the methods have been described in a particular order for purposes of illustration. It should be understood that in alternative examples, the methods may be performed in an order different from that described. It should also be understood that the methods described above may be performed by hardware components or embodied in a sequence of machine-executable instructions that can be used to cause a machine, such as a general-purpose or special-purpose processor or logic circuitry programmed with the instructions, to perform the methods. These machine-executable instructions may be stored on one or more machine-readable media, such as a CD-ROM or other type of optical disk, a floppy diskette, ROM, RAM, EPROM, EEPROM, magnetic or optical card, flash memory, or other type of machine-readable medium suitable for storing electronic instructions. Alternatively, the methods may be performed by a combination of hardware and software.

[0194] Where a component is described as being configured to perform a particular operation, such configuration may be achieved, for example, by designing electronic circuitry or other hardware to perform the operation, by programming a programmable electronic circuitry (e.g., a microprocessor or other suitable electronic circuitry) to perform the operation, or by any combination thereof.

[0195] While illustrative examples of the present application have been described in detail herein, it is to be understood that the inventive concepts may be embodied and employed in various ways, and the appended claims are intended to be construed to embrace such variations except insofar as limited by the prior art.< / date>

Claims

1. 1. A method comprising: a data processing system generating a list of values to cover the entity; the data processing system selecting, from a dataset, an utterance having a context for the entity; and the data processing system converting the utterance into a plurality of templates, each of the plurality of templates including a slot for the entity that maps to the list of values, the method further comprising: the data processing system selecting a template from the plurality of templates; the data processing system selecting a value from the list of values based on a mapping between the slot in the selected template and the list of values for the entity; and generating artificial speech using the selected template and the selected value, wherein generating the artificial speech includes inserting the selected value into the slot of the selected template.

2. The method further comprises, before selecting an utterance from the dataset, the data processing system training a model using a training dataset, the training dataset being the dataset or a subset thereof, the method further comprising: the data processing system testing the model using a test data set or a validation data set; the data processing system evaluating performance of the model based on testing the model; The method of claim 1 , further comprising the data processing system identifying the entity based on the evaluating.

3. the data processing system augmenting the data set or the training data set with the artificial utterances to generate an augmented training data set; the data processing system training the model using the augmented training data set; the data processing system retesting the model using the test data set or the validation data set, or another test data set or another validation data set; the data processing system evaluating the performance of the model based on retesting the model; determining by the data processing system that the performance of the model meets predetermined criteria; 3. The method of claim 2, further comprising: in response to the model satisfying the predetermined criteria, deploying the model as part of a chatbot system.

4. the data processing system determining that the slot in the selected template includes a first element that maps to the list of values and a second element that maps to a function; the data processing system processes the function to generate a function value based on a mapping between the slot in the selected template and the function; the data processing system further comprising combining the selected value with the function value to obtain a slot value; The method of claim 1 , wherein generating the artificial utterance comprises inserting the slot value into the slot of the selected template.

5. the data processing system determining that the selected template maps to a nested set of slot mappings that includes the slot and other slots; the data processing system determining that the other slots in the nested set of slot mappings map to other lists of values; the data processing system further comprising: selecting another value from the list of other values based on a mapping between the other slot in the selected template and the list of other values; 2. The method of claim 1 , wherein generating the artificial utterance comprises inserting the selected value into the slot of the selected template and inserting the selected other value into the other slot of the selected template.

6. 10. The method of claim 1, further comprising the data processing system augmenting the data set or a training data set with the artificial utterance to generate an augmented training data set for training a model to recognize and classify multiple entities including the entity.

7. 10. The method of claim 1, wherein the templates are selected based on heuristics including: (i) a choice between templates with multiple entity classes and templates with a single entity class; (ii) a trade-off between template diversity and template quality; (iii) a classification of templates according to context; (iv) the model requires a minimum number of occurrences per template / specific context that a user wants to teach the model; or (v) any combination thereof.

8. 1. A system comprising: one or more processors; a memory coupled to the one or more processors, the memory storing a plurality of instructions executable by the one or more processors, the plurality of instructions, when executed by the one or more processors, causing the one or more processors to perform an operation, the operation comprising: generating a list of values to cover the entity; selecting an utterance from a dataset that has a context of the entity; and converting the utterance into a plurality of templates, each of the plurality of templates including a slot for the entity that maps to the list of values, the process further comprising: selecting a template from the plurality of templates; selecting a value from the list of values based on a mapping between the slot in the selected template and the list of values for the entity; generating an artificial utterance using the selected template and the selected value, wherein generating the artificial utterance includes inserting the selected value into the slot of the selected template.

9. The process further comprises: and prior to selecting utterances from the dataset, training a model using a training dataset, the training dataset being the dataset or a subset thereof, the process further comprising: testing the model using a test or validation data set; 、 evaluating the performance of the model based on testing the model; and and identifying the entity based on the evaluating.

10. The process further comprises: augmenting the dataset or the training dataset with the artificial utterances to generate an augmented training dataset; training the model using the augmented training data set; retesting the model using the test or validation data set or another test or validation data set; assessing the performance of the model based on retesting the model; and determining that the performance of the model meets predetermined criteria; and deploying the model as part of a chatbot system in response to the model satisfying the predetermined criteria.

11. The process further comprises: determining that the slot in the selected template includes a first element that maps to the list of values and a second element that maps to a function; processing the function to generate a function value based on a mapping between the slots in the selected template and the function; combining the selected value with the function value to obtain a slot value; The system of claim 8 , wherein generating the artificial utterance comprises inserting the slot value into the slot of the selected template.

12. The process further comprises: determining that the selected template maps to a nested set of slot mappings that includes the slot and other slots; determining that the other slots in the nested set of slot mappings map to other lists of values; selecting another value from the list of other values based on a mapping between the other slot in the selected template and the list of other values; 9. The system of claim 8, wherein generating the artificial utterance includes inserting the selected value into the slot of the selected template and inserting the selected other value into the other slot of the selected template.

13. 10. The system of claim 8, wherein the processing further comprises augmenting the data set or a training data set with the artificial utterances to generate an augmented training data set for training a model to recognize and classify multiple entities including the entity.

14. 10. The system of claim 8, wherein the templates are selected based on heuristics including: (i) a choice between templates with multiple entity classes or templates with a single entity class; (ii) a trade-off between template diversity and template quality; (iii) a classification of templates according to context; (iv) the model requires a minimum number of occurrences per template / specific context that a user wants to teach the model; or (v) any combination thereof.

15. A non-transitory computer-readable memory storing a plurality of instructions executable by one or more processors, the plurality of instructions comprising: and instructions that, when executed by the one or more processors, cause the one or more processors to perform a process, the process comprising: generating a list of values to cover the entity; selecting an utterance from a dataset that has a context of the entity; and converting the utterance into a plurality of templates, each of the plurality of templates including a slot for the entity that maps to the list of values, the process further comprising: selecting a template from the plurality of templates; selecting a value from the list of values based on a mapping between the slot in the selected template and the list of values for the entity; generating an artificial utterance using the selected template and the selected value, wherein generating the artificial utterance includes inserting the selected value into the slot of the selected template.

16. The process further comprises: and prior to selecting utterances from the dataset, training a model using a training dataset, the training dataset being the dataset or a subset thereof, the process further comprising: testing the model using a test or validation data set; evaluating the performance of the model based on testing the model; and and identifying the entity based on the evaluating.

17. The process further comprises: augmenting the dataset or the training dataset with the artificial utterances to generate an augmented training dataset; training the model using the augmented training data set; retesting the model using the test or validation data set or another test or validation data set; assessing the performance of the model based on retesting the model; and determining that the performance of the model meets predetermined criteria; and in response to the model satisfying the predetermined criteria, deploying the model as part of a chatbot system.

18. The process further comprises: determining that the slot in the selected template includes a first element that maps to the list of values and a second element that maps to a function; processing the function to generate a function value based on a mapping between the slots in the selected template and the function; combining the selected value with the function value to obtain a slot value; 16. The non-transitory computer-readable memory of claim 15, wherein generating the artificial utterance comprises inserting the slot value into the slot of the selected template.

19. The process further comprises: determining that the selected template maps to a nested set of slot mappings that includes the slot and other slots; determining that the other slots in the nested set of slot mappings map to other lists of values; selecting another value from the list of other values based on a mapping between the other slot in the selected template and the list of other values; 16. The non-transitory computer-readable memory of claim 15, wherein generating the artificial utterance includes inserting the selected value into the slot of the selected template and inserting the selected other value into the other slot of the selected template.

20. 16. The non-transitory computer-readable memory of claim 15, wherein the processing further comprises augmenting the data set or training data set with the artificial utterances to generate an augmented training data set for training a model to recognize and classify multiple entities including the entity.

Images